Lemmatization

Sometimes, it is more appropriate to consider the canonical forms as tokens instead of their variations. For example, if you want to analyze the usage of the word "transformer" in NLP literature for each year, you want to count both "transformer" and 'transformers' as a single item.

Lemmatization is a task that simplifies words into their base or dictionary forms, known as lemmas, to simplify the interpretation of their core meaning.

Q7: What is the difference between a lemmatizer and a stemmer [3]?

When analyzing dat/word_types-token.txt obtained by the tokenizer in the previous section, the following tokens are recognized as separate word types:

• Universities

• University

• universities

• university

Two variations are applied to the noun "university" - capitalization, generally used for proper nouns or initial words, and pluralization, which indicates multiple instances of the term. On the other hand, verbs can also take several variations regarding tense and aspect:

• study

• studies

• studied

• studying

We want to develop a lemmatizer that normalizes all variations into their respective lemmas.

Lemma Lexica

Let us create lexica for lemmatization:

import json
import os
from types import SimpleNamespace

• L1: JSON encoder and decoder

• L2: Common pathname manipulations

• L3: SimpleNamespace

def get_lexica(res_dir: str) -> SimpleNamespace:
    with open(os.path.join(res_dir, 'nouns.txt')) as fin: nouns = {noun.strip() for noun in fin}
    with open(os.path.join(res_dir, 'verbs.txt')) as fin: verbs = {verb.strip() for verb in fin}
    with open(os.path.join(res_dir, 'nouns_irregular.json')) as fin: nouns_irregular = json.load(fin)
    with open(os.path.join(res_dir, 'verbs_irregular.json')) as fin: verbs_irregular = json.load(fin)
    with open(os.path.join(res_dir, 'nouns_rules.json')) as fin: nouns_rules = json.load(fin)
    with open(os.path.join(res_dir, 'verbs_rules.json')) as fin: verbs_rules = json.load(fin)

    return SimpleNamespace(
        nouns=nouns,
        verbs=verbs,
        nouns_irregular=nouns_irregular,
        verbs_irregular=verbs_irregular,
        nouns_rules=nouns_rules,
        verbs_rules=verbs_rules
    )

• L1: res_dir: the path to the root directory where all lexica files are located.

• L2: nouns.txt: a list of base nouns

• L3: verbs.txt: a list of base verbs

• L4: nouns_irregular.json: a dictionary of nouns whose plural forms are irregular (e.g., mouse → mice)

• L5: verbs_irregular.json: a dictionary of verbs whose inflection forms are irregular (e.g., buy → bought)

• L6: nouns_rules.json: a list of pluralization rules for nouns

• L7: verbs_rules.json: a list of inflection rules for verbs

We then verify that all lexical resources are loaded correctly:

Code

print(len(lexica.nouns))
print(len(lexica.verbs))
print(lexica.nouns_irregular)
print(lexica.verbs_irregular)
print(lexica.nouns_rules)
print(lexica.verbs_rules)

Output

Q8: What are the key differences between inflectional and derivational morphology?

Lemmatizing

Let us write the lemmatize() function that takes a word and lemmatizes it using the lexica:

def lemmatize(lexica: SimpleNamespace, word: str) -> str:
    def aux(word: str, vocabs: dict[str, str], irregular: dict[str, str], rules: list[tuple[str, str]]):
        lemma = irregular.get(word, None)
        if lemma is not None: return lemma

        for p, s in rules:
            lemma = word[:-len(p)] + s
            if lemma in vocabs: return lemma

        return None

    word = word.lower()
    lemma = aux(word, lexica.verbs, lexica.verbs_irregular, lexica.verbs_rules)

    if lemma is None:
        lemma = aux(word, lexica.nouns, lexica.nouns_irregular, lexica.nouns_rules)

    return lemma if lemma else word

• L2: Define a nested function aux to handle lemmatization.

• L3-4: Check if the word is in the irregular dictionary (get()), if so, return its lemma.

• L6-7: Try applying each rule in the rules list to word.

• L8: If the resulting lemma is in the vocabulary, return it.

• L10: If no lemma is found, return None.

• L12: Convert the input word to lowercase for case-insensitive processing.

• L13: Try to lemmatize the word using verb-related lexica.

• L15-16: If no lemma is found among verbs, try to lemmatize using noun-related lexica.

• L18: Return the lemma if found or the decapitalized word if no lemmatization occurred.

We now test our lemmatizer for nouns and verbs:

Code

nouns = ['studies', 'crosses', 'areas', 'gentlemen', 'vertebrae', 'alumni', 'children', 'crises']
nouns_lemmatized = [lemmatize(lexica, word) for word in nouns]
for word, lemma in zip(nouns, nouns_lemmatized): print('{} -> {}'.format(word, lemma))

verbs = ['applies', 'cried', 'pushes', 'entered', 'takes', 'heard', 'lying', 'studying', 'taking', 'drawn', 'clung', 'was', 'bought']
verbs_lemmatized = [lemmatize(lexica, word) for word in verbs]
for word, lemma in zip(verbs, verbs_lemmatized): print('{} -> {}'.format(word, lemma))

Nouns

Verbs

Finally, let us recount word types in dat/emory-wiki.txt using the lemmatizer and save them to dat/word_types-token-lemma.txt:

Code

from collections import Counter
from src.tokenization import tokenize

corpus = 'dat/emory-wiki.txt'
delims = {'"', "'", '(', ')', '[', ']', ':', '-', ',', '.'}
words = [lemmatize(lexica, word) for word in tokenize(corpus, delims)]
counts = Counter(words)

print(f'# of word tokens: {len(words)}')
print(f'# of word types: {len(counts)}')

output = 'dat/word_types-token-lemma.txt'
with open(output, 'w') as fout:
    for key in sorted(counts.keys()): fout.write(f'{key}\n')

• L2: Import the tokenize() function from the src/tokenization.py module.

Output

When the words are further normalized by lemmatization, the number of word tokens remains the same as without lemmatization, but the number of word types is reduced from 197 to 177.

Q9: In which tasks can lemmatization negatively impact performance?

References

1. Source: lemmatization.py

2. ELIT Morphological Analyzer - A heuristic-based lemmatizer

3. An Algorithm for Suffix Stripping, Porter, Program: Electronic Library and Information Systems, 14(3), 1980 (PDF)