Sometimes, it is more appropriate to consider the canonical forms as tokens instead of their variations. For example, if you want to analyze the usage of the word "transformer" in NLP literature for each year, you want to count both "transformer" and 'transformers' as a single item.

Lemmatization is a task that simplifies words into their base or dictionary forms, known as lemmas, to simplify the interpretation of their core meaning.

When analyzing obtained by the tokenizer in the previous section, the following tokens are recognized as separate word types:

• Universities

• University

• universities

Two variations are applied to the noun "university" - capitalization, generally used for proper nouns or initial words, and pluralization, which indicates multiple instances of the term. On the other hand, verbs can also take several variations regarding tense and aspect:

• study

• studies

• studied

We want to develop a lemmatizer that normalizes all variations into their respective lemmas.

Lemma Lexica

Let us create lexica for lemmatization:

• L1:

• L2:

• L3:

• L1: res_dir: the path to the root directory where all lexica files are located.

• L2: : a list of base nouns

• L3: : a list of base verbs

We then verify that all lexical resources are loaded correctly:

Lemmatizing

Let us write the lemmatize() function that takes a word and lemmatizes it using the lexica:

• L2: Define a nested function aux to handle lemmatization.

• L3-4: Check if the word is in the irregular dictionary (), if so, return its lemma.

• L6-7: Try applying each rule in the rules list to word

We now test our lemmatizer for nouns and verbs:

Finally, let us recount word types in using the lemmatizer and save them to :

When the words are further normalized by lemmatization, the number of word tokens remains the same as without lemmatization, but the number of word types is reduced from 197 to 177.

References

1. Source:

2. - A heuristic-based lemmatizer

3. , Porter, Program: Electronic Library and Information Systems, 14(3), 1980 ()

Frequency Analysis examines how often each word appears in a corpus. It helps understand language patterns and structure by measuring how often words appear in text.

Word Counting

Consider the following text from Wikipedia about (as of 2023-10-18):

Regular expressions, commonly abbreviated as regex, form a language for string matching, enabling operations to search, match, and manipulate text based on specific patterns or rules.

• Online Interpreter: Regular Expressions 101

Core Syntax

Metacharacters

Regex provides metacharacters with specific meanings, making it convenient to define patterns:

• .: any single character except a newline character

  e.g., M.\. matches "Mr." and "Ms.", but not "Mrs." (\ escapes the metacharacter .).

• [ ]

Repetitions

Repetitions allow you to define complex patterns that can match multiple occurrences of a character or group of characters:

• *: the preceding character or group appears zero or more times

  e.g., \d* matches "90" in "90s" as well as "" (empty string) in "ABC".

• +: the preceding character or group appears one or more times

Groupings

Grouping allows you to treat multiple characters, subpatterns, or metacharacters as a single unit. It is achieved by placing these characters within parentheses ( and ).

• |: a logical OR, referred to as a "pipe" symbol, allowing you to specify alternatives

  e.g., (cat|dog) matches either "cat" or "dog".

• ( ): a capturing group; any text that matches the parenthesized pattern is "captured" and can be extracted or used in various ways

Assertions

Assertions define conditions that must be met for a match to occur. They do not consume characters in the input text but specify the position where a match should happen based on specific criteria.

• A positive lookahead assertion (?= ) checks that a specific pattern is present immediately after the current position

  e.g., apple(?=[ -]pie) matches "apple" in "apple pie" or "apple-pie", but not in "apple juice".

• A negative lookahead assertion (?! ) checks that a specific pattern is not present immediately after the current position

Functions

Python provides several functions to make use of regular expressions.

match()

Let us create a regular expression that matches "Mr." and "Ms.":

group()

Currently, no group has been specified for re_mr:

Let us capture the letters and the period as separate groups:

If the pattern does not find a match, it returns None.

search()

Let us match the following strings with re_mr:

s1 matches "Mr." but not "Ms." while s2 does not match any pattern. It is because the match() function matches patterns only at the beginning of the string. To match patterns anywhere in the string, we need to use search() instead:

findall()

The search() function matches "Mr." in both s1 and s2 but still does not match "Ms.". To match them all, we need to use the findall() function:

finditer()

While the findall() function matches all occurrences of the pattern, it does not provide a way to locate the positions of the matched results in the string. To find the locations of the matched results, we need to use the finditer() function:

sub()

Finally, you can replace the matched results with another string by using the sub() function:

Tokenization

Finally, let us write a simple tokenizer using regular expressions. We will define a regular expression that matches the necessary patterns for tokenization:

• L2: Create a regular expression to match delimiters and a special case:

  • Delimiters: ',', '.', or whitespaces ('\s+').

Test cases for the tokenizer:

References

1. Source:

2. , Kuchling, HOWTOs in Python Documentation

Text processing refers to the manipulation and analysis of textual data through techniques applied to raw text, making it more structured, understandable, and suitable for various applications.

Sections

• Frequency Analysis

Task 1: Chronicles of Narnia

Your goal is to extract and organize structured information from C.S. Lewis's Chronicles of Narnia series, focusing on book metadata and chapter statistics.

Data Collection

For each book, gather the following details:

• Book Title (preserve exact spacing as shown in text)

• Year of Publishing (indicated in the title)

For each chapter within every book, collect the following information:

• Chapter Number (as Arabic numeral)

• Chapter Title

• Token Count of the Chapter Content

Implementation

1. Download the file and place it under the directory.

   • The text file is pre-tokenized using the .

   • Each token is separated by whitespace.

Task 2: Regular Expressions

Define a function named regular_expressions() in src/homework/text_processing.py that takes a string and returns one the four types, "email", "date", "url", "cite", or None if nothing matches:

Submission

Commit and push the text_processing.py file to your GitHub repository.

Rubric

• Task 1: Chronicles of Narnia (7 points)

• Task 2: Regular Expressions (3 points)

• Concept Quiz (2 points)

Tokenization is the process of breaking down a text into smaller units, typically words or subwords, known as tokens. Tokens serve as the basic building blocks used for a specific task.

When examining the dat/word_types.txt from the previous section, you notice several words that need further tokenization, where many of them can be resolved by leveraging punctuation:

• "R1: → ['"', "R1", ":"]

• (R&D) → ['(', 'R&D', ')']

• 15th-largest → ['15th', '-', 'largest']

• Atlanta, → ['Atlanta', ',']

• Department's → ['Department', "'s"]

• activity"[26] → ['activity', '"', '[26]']

• centers.[21][22] → ['centers', '.', '[21]', '[22]']

Delimiters

Let us write the delimit() function that takes a word and a set of delimiters, and returns a list of tokens by splitting the word using the delimiters:

• L1: .

• L2: Find the index of the first character in word that is in a of delimiters (, ). If no delimiter is found in word, return -1 ().

Let us define a set of delimiters and test delimit() using various input:

Post-Processing

When reviewing the output of delimit(), the first four test cases yield accurate results, while the last five are not handled properly, which should have been tokenized as follows:

• Department's → ['Department', "'s"]

• activity"[26] → ['activity', '"', '[26]']

• centers.[21][22] → ['centers', '.', '[21]', '[22]']

To handle these special cases, let us post-process the tokens generated by delimit():

• L2: Initialize variables i for the current position and new_tokens for the resulting tokens.

• L4: Iterate through the input tokens.

• L5: Case 1: Handling apostrophes for contractions like "'s" (e.g., it's).

Once the post-processing is applied, all outputs are now handled properly:

Tokenizing

Finally, let us write tokenize() that takes a path to a corpus and a set of delimiters, and returns a list of tokens from the corpus:

• L2: Read the corpus file.

• L3: Split the text into words.

• L4: Tokenize each word in the corpus using the specified delimiters. postprocess() is used to process the special cases further. The resulting tokens are collected in a list and returned ().

Given the new tokenizer, let us recount word types in the corpus, , and save them:

Compared to the original tokenization, where all words are split solely by whitespaces, the more advanced tokenizer increases the number of word tokens from 305 to 363 and the number of word types from 180 to 197 because all punctuation symbols, as well as reference numbers, are now introduced as individual tokens.

References

1. Source:

2. - a heuristic-based tokenizer