Smoothing

Unigram Smoothing

The unigram model in the previous section faces a challenge when confronted with words that do not occur in the corpus, resulting in a probability of 0. One common technique to address this challenge is smoothing, which tackles issues such as zero probabilities, data sparsity, and overfitting that emerge during probability estimation and predictive modeling with limited data.

Laplace smoothing (aka. add-one smoothing) is a simple yet effective technique that avoids zero probabilities and distributes the probability mass more evenly. It adds the count of 1 to every word and recalculates the unigram probabilities:

Thus, the probability of any unknown word with Laplace smoothing is calculated as follows:

The unigram probability of an unknown word is guaranteed to be lower than the unigram probabilities of any known words, whose counts have been adjusted to be greater than 1.

Note that the sum of all unigram probabilities adjusted by Laplace smoothing is still 1:

Let us define a function unigram_smoothing() that takes a file path and returns a dictionary with bigrams and their probabilities as keys and values, respectively, estimated by Laplace smoothing:

• L1: Import the unigram_count() function from the package.

• L3: Define a constant representing the unknown word.

• L7: Increment the total count by the vocabulary size.

We then test unigram_smoothing() with a text file :

Compared to the unigram results without smoothing (see the "Comparison" tab above), the probabilities for these top unigrams have slightly decreased.

The unigram probability of any word (including unknown) can be retrieved using the UNKNOWN key:

• L2: Use to retrieve the probability of the target word from probs. If the word is not present, default to the probability of the UNKNOWN token.

Bigram Smoothing

The bigram model can also be enhanced by applying Laplace smoothing:

Thus, the probability of an unknown bigram where is known but is unknown is calculated as follows:

Let us define a function bigram_smoothing() that takes a file path and returns a dictionary with unigrams and their probabilities as keys and values, respectively, estimated by Laplace smoothing:

• L1: Import the bigram_count() function from the package.

• L5: Create a set vocab containing all unique .

• L6-7: Add all unique to vocab

We then test bigram_smoothing() with the same text file:

Finally, we test the bigram estimation using smoothing for unknown sequences:

• L2: Retrieve the bigram probabilities of the previous word, or set it to None if not present.

• L3: Return the probability of the current word given the previous word with smoothing. If the previous word is not present, return the probability for an unknown previous word.

Normalization

Unlike the unigram case, the sum of all bigram probabilities adjusted by Laplace smoothing given a word is not guaranteed to be 1. To illustrate this point, let us consider the following corpus comprising only two sentences:

There are seven word types in this corpus, {"I", "You", "a", "and", "are", "student", "students"}, such that . Before Laplace smoothing, the bigram probabilities of are estimated as follows:

However, after applying Laplace smoothing, the bigram probabilities undergo significant changes, and their sum no longer equals 1:

The bigram distribution for can be normalized to 1 by adding the total number of word types occurring after , denoted as , to the denominator instead of :

Consequently, the probability of an unknown bigram can be calculated with the normalization as follows:

For the above example, . Once you apply to , the sum of its bigram probabilities becomes 1:

A major drawback of this normalization is that the probability cannot be measured when is unknown. Thus, we assign the minimum unknown probability across all bigrams as the bigram probability of , where the previous word is unknown, as follows:

Reference

1. Source:

2. , Wikipedia