Word2Vec

Neural language models leverage neural networks trained on extensive text data, enabling them to discern patterns and connections between terms and documents. Through this training, neural language models gain the ability to comprehend and generate human-like language with remarkable fluency and coherence.

Word2Vec is a neural language model that maps words into a high-dimensional embedding space, positioning similar words closer to each other.

Continuous Bag-of-Words

Consider a sequence of words, $\{w_{k-2}, w_{k-1}, w_{k}, w_{k+1}, w_{k+2}\}$. We can predict $w_i$ by leveraging its contextual words using a similar to the discussed previously (: a vocabulary list comprising all unique words in the corpus):

This objective can also be achieved by using a such as Continuous Bag-of-Words (CBOW) using a . Let be an input vector, where . is created by the model on a set of context words, , such that only the dimensions of representing words in have a value of ; otherwise, they are set to .

Let be an output vector, where all dimensions have the value of except for the one representing , which is set to .

Let be a hidden layer between and and be the weight matrix between and , where the sigmoid function is used as the activation function:

Finally, let be the weight matrix between and :

Thus, each dimension in represents the probability of the corresponding word being given the set of context words .

Skip-gram

In CBOW, a word is predicted by considering its surrounding context. Another approach, known as Skip-gram, reverses the objective such that instead of predicting a word given its context, it predicts each of the context words in given . Formally, the objective of a Skip-gram model is as follows:

Let be an input vector, where only the dimension representing is set to ; all the other dimensions have the value of (thus, in Skip-gram is the same as in CBOW). Let be an output vector, where only the dimension representing is set to ; all the other dimensions have the value of . All the other components, such as the hidden layer and the weight matrices and , stay the same as the ones in CBOW.

Distributional Embeddings

What does each dimension in the hidden layer represent for CBOW? It represents a feature obtained by aggregating specific aspects from each context word in , deemed valuable for predicting the target word . Formally, each dimension is computed as the sigmoid activation of the weighted sum between the input vector and the column vector such that:

Then, what does each row vector represent? The 'th dimension in denotes the weight of the 'th feature in with respect to the 'th word in the vocabulary. In other words, it indicates the importance of the corresponding feature in representing the 'th word. Thus, can serve as an embedding for the 'th word in .

What about the other weight matrix ? The 'th column vector denotes the weights of the 'th feature in for all words in the vocabulary. Thus, the 'th dimension of indicates the importance of 'th feature for the 'th word being predicted as the target word .

On the other hand, the 'th row vector denotes the weights of all features for the 'th word in the vocabulary, enabling it to be utilized as an embedding for . However, in practice, only the row vectors of the first weight matrix are employed as word embeddings because the weights in are often optimized for the downstream task, in this case predicting , whereas the weights in are optimized for finding representations that are generalizable across various tasks.

References

1. , Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, Proceedings of the International Conference on Learning Representations (ICLR), 2013.

2. , Jeffrey Pennington, Richard Socher, Christopher Manning, Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.

3. , Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov, Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2017.