Self-attention, also known as scaled dot-product attention, is a fundamental mechanism used in deep learning and natural language processing, particularly in transformer-based models like BERT, GPT, and their variants. Self-attention is a crucial component that enables these models to understand relationships and dependencies between words or tokens in a sequence.

Here's an overview of self-attention:

1. The Motivation:

   • The primary motivation behind self-attention is to capture dependencies and relationships between different elements within a sequence, such as words in a sentence or tokens in a document.

   • It allows the model to consider the context of each element based on its relationships with other elements in the sequence.

2. The Mechanism:

   • Self-attention computes a weighted sum of the input elements (usually vectors) for each element in the sequence. This means that each element can attend to and be influenced by all other elements.

   • The key idea is to learn weights (attention scores) that reflect how much focus each element should give to the others. These weights are often referred to as "attention weights."

3. Attention Weights:

   • Attention weights are calculated using a similarity measure (typically the dot product) between a query vector and a set of key vectors.

   • The resulting attention weights are then used to take a weighted sum of the value vectors. This weighted sum forms the output for each element.

4. Scaling and Softmax:

   • To stabilize the gradients during training, the dot products are often scaled by the square root of the dimension of the key vectors.

   • After scaling, a softmax function is applied to obtain the attention weights. The softmax ensures that the weights are normalized and sum to 1.

5. Multi-Head Attention:

   • Many models use multi-head attention, where multiple sets of queries, keys, and values are learned. Each set of attention weights captures different aspects of relationships in the sequence.

   • These multiple sets of attention results are concatenated and linearly transformed to obtain the final output.

6. Applications:

   • Self-attention is widely used in transformer-based models for various NLP tasks, including machine translation, text classification, text generation, and more.

   • It is also applied in computer vision tasks, such as image captioning, where it can capture relationships between different parts of an image.

Self-attention is a powerful mechanism because it allows the model to focus on different elements of the input sequence depending on the context. This enables the model to capture long-range dependencies, word relationships, and nuances in natural language, making it a crucial innovation in deep learning for NLP and related fields.

Multi-head attention is a crucial component in transformer-based models, such as BERT, GPT, and their variants. It extends the basic self-attention mechanism to capture different types of relationships and dependencies in a sequence. Here's an explanation of multi-head attention:

1. Motivation:

   • The primary motivation behind multi-head attention is to enable a model to focus on different parts of the input sequence when capturing dependencies and relationships.

   • It allows the model to learn multiple sets of attention patterns, each suited to capturing different kinds of associations in the data.

2. Mechanism:

   • In multi-head attention, the input sequence (e.g., a sentence or document) is processed by multiple "attention heads."

   • Each attention head independently computes attention scores and weighted sums for the input sequence, resulting in multiple sets of output values.

   • These output values from each attention head are then concatenated and linearly transformed to obtain the final multi-head attention output.

3. Learning Different Dependencies:

   • Each attention head can learn to attend to different aspects of the input sequence. For instance, one head may focus on syntactic relationships, another on semantic relationships, and a third on longer-range dependencies.

   • By having multiple heads, the model can learn to capture a variety of dependencies, making it more versatile and robust.

4. Multi-Head Processing:

   • In each attention head, there are three main components: queries, keys, and values. These are linearly transformed projections of the input data.

   • For each head, queries are compared to keys to compute attention weights, which are then used to weight the values.

   • Each attention head performs these calculations independently, allowing it to learn a unique set of attention patterns.

5. Concatenation and Linear Transformation:

   • The output values from each attention head are concatenated into a single tensor.

   • A linear transformation is applied to this concatenated output to obtain the final multi-head attention result. The linear transformation helps the model combine information from all heads appropriately.

6. Applications:

   • Multi-head attention is widely used in NLP tasks, such as text classification, machine translation, and text generation.

   • It allows models to capture diverse dependencies and relationships within text data, making it highly effective in understanding and generating natural language.

Multi-head attention has proven to be a powerful tool in transformer architectures, enabling models to handle complex and nuanced relationships within sequences effectively. It contributes to the remarkable success of transformer-based models in a wide range of NLP tasks.

References

1. Attention is All you Need, Vaswani et al., NIPS 2017.

Quiz

1. The EM algorithm stands as a classic method in unsupervised learning. What are the advantages of unsupervised learning over supervised learning, and which tasks align well with unsupervised learning?

2. What are the disadvantages of using BPE-based tokenization instead of rule-based tokenization? What are the potential issues with the implementation of BPE above?

3. How does self-attention operate given an embedding matrix $\mathrm{W} \in \mathbb{R}^{n \times d}$ representing a document, where $n$ is the number of words and $d$ is the embedding dimension?

4. Given the same embedding matrix as in question #3, how does multi-head attention function? What advantages does multi-head attention offer over self-attention?

5. What are the outputs of each layer in the Transformer model? How do the embeddings learned in the upper layers of the Transformer differ from those in the lower layers?

6. How is a Masked Language Model used in training a language model with a transformer?

7. How can one train a document-level embedding using a transformer?

8. What are the advantages of embeddings generated by transformers compared to those generated by Word2Vec?

9. Neural networks gained widespread popularity for training natural language processing models since 2013. What factors enabled this popularity, and how do they differ from traditional NLP methods?

10. Recent large language models like ChatGPT or Claude are trained quite differently from traditional NLP models. What are the main differences, and what factors enabled their development?

References

• Attention is All you Need, Vaswani et al., NIPS 2017.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al., NAACL 2019.

Byte Pair Encoding (BPE) is a data compression algorithm that is commonly used in the context of subword tokenization for . BPE text into smaller units, such as subword pieces or characters, to handle out-of-vocabulary words, reduce vocabulary size, and enhance the efficiency of language models.

Algorithm

The following describes the steps of BPE in terms of the :

1. Initialization: Given a dictionary consisting of all words and their counts in a corpus, the symbol vocabulary is initialized by tokenizing each word into its most basic subword units, such as characters.

2. Expectation: With the (updated) symbol vocabulary, it calculates the frequency of every symbol pair within the vocabulary.

3. Maximization: Given all symbol pairs and their frequencies, it merges the top-k most frequent symbol pairs in the vocabulary.

4. Steps 2 and 3 are repeated until meaningful sets of subwords are found for all words in the corpus.

Implementation

Let us consider a toy vocabulary:

First, we create the symbol vocabulary by inserting a space between every pair of adjacent characters and adding a special symbol [EoW] at the end to indicate the End of the Word:

Next, we count the frequencies of all symbol pairs in the vocabulary:

Finally, we update the vocabulary by merging the most frequent symbol pair across all words:

The expect() and maximize() can be repeated for multiple iterations until the tokenization becomes reasonable:

When you uncomment L7 in bpe_vocab(), you can see how the symbols are merged in each iteration:

References

Contextual representations are representations of words, phrases, or sentences within the context of the surrounding text. Unlike word embeddings from Word2Vec where each word is represented by a fixed vector regardless of its context, contextual representations capture the meaning of a word or sequence of words based on their context in a particular document such that the representation of a word can vary depending on the words surrounding it, allowing for a more nuanced understanding of meaning in natural language processing tasks.

References

• Attention is All You Need, Vaswani et al., Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.