Self Attention

Self-attention, also known as scaled dot-product attention, is a fundamental mechanism used in deep learning and natural language processing, particularly in transformer-based models like BERT, GPT, and their variants. Self-attention is a crucial component that enables these models to understand relationships and dependencies between words or tokens in a sequence.

Here's an overview of self-attention:

1. The Motivation:

   • The primary motivation behind self-attention is to capture dependencies and relationships between different elements within a sequence, such as words in a sentence or tokens in a document.

   • It allows the model to consider the context of each element based on its relationships with other elements in the sequence.

2. The Mechanism:

   • Self-attention computes a weighted sum of the input elements (usually vectors) for each element in the sequence. This means that each element can attend to and be influenced by all other elements.

   • The key idea is to learn weights (attention scores) that reflect how much focus each element should give to the others. These weights are often referred to as "attention weights."

3. Attention Weights:

   • Attention weights are calculated using a similarity measure (typically the dot product) between a query vector and a set of key vectors.

   • The resulting attention weights are then used to take a weighted sum of the value vectors. This weighted sum forms the output for each element.

4. Scaling and Softmax:

   • To stabilize the gradients during training, the dot products are often scaled by the square root of the dimension of the key vectors.

   • After scaling, a softmax function is applied to obtain the attention weights. The softmax ensures that the weights are normalized and sum to 1.

5. Multi-Head Attention:

   • Many models use multi-head attention, where multiple sets of queries, keys, and values are learned. Each set of attention weights captures different aspects of relationships in the sequence.

   • These multiple sets of attention results are concatenated and linearly transformed to obtain the final output.

6. Applications:

   • Self-attention is widely used in transformer-based models for various NLP tasks, including machine translation, text classification, text generation, and more.

   • It is also applied in computer vision tasks, such as image captioning, where it can capture relationships between different parts of an image.

Self-attention is a powerful mechanism because it allows the model to focus on different elements of the input sequence depending on the context. This enables the model to capture long-range dependencies, word relationships, and nuances in natural language, making it a crucial innovation in deep learning for NLP and related fields.

Multi-head Attention

Multi-head attention is a crucial component in transformer-based models, such as BERT, GPT, and their variants. It extends the basic self-attention mechanism to capture different types of relationships and dependencies in a sequence. Here's an explanation of multi-head attention:

1. Motivation:

   • The primary motivation behind multi-head attention is to enable a model to focus on different parts of the input sequence when capturing dependencies and relationships.

   • It allows the model to learn multiple sets of attention patterns, each suited to capturing different kinds of associations in the data.

Multi-head attention has proven to be a powerful tool in transformer architectures, enabling models to handle complex and nuanced relationships within sequences effectively. It contributes to the remarkable success of transformer-based models in a wide range of NLP tasks.

Transformer Architecture

BERT

References

1. , Vaswani et al., NIPS 2017.

2. , Devlin et al., NAACL, 2019.

A Recurrent Neural Network (RNN) [1] maintains hidden states of previous inputs and uses them to predict outputs, allowing it to model temporal dependencies in sequential data.

The hidden state is a vector representing the network's internal memory of the previous time step. It captures information from previous time steps and influences the predictions made at the current time step, often updated at each time step as the RNN processes a sequence of inputs.

RNN for Sequence Tagging

Given an input sequence where , an RNN for defines two functions, and :

• takes the current input and the hidden state of the previous input , and returns a hidden state such that , where , , and is an .

• takes the hidden state and returns an output such that , where .

Figure 1 shows an example of an RNN for sequence tagging, such as :

Notice that the output for the first input is predicted by considering only the input itself such that (e.g., the POS tag of the first word "I" is predicted solely using that word). However, the output for every other input is predicted by considering both and , an intermediate representation created explicitly for the task. This enables RNNs to capture sequential information that cannot.

RNN for Text Classification

Unlike sequence tagging where the RNN predicts a sequence of output for the input , an RNN designed for predicts only one output for the entire input sequence such that:

• Sequence Tagging

• Text Classification:

To accomplish this, a common practice is to predict the output from the last hidden state using the function . Figure 2 shows an example of an RNN for text classification, such as :

Bidirectional RNN

The above does not consider the words that follow the current word when predicting the output. This limitation can significantly impact model performance since contextual information following the current word can be crucial.

For example, let us consider the word "early" in the following two sentences:

• They are early birds -> "early" is an adjective.

• They are early today -> "early" is an adverb.

The POS tags of "early" depend on the following words, "birds" and "today", such that making the correct predictions becomes challenging without the following context.

To overcome this challenge, a Bidirectional RNN is suggested [2] that considers both forward and backward directions, creating twice as many hidden states to capture a more comprehensive context. Figure 3 illustrates a bidirectional RNN for sequence tagging:

For every , the hidden states and are created by considering and , respectively. The function takes both and and returns an output such that , where is a concatenation of the two hidden states and .

Advanced Topics

• Long Short-Term Memory (LSTM) Networks [3-5]

• Gated Recurrent Units (GRUs) [6-7]

References

1. , Elman, Cognitive Science, 14(2), 1990.

2. , Schuster and Paliwal, IEEE Transactions on Signal Processing, 45(11), 1997.

3. , Hochreiter and Schmidhuber, Neural Computation, 9(8), 1997 ( available at ResearchGate).

Contextual representations are representations of words, phrases, or sentences within the context of the surrounding text. Unlike word embeddings from where each word is represented by a fixed vector regardless of its context, contextual representations capture the meaning of a word or sequence of words based on their context in a particular document such that the representation of a word can vary depending on the words surrounding it, allowing for a more nuanced understanding of meaning in natural language processing tasks.

Contents

Quiz

1. The EM algorithm stands as a classic method in unsupervised learning. What are the advantages of unsupervised learning over supervised learning, and which tasks align well with unsupervised learning?

2. What are the disadvantages of using BPE-based tokenization instead of ? What are the potential issues with the implementation of BPE above?

3. How does self-attention operate given an embedding matrix representing a document, where is the number of words and is the embedding dimension?

4. Given the same embedding matrix as in question #3, how does multi-head attention function? What advantages does multi-head attention offer over self-attention?

5. What are the outputs of each layer in the Transformer model? How do the embeddings learned in the upper layers of the Transformer differ from those in the lower layers?

6. How is a Masked Language Model used in training a language model with a transformer?

7. How can one train a document-level embedding using a transformer?

8. What are the advantages of embeddings generated by transformers compared to those generated by ?

9. Neural networks gained widespread popularity for training natural language processing models since 2013. What factors enabled this popularity, and how do they differ from traditional NLP methods?

10. Recent large language models like ChatGPT or Claude are trained quite differently from traditional NLP models. What are the main differences, and what factors enabled their development?

References

• , Vaswani et al., NIPS 2017.

• , Devlin et al., NAACL 2019.

The Encoder-Decoder Framework is commonly used for solving sequence-to-sequence tasks, where it takes an input sequence, processes it through an encoder, and produces an output sequence. This framework consists of three main components: an encoder, a context vector, and a decoder, as illustrated in Figure 1:

Encoder

An encoder processes an input sequence and creates a context vector that captures context from the entire sequence and serves as a summary of the input.

Let be an input sequence, where is the 'th word in the sequence and is an artificial token appended to indicate the end of the sequence. The encoder utilizes two functions, and , which are defined in the same way as in the . Notice that the end-of-sequence token is used to create an additional hidden state , which in turn creates the context vector .

Figure 2 shows an encoder example that takes the input sequence, "I am a boy", appended with the end-of-sequence token "[EOS]":

Decoder

A decoder is conditioned on the context vector, which allows it to generate an output sequence contextually relevant to the input, often one token at a time.

Let be an output sequence, where is the 'th word in the sequence, and is an artificial token to indicate the end of the sequence. To generate the output sequence, the decoder defines two functions, and :

• takes the previous output and its hidden state , and returns a hidden state such that , where , , and is an activation function.

• takes the hidden state and returns an output such that , where .

Note that the initial hidden state is created by considering only the context vector such that the first output is solely predicted by the context in the input sequence. However, the prediction of every subsequent output is conditioned on both the previous output and its hidden state . Finally, the decoder stops generating output when it predicts the end-of-sequence token .

Figure 3 illustrates a decoder example that takes the context vector and generates the output sequence, "나(I) +는(SBJ) 소년(boy) +이다(am)", terminated by the end-of-sequence token "[EOS]", which translates the input sequence from English to Korean:

The likelihood of the current output can be calculated as:

where is a function that takes the context vector , previous input and its hidden state , and returns the probability of . Then, the of the output sequence can be estimated as follows ():

References

1. , Sutskever et al., NeurIPS, 2014.*

2. , Bahdanau et al., ICLR, 2015.*

Byte Pair Encoding (BPE) is a data compression algorithm that is commonly used in the context of subword tokenization for neural language models. BPE tokenizes text into smaller units, such as subword pieces or characters, to handle out-of-vocabulary words, reduce vocabulary size, and enhance the efficiency of language models.

Algorithm

The following describes the steps of BPE in terms of the EM algorithm:

1. Initialization: Given a dictionary consisting of all words and their counts in a corpus, the symbol vocabulary is initialized by tokenizing each word into its most basic subword units, such as characters.

2. Expectation: With the (updated) symbol vocabulary, it calculates the frequency of every symbol pair within the vocabulary.

3. Maximization: Given all symbol pairs and their frequencies, it merges the top-k most frequent symbol pairs in the vocabulary.

4. Steps 2 and 3 are repeated until meaningful sets of subwords are found for all words in the corpus.

Implementation

Let us consider a toy vocabulary:

First, we create the symbol vocabulary by inserting a space between every pair of adjacent characters and adding a special symbol [EoW] at the end to indicate the End of the Word:

Next, we count the frequencies of all symbol pairs in the vocabulary:

Finally, we update the vocabulary by merging the most frequent symbol pair across all words:

The expect() and maximize() can be repeated for multiple iterations until the tokenization becomes reasonable:

When you uncomment L7 in bpe_vocab(), you can see how the symbols are merged in each iteration:

References

Source code:

• , Sennrich et al., ACL, 2016.

• , Gage, The C Users Journal, 1994.

• , Kudo and Richardson, EMNLP, 2018.