Attention in Natural Language Processing (NLP) refers to a mechanism or technique that allows models to focus on specific parts of the input data while making predictions or generating output. It's a crucial component in many modern NLP models, especially in sequence-to-sequence tasks and transformer architectures. Attention mechanisms help the model assign different weights to different elements of the input sequence, allowing it to pay more attention to relevant information and ignore irrelevant or less important parts.

• Neural Machine Translation by Jointly Learning to Align and Translate

Key points about attention in NLP:

1. Contextual Focus: Attention enables the model to focus on the most relevant parts of the input sequence for each step of the output sequence. It creates a dynamic and contextually adaptive way of processing data.

2. Weighted Information: Each element in the input sequence is associated with a weight or attention score, which determines its importance when generating the output. Elements with higher attention scores have a stronger influence on the model's predictions.

3. Self-Attention: Self-attention mechanisms allow a model to consider all elements of the input sequence when making predictions, and it learns to assign different attention weights to each element based on its relevance.

4. Multi-Head Attention: Many NLP models use multi-head attention, which allows the model to focus on different aspects of the input simultaneously. This can improve the capture of various patterns and dependencies.

5. Transformer Architecture: Attention mechanisms are a fundamental component of the transformer architecture, which has been highly influential in NLP. Transformers use self-attention to process sequences, enabling them to capture long-range dependencies and context.

Applications of attention mechanisms in NLP include:

• Machine Translation: Attention helps the model align words in the source language with words in the target language.

• Text Summarization: Attention identifies which parts of the source text are most important for generating a concise summary.

• Question Answering: It helps the model find the most relevant parts of a passage to answer a question.

• Named Entity Recognition: Attention can be used to focus on specific words or subwords to identify named entities.

• Language Modeling: In tasks like text generation, attention helps the model decide which words or tokens to generate next based on the context.

Attention mechanisms have revolutionized the field of NLP by allowing models to handle complex and long sequences effectively, making them suitable for a wide range of natural language understanding and generation tasks.

Pointer Networks are a type of neural network architecture designed to handle tasks that involve selecting elements from an input sequence and generating output sequences that reference those elements. These networks are particularly useful when the output sequence is conditioned on the content of the input sequence, and the order or content of the output sequence can vary dynamically based on the input.

Key features of Pointer Networks:

1. Input Sequence Reference: In tasks that involve sequences, Pointer Networks learn to refer to specific elements from the input sequence. This is particularly valuable in problems like content selection or summarization, where elements from the input sequence are selectively copied to the output sequence.

2. Variable-Length Output: Pointer Networks are flexible in generating output sequences of variable lengths, as the length and content of the output sequence can depend on the input. This is in contrast to fixed-length output sequences common in many other sequence-to-sequence tasks.

3. Attention Mechanism: Attention mechanisms are a fundamental part of Pointer Networks. They allow the model to assign different weights or probabilities to elements in the input sequence, indicating which elements should be referenced in the output.

Applications of Pointer Networks include:

• Content Selection: Selecting and copying specific elements from the input sequence to generate the output. This is useful in tasks like text summarization, where relevant sentences or phrases from the input are selectively included in the summary.

• Entity Recognition: Identifying and referencing named entities from the input text in the output sequence. This is valuable for named entity recognition tasks in information extraction.

• Geographic Location Prediction: Predicting geographic locations mentioned in text and generating a sequence of location references.

Pointer Networks have proven to be effective in tasks that involve content selection and variable-length output generation, addressing challenges that traditional sequence-to-sequence models with fixed vocabularies may encounter. They provide a way to dynamically handle content in natural language processing tasks where the input-output relationship can be complex and context-dependent.

Sequence Modeling focuses on comprehending and processing data sequences (e.g., sequences of words). Given the inherently sequential nature of language, understanding the relationships within the sequence is essential for meaningful text interpretation and generation.

Contents

• Recurrent Neural Networks

• Encoder-Decoder Framework

• Attentions

A Recurrent Neural Network (RNN) [1] maintains hidden states of previous inputs and uses them to predict outputs, allowing it to model temporal dependencies in sequential data.

The hidden state is a vector representing the network's internal memory of the previous time step. It captures information from previous time steps and influences the predictions made at the current time step, often updated at each time step as the RNN processes a sequence of inputs.

RNN for Sequence Tagging

Given an input sequence $X = [x_1, \ldots, x_n]$ where $x_i \in \mathbb{R}^{d \times 1}$, an RNN for sequence tagging defines two functions, $f$ and $g$:

• $f$ takes the current input $x_i \in X$ and the hidden state $h_{i-1}$ of the previous input $x_{i-1}$, and returns a hidden state $h_i \in \mathbb{R}^{e \times 1}$ such that $f(x_i, h_{i-1}) = \alpha(W^x x_i + W^h h_{i-1}) = h_i$, where $W^x \in \mathbb{R}^{e \times d}$, $W^h \in \mathbb{R}^{e \times e}$, and $\alpha$ is an activation function.

• $g$ takes the hidden state $h_i$ and returns an output $y_i \in \mathbb{R}^{o \times 1}$ such that $g(h_i) = W^o h_i = y_i$, where $W^o \in \mathbb{R}^{o \times e}$.

Figure 1 shows an example of an RNN for sequence tagging, such as part-of-speech tagging:

RNN for Text Classification

Bidirectional RNN

The RNN for sequence tagging above does not consider the words that follow the current word when predicting the output. This limitation can significantly impact model performance since contextual information following the current word can be crucial.

For example, let us consider the word "early" in the following two sentences:

• They are early birds -> "early" is an adjective.

• They are early today -> "early" is an adverb.

The POS tags of "early" depend on the following words, "birds" and "today", such that making the correct predictions becomes challenging without the following context.

To overcome this challenge, a Bidirectional RNN is suggested [2] that considers both forward and backward directions, creating twice as many hidden states to capture a more comprehensive context. Figure 3 illustrates a bidirectional RNN for sequence tagging:

Advanced Topics

• Long Short-Term Memory (LSTM) Networks [3-5]

• Gated Recurrent Units (GRUs) [6-7]

References

1. Finding Structure in Time, Elman, Cognitive Science, 14(2), 1990.

2. Bidirectional Recurrent Neural Networks, Schuster and Paliwal, IEEE Transactions on Signal Processing, 45(11), 1997.

3. Long Short-Term Memory, Hochreiter and Schmidhuber, Neural Computation, 9(8), 1997 (PDF available at ResearchGate).

4. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF, Ma and Hovy, ACL, 2016.*

5. Contextual String Embeddings for Sequence Labeling, Akbik et al., COLING, 2018.*

6. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, Cho et al., EMNLP, 2014.*

7. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Chung et al., NeurIPS Workshop on Deep Learning and Representation Learning, 2014.*

The Encoder-Decoder Framework is commonly used for solving sequence-to-sequence tasks, where it takes an input sequence, processes it through an encoder, and produces an output sequence. This framework consists of three main components: an encoder, a context vector, and a decoder, as illustrated in Figure 1:

Encoder

An encoder processes an input sequence and creates a context vector that captures context from the entire sequence and serves as a summary of the input.

Let $X =[x_1, \ldots, x_n,x_c]$ be an input sequence, where $x_i \in \mathbb{R}^{d \times 1}$ is the $i$'th word in the sequence and $x_c$is an artificial token appended to indicate the end of the sequence. The encoder utilizes two functions, $f$ and $g$, which are defined in the same way as in the RNN for text classification. Notice that the end-of-sequence token $x_c$ is used to create an additional hidden state $h_c$, which in turn creates the context vector $y_c$.

Figure 2 shows an encoder example that takes the input sequence, "I am a boy", appended with the end-of-sequence token "[EOS]":

Decoder

A decoder is conditioned on the context vector, which allows it to generate an output sequence contextually relevant to the input, often one token at a time.

Let $Y = [y_1, \ldots, y_m, y_{t}]$ be an output sequence, where $y_i \in \mathbb{R}^{o \times 1}$ is the $i$'th word in the sequence, and $y_t$ is an artificial token to indicate the end of the sequence. To generate the output sequence, the decoder defines two functions, $f$ and $g$ :

• $f$ takes the previous output $y_{i-1}$ and its hidden state $s_{i-1}$, and returns a hidden state $s_{i} \in \mathbb{R}^{e \times 1}$ such that $f(y_{i-1}, s_{i-1}) = \alpha(W^y y_{i-1}​ + W^s s_ {i−1}) = s_i$, where $W^y \in \mathbb{R}^{e \times o}$, $W^s \in \mathbb{R}^{e \times e}$, and $\alpha$ is an activation function.

• $g$ takes the hidden state $s_i$ and returns an output $y_{i} \in \mathbb{R}^{o \times 1}$ such that $g(s_i) = W^o s_i = y_{i}$, where $W^o \in \mathbb{R}^{o \times e}$.

Note that the initial hidden state $s_1$ is created by considering only the context vector $y_c$such that the first output $y_1$ is solely predicted by the context in the input sequence. However, the prediction of every subsequent output $y_{i}$ is conditioned on both the previous output $y_{i-1}$ and its hidden state $s_{i-1}$. Finally, the decoder stops generating output when it predicts the end-of-sequence token $y_t$.

Figure 3 illustrates a decoder example that takes the context vector $y_0$ and generates the output sequence, "나(I) +는(SBJ) 소년(boy) +이다(am)", terminated by the end-of-sequence token "[EOS]", which translates the input sequence from English to Korean:

The likelihood of the current output $y_i$ can be calculated as:

where $q$ is a function that takes the context vector $y_c$, previous input $y_{i-1}$ and its hidden state $s_{i-1}$, and returns the probability of $y_i$. Then, the maximum likelihood of the output sequence can be estimated as follows ($y_0 = s_0 = \mathbf{0}$):

References

1. Sequence to Sequence Learning with Neural Networks, Sutskever et al., NeurIPS, 2014.*

2. Neural Machine Translation by Jointly Learning to Align and Translate, Bahdanau et al., ICLR, 2015.*