Update: 2023-10-26
A Recurrent Neural Network (RNN) [1] maintains hidden states of previous inputs and uses them to predict outputs, allowing it to model temporal dependencies in sequential data.
The hidden state is a vector representing the network's internal memory of the previous time step. It captures information from previous time steps and influences the predictions made at the current time step, often updated at each time step as the RNN processes a sequence of inputs.
Given an input sequence where , an RNN for defines two functions, and :
takes the current input and the hidden state of the previous input , and returns a hidden state such that , where , , and is an .
takes the hidden state and returns an output such that , where .
Figure 1 shows an example of an RNN for sequence tagging, such as :
Notice that the output for the first input is predicted by considering only the input itself such that (e.g., the POS tag of the first word "I" is predicted solely using that word). However, the output for every other input is predicted by considering both and , an intermediate representation created explicitly for the task. This enables RNNs to capture sequential information that cannot.
Q6: How does each hidden state in a RNN encode information relevant to sequence tagging tasks?
Unlike sequence tagging where the RNN predicts a sequence of output for the input , an RNN designed for predicts only one output for the entire input sequence such that:
Sequence Tagging
Text Classification:
To accomplish this, a common practice is to predict the output from the last hidden state using the function . Figure 2 shows an example of an RNN for text classification, such as :
Q7: In text classification tasks, what specific information is captured by the final hidden state of a RNN?
The above does not consider the words that follow the current word when predicting the output. This limitation can significantly impact model performance since contextual information following the current word can be crucial.
For example, let us consider the word "early" in the following two sentences:
They are early birds -> "early" is an adjective.
They are early today -> "early" is an adverb.
The POS tags of "early" depend on the following words, "birds" and "today", such that making the correct predictions becomes challenging without the following context.
To overcome this challenge, a Bidirectional RNN is suggested [2] that considers both forward and backward directions, creating twice as many hidden states to capture a more comprehensive context. Figure 3 illustrates a bidirectional RNN for sequence tagging:
For every , the hidden states and are created by considering and , respectively. The function takes both and and returns an output such that , where is a concatenation of the two hidden states and .
Q8: What are the advantages and limitations of implementing bidirectional RNNs for text classification and sequence tagging tasks?
Long Short-Term Memory (LSTM) Networks [3-5]
Gated Recurrent Units (GRUs) [6-7]
, Elman, Cognitive Science, 14(2), 1990.
, Schuster and Paliwal, IEEE Transactions on Signal Processing, 45(11), 1997.
, Hochreiter and Schmidhuber, Neural Computation, 9(8), 1997 ( available at ResearchGate).
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF, Ma and Hovy, ACL, 2016.*
Contextual String Embeddings for Sequence Labeling, Akbik et al., COLING, 2018.*
Learning Phrase Representations using RNN EncoderβDecoder for Statistical Machine Translation, Cho et al., EMNLP, 2014.*
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Chung et al., NeurIPS Workshop on Deep Learning and Representation Learning, 2014.*
