Given the input text W={w1,…,wn} where wi is the i'th token in W, a contextualized encoder (e.g., BERT) takes W and generates an embedding ei∈R1×d for every token wi∈Wusing wi as well as its context. The challenge is that this encoder can take only up the m-number of tokens such that it cannot handle any input where n>m.
What are the ways to handle arbitrarily large input using a contextualized encoder?
One popular method is called the "Sliding Window", which splits the input into multiple blocks of text, generates embeddings for each block separately, and merges them at the end.
Let where if ; otherwise, such that . Then, the encoder takes each and generates for every token in . Finally, the embedding matrix is created by sequentially stacking all embeddings in .
The baseline method does not have enough context to generate high-quality embeddings for tokens on the edge of each block.
Modify the baseline method such that a block has overlapped tokens with its surrounding blocks (both front and back). Once all blocks are encoded, each overlapped token should have two embeddings. Create an average embedding of those two embeddings and make it the final embedding for the overlapped token.
In a (aka, an encoder-decoder model), a decoder takes an embedding matrix and predicts what token should come next. It is often the case that this embedding matrix is also bounded by a certain size, which becomes an issue when the size of the matrix becomes larger than (for the case above, where ). One common method to handle this issue is to use an attention matrix for dimensionality reduction as follows:
The embedding matrix is first transposed to then multiplied by an attention matrix such that . Finally, the transpose of , that is gets fed into the decoder.
Would the following method be equivalent to the above method?
An attention matrix is multiplied by the embedding matrix such that . Finally, gets fed into the decoder.