5.1. Language Models

Statistical

• N-gram Language Models, Jurafsky and Martin, Chapter 3 in Speech and Language Processing (3rd ed.), 2023.

Neural-based

• , Mikolov et al., ICLR, 2013. <- Word2Vec

• , Pennington et al., EMNLP, 2014.

• , Ppeters et al., NAACL, 2018. <- ELMo

Transformers

• , Vaswani et al., NIPS, 2017. <- Transformer

• , Liu et al., ICLR, 2018.

• , Devlin et al., NAACL, 2018.

Tokenization

• , Sennrich et al., ACL, 2016. <- Byte-Pair Encoding (BPE)

• , Wu et al., arXiv, 2016. <- WordPiece

• , Kudo and Richardson, EMNLP, 2018.

GPT (Generative Pre-trained Transformer)

• , Radford et al., OpenAI, 2018. <- GPT-1

• , Radford et al., OpenAI, 2019. <- GPT-2

• , Brown et al., NeurIPS, 2020. <- GPT-3