# Homework

## Quiz

1. The EM algorithm stands as a classic method in unsupervised learning. What are the advantages of unsupervised learning over supervised learning, and which tasks align well with unsupervised learning?
2. What are the disadvantages of using BPE-based tokenization instead of [rule-based tokenization](https://emory.gitbook.io/nlp-essentials/text-processing/tokenization)? What are the potential issues with the implementation of BPE above?
3. How does self-attention operate given an embedding matrix $$\mathrm{W} \in \mathbb{R}^{n \times d}$$ representing a document, where $$n$$ is the number of words and $$d$$ is the embedding dimension?
4. Given the same embedding matrix as in question #3, how does multi-head attention function? What advantages does multi-head attention offer over self-attention? 
5. What are the outputs of each layer in the Transformer model? How do the embeddings learned in the upper layers of the Transformer differ from those in the lower layers?
6. How is a Masked Language Model used in training a language model with a transformer?
7. How can one train a document-level embedding using a transformer?
8. What are the advantages of embeddings generated by transformers compared to those generated by [Word2Vec](/nlp-essentials/distributional-semantics/word2vec.md)?
9. Neural networks gained widespread popularity for training natural language processing models since 2013. What factors enabled this popularity, and how do they differ from traditional NLP methods?
10. Recent large language models like ChatGPT or Claude are trained quite differently from traditional NLP models. What are the main differences, and what factors enabled their development?

## References

* [Attention is All you Need](https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html), Vaswani et al., NIPS 2017.
* [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://aclanthology.org/N19-1423/), Devlin et al., NAACL 2019.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://emory.gitbook.io/nlp-essentials/contextual-encoding/homework.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.