CS 5/662, Winter 2023

Week 7 / Tue, Feb 21 & Thu, Feb 23

Capturing Context

Tuesday

CWEs, Sentences & Documents

Readings

• Smith, “Contextual Word Representations: A Contextual Introduction”. April 2020, arXiv
• Rogers et al. “A Primer in BERTology: What we know about how BERT works” Nov. 2020, to appear in TACL
• Ettinger et al. “Assessing Composition in Sentence Vector Representations”. in Proc COLING 2018
• Reimers & Gurevych. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”. in Proc EMNLP 2019
• Boyd-Graber et al. “Applications of Topic Models.” F&TIR 11(2-3)
  • Read sections 1, 3, and 9
  • The other chapters are surveys of different application areas (historical text, scientific literature, etc.)- pick one that is of interest.

Other Resources

CWEs

• Ettinger, “What BERT Is Not: Lessons From a New Suite of Psycholinguistic Diagnostics for Language Models.” TACL (2020) vol. 8, pp. 34-48
• Jay Alammar’s “The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)” is a fabulous resource
• Metzler et al. “Efficient Transformers: A Survey” Sep. 2020, arXiv

Topic Models

• Blei et al. “Latent Dirichlet Allocation” JLMR 3(Jan) pp. 993-1022 (2003)
• Sia et al. “Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!”. in Proc. EMNLP 2020
• David Mimno’s “jsLDA” is an in-browser LDA implementation that is useful for developing an intuition about how the model works.

Thursday

Readings

TBA (Probably large language models, zero-shot learning, & prompting)