More MT & Applications

J&M Chapter 11
- Focus on the middle parts of this chapter (sections 11.2 through 11.6)
- You may find Eisenstein Chapter 18 to be helpful
Bahdanau, Cho, & Bengio. (2015) Neural Machine Translation by Jointly Learning to Align and Translate. Proc. ICLR 2015
Olah & Carter, “Attention and Augmented Recurrent Neural Networks”, Distill, 2016.
Vaswani et al., “Attention is All You Need”. Proc. NeurIPS 2017

Lilian Weng’s epic blog post, “Attention? Attention!” is an incredible resource for understanding attention mechanisms
I also found Guillaume Genthial’s “Seq2Seq with Attention and Beam Search” helpful when I was first learning about attention.
Jay Alammar’s “The Illustrated Transformer” is excellent.
Ilharco et al.’s superb tutorial from this year’s EMNLP on Transformers, “High Performance Natural Language Processing”, is available online
Sharon Zhou’s “Has AI surpassed humans at translation? Not even close!”[https://www.skynettoday.com/editorials/state_of_nmt] is a great overview of limitations of SoTA NMT systems in practice
If you’re interested in some current research, here are two very recent papers that cover important and relevant topics:
- Fadaee, M., & Monz, C. (2020). The Unreasonable Volatility of Neural Machine Translation Models (pp. 88–96). Presented at the Proceedings of the Fourth Workshop on Neural Generation and Translation, Online: Association for Computational Linguistics.
- Gowda, T., & May, J. (2020). Finding the Optimal Vocabulary Size for Neural Machine Translation (pp. 3955–3964). Presented at the Findings of the Association for Computational Linguistics: EMNLP 2020, Online: Association for Computational Linguistics.

TBD