CS 5/662, Winter 2021

Week 4 / Wed, Jan 27 &

Neural Methods

Wednesday

Readings

• J&M Chapter 7
• Goldberg, Chapters 4, 5, 9, and 14 (don’t worry, they are short!)

Other Resources

• Possibly (definitely!) useful lecture notes from Stanford:
  • CS231n notes on network architectures
  • CS231n notes on backprop
  • Derivatives, Backpropagation, and Vectorization
  • cs224n: natural language processing with deep learning lecture notes: part III
• Also very useful: 
  • Understanding LSTM Networks, by Christopher Olah
  • Understanding LSTM and its diagrams, by Shi Yan
• Suggested: On the Practical Computational Power of Finite Precision RNNs for Language Recognition. Gail Weiss, Yoav Goldberg, Eran Yahav. In proc. ACL 2018

If you’re interested in the history of neural networks, here are a few places to start:

• Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211.
• Olazaran, M. (1996). A Sociological Study of the Official History of the Perceptrons Controversy. Social Studies of Science, 26(3), 611–659.

Friday

Readings

• Goldberg, Chapters 15, 16, and 17
• Optional but recommended: J&M Chapter 9
• Li et al., “Visualizing and Understanding Neural Models in NLP”, in Proc. NAACL 2016

These two are optional but strongly recommended:

• Pater, Joe. “Generative linguistics and neural networks at 60: Foundation, friction, and fusion.” Language 95(1) pp. e41-e74
• Linzen, Tal. “What can linguistics and deep learning contribute to each other? Response to Pater.” Language 95(1) pp. e99-108

Other Resources

• Karpathy et al., “Visualizing and Understanding Recurrent Networks”, in Proc. ICLR 2016
• Deep Learning with Pytorch: A 60 Minute Blitz