Preliminaries

Eisenstein, INLP: Chapter 1
Bender, LFNLP: Chapter 1
Bender E. (2017), Linguistics, NLP, and Interdisciplinarity (Or: Look at Your Data)
Winograd T. (1973), A Procedural Model of Language Understanding. In: Grosz B., Spärck Jones K., Webber B. (eds) Readings in Natural Language Processing. Morgan Kaufmann, San Francisco
Spärck Jones K. (1994) Natural Language Processing: A Historical Review. In: Zampolli A., Calzolari N., Palmer M. (eds) Current Issues in Computational Linguistics: In Honour of Don Walker. Linguistica Computazionale, vol 9. Springer, Dordrecht
Manning C. (2015), Last Words: Computational Linguistics and Deep Learning. Computational Linguistics (41) 4:701–707

J&M Chapter 3
Church, K. W., & Hanks, P. (1990). Word Association Norms, Mutual Information, and Lexicography. Computational Linguistics, 16(1), 22–29.

There are plenty of descriptions of how concepts of information theory apply (or don’t) to language, but I personally find George Miller’s essay from 1953 to be extremely readable and helpful:
- Miller, G. (1953). What is information measurement? The American Psychologist, 8(1), 3–11.
Peter Norvig’s essay, “On Chomsky and the Two Cultures of Statistical Learning” is thought-provoking
- As is the essay by Leo Breiman that Norvig is referring to
- And since it came up, Jason Eisner’s take on Breiman’s article is worth glancing at as well.