Skip to content

Chapter 4 — Probability and Statistics

Part I — Mathematical Foundations · 6–10 weeks

What you will learn

This chapter covers probability theory, the elements of statistical inference, and basic information theory. A large language model (LLM) is a probabilistic model: it outputs a probability distribution over the next token (a unit of text), the probability of a whole text follows from the chain rule of probability, and text is generated by sampling from these distributions. Training an LLM is maximum likelihood estimation (MLE), and the loss function that is minimized is the cross-entropy between the training data and the model. The Kullback–Leibler (KL) divergence measures the difference between two distributions and is used as a penalty term by the alignment methods of Chapter 18. Confidence intervals and hypothesis tests determine whether the difference between two evaluation scores in Chapter 20 exceeds sampling noise.

Topics

  • Sample spaces, events, counting, and the axioms of probability
  • Conditional probability, independence, and Bayes' rule
  • Random variables and discrete distributions (Bernoulli, binomial, categorical, Poisson)
  • Continuous distributions (uniform, exponential, normal)
  • Expectation, variance, and covariance
  • Joint, marginal, and conditional distributions, and the chain rule of probability
  • The law of large numbers and the central limit theorem
  • Sampling and Monte Carlo estimation
  • Markov chains, at the level needed for the n-gram language models of Chapter 11
  • Maximum likelihood estimation and Bayesian updating
  • Confidence intervals, hypothesis tests, and the bootstrap
  • Entropy, cross-entropy, KL divergence, and perplexity

Resources

Suggested path. Use the Statistics 110 lectures with the textbook by Blitzstein and Hwang (chapters 1–7 and 9–11) as the main course, and work the Stat110x or textbook exercises in parallel. Then take the statistics topics (MLE, Bayesian updating, confidence intervals, hypothesis tests, and the bootstrap) from MIT 18.05, and finish with Visual Information Theory and the two 3Blue1Brown lessons on entropy and cross-entropy. When time is short, replace 18.05 with chapter 6 of Mathematics for Machine Learning and the last two weeks of the DeepLearning.AI course, and skip 6.041SC, CS109, and the books marked optional or advanced.

University courses

  • Harvard — Statistics 110: Probability (lecture playlist) by Joe Blitzstein (free; 35 videos, with handouts and practice problems on the course site; start here: a first course in probability, from counting to Markov chains, taught through worked problems).
  • MIT OpenCourseWare — 6.041SC Probabilistic Systems Analysis and Applied Probability by John Tsitsiklis (Fall 2013; free; lecture, recitation, and tutorial videos + problem sets with solutions; an engineering-oriented alternative to Statistics 110 that ends with Bayesian and classical inference).
  • Stanford — CS109 Probability for Computer Scientists (free slides + problem sets; the companion course reader by Chris Piech covers counting through maximum likelihood, information theory, and logistic regression, with Python examples).
  • MIT OpenCourseWare — 18.05 Introduction to Probability and Statistics by Jeremy Orloff and Jennifer French Kamrin (Spring 2022; free; readings + problem sets and exams with solutions, no videos; the source for the statistics topics: Bayesian updating, MLE, confidence intervals, hypothesis tests, and the bootstrap).

Online courses (MOOCs)

  • HarvardX (edX) — Introduction to Probability (Stat110x) by Joe Blitzstein (free to audit, paid certificate; start here: adds automatically graded exercises to the Statistics 110 lectures).
  • DeepLearning.AI (Coursera) — Probability & Statistics for Machine Learning & Data Science by Luis Serrano (free to audit, paid certificate; 4 weeks, about 30 hours; course 3 of the specialization described in Chapter 5; an introductory treatment of distributions, the central limit theorem, MLE, confidence intervals, and hypothesis tests, with Python labs).
  • Khan Academy — Statistics and probability (free; videos + self-grading exercises; review of basic probability, descriptive statistics, and significance tests).

Books

  • Book: Joseph K. Blitzstein and Jessica Hwang, Introduction to Probability (2nd ed., Chapman & Hall/CRC, 2019) — official free PDF (free PDF, paid print edition; start here: the textbook of Statistics 110; chapters 1–7, 9, and 10 alongside the lectures, and chapter 11 for Markov chains).
  • Book: Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, Mathematics for Machine Learning (Cambridge University Press, 2020) — official page (free PDF; chapter 6, "Probability and Distributions": a summary of the probability used in machine learning, including the Gaussian distribution and Bayes' rule).
  • Book: Dimitri P. Bertsekas and John N. Tsitsiklis, Introduction to Probability (2nd ed., Athena Scientific, 2008) — official page (paid; the textbook of 6.041; chapters 1–5, then 8–9 on Bayesian and classical inference).
  • Book: Larry Wasserman, All of Statistics: A Concise Course in Statistical Inference (Springer, 2004) — official page (paid; a compact statistics course written for computer science and machine learning students; chapters 1–6 and 8–11 cover convergence, the bootstrap, MLE, hypothesis testing, and Bayesian inference).
  • Book: Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022) — official page (free PDF; chapters 2–6: probability, statistics, decision theory, and information theory, including cross-entropy and KL divergence, in machine learning notation).
  • Book: David J. C. MacKay, Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003) — official page (free PDF for on-screen reading; optional; chapters 1–4 for entropy and source coding, the theory that relates prediction to data compression).
  • Book: Christopher M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006) — official page (free PDF; advanced; chapters 1 and 2 on probability theory, information theory, and probability distributions).

Lectures, papers and articles

Milestone

Derive by hand the maximum likelihood estimates of the bias of a coin and of the mean and variance of a Gaussian, and verify them by simulation in NumPy. Build a character-level bigram model of a small text by counting (the normalized counts are the maximum likelihood estimate; add-one smoothing removes zero probabilities), sample text from it, and show that its cross-entropy and perplexity on held-out text are lower than those of a unigram model and of a uniform model. Verify numerically on a small example that cross-entropy equals entropy plus KL divergence.

Estimated time

6–10 weeks.