Skip to content

Chapter 2 — Calculus I and II (Single and Multivariable)

Part I — Mathematical Foundations · 6–10 weeks

What you will learn

This chapter covers single-variable and multivariable calculus: limits, derivatives, integrals, Taylor series, partial derivatives, gradients, Jacobians, Hessians, and the multivariable chain rule. A neural network is trained by gradient descent, which changes each parameter in the direction that lowers a loss function (a number that measures the error of the model); the required gradients are computed by backpropagation, which is the multivariable chain rule applied layer by layer. Integrals are required for the continuous probability distributions of Chapter 4, and Taylor approximations and Hessians for the optimization methods of Chapter 5. The emphasis is on differentiation and on functions of several variables; techniques of integration and the calculus of vector fields are not used in later chapters.

Topics

  • Limits and continuity
  • Derivatives and the rules of differentiation, including the chain rule
  • Optimization of single-variable functions
  • Integrals and the fundamental theorem of calculus
  • Sequences, series, and Taylor approximations
  • Vectors and functions of several variables
  • Partial derivatives
  • Gradients and directional derivatives
  • The chain rule in several variables
  • Jacobians and Hessians
  • Multivariable optimization: critical points, the second-derivative test, and Lagrange multipliers
  • Double integrals, at the level needed for the joint probability densities of Chapter 4

Resources

Suggested path. Watch The Essence of Calculus first for the geometric interpretation. Follow 18.01SC units 1–3 and 5 (differentiation, applications of differentiation, the definite integral, and series), then 18.02SC units 1–2 and the double-integral sessions of unit 3, and work the Khan Academy exercises on the same topics in parallel. Finish with the article by Parr and Howard and chapter 5 of Mathematics for Machine Learning, which restate the chapter in the notation of machine learning. When time is short, skip 18.01SC unit 4 (techniques of integration), the 18.02SC sessions on line, triple, and surface integrals, and every item marked optional or advanced.

University courses

  • MIT OpenCourseWare — 18.01SC Single Variable Calculus by David Jerison (Fall 2010; free; lecture and recitation videos, problem sets and exams with solutions; start here: the OpenCourseWare Scholar version, arranged for independent study; the original course page is 18.01 Fall 2006).
  • MIT OpenCourseWare — 18.02SC Multivariable Calculus by Denis Auroux (Fall 2010; free; lecture and recitation videos, problem sets and exams with solutions; vectors, partial derivatives, gradients, Lagrange multipliers, and multiple integrals; the original course page is 18.02 Fall 2007).
  • MIT OpenCourseWare — RES.18-005 Highlights of Calculus by Gilbert Strang (Spring 2010; free; optional; short videos; an overview of the main ideas of differential and integral calculus, usable as a preview or as a refresher).

Online courses (MOOCs)

Books

  • Book: Gilbert Strang, Calculus (3rd ed., Wellesley-Cambridge Press, 2017) — MIT OpenCourseWare page (free PDF, CC BY-NC-SA; start here: chapters 1–8 and 10 for single-variable calculus and series, 11 and 13 for vectors and partial derivatives).
  • Book: Gilbert Strang, Edwin Herman et al., Calculus Volumes 1–3 (OpenStax, 2016) — Volume 1, Volume 2, and Volume 3 (free PDF and web; all of Volume 1, Volume 2 chapters 5–6 for series and Taylor series, Volume 3 chapters 2 and 4 for vectors and multivariable differentiation).
  • Book: James Stewart, Daniel Clegg, and Saleem Watson, Calculus: Early Transcendentals (9th ed., Cengage, 2020) — author page (paid; optional; a widely adopted university calculus textbook; chapters 1–5, 11, 12, and 14).
  • Book (reference): Paul Dawkins, Paul's Online Math Notes (Lamar University) — Calculus I, Calculus II, and Calculus III (free web notes with worked problems; a reference for individual techniques).
  • Book: Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, Mathematics for Machine Learning (Cambridge University Press, 2020) — official page (free PDF; chapter 5, "Vector Calculus": gradients of vector- and matrix-valued functions, backpropagation, automatic differentiation, and the multivariate Taylor series).

Lectures, papers and articles

  • Lecture series: Grant Sanderson (3Blue1Brown), The Essence of Calculus (2017; free; 12 lessons; start here: the geometric interpretation of derivatives, integrals, the chain rule, and Taylor series; watch before or alongside the first course).
  • Paper/Article: Terence Parr and Jeremy Howard, The Matrix Calculus You Need For Deep Learning (2018; connects this chapter to neural networks: Jacobians, derivatives of element-wise operations, and the vector chain rule).
  • Paper/Article: Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind, Automatic Differentiation in Machine Learning: a Survey (Journal of Machine Learning Research, 2018; advanced; how frameworks such as PyTorch apply the chain rule automatically).

Milestone

Compute by hand the gradient and the Hessian of f(x, y) = x^2 y + e^(xy), and verify both with SymPy. For the composition L = (s(w x + b) − y)^2, where s(z) = 1 / (1 + e^(−z)) is the logistic function, derive dL/dw and dL/db with the chain rule, and confirm the result numerically with central finite differences at one chosen point.

Estimated time

6–10 weeks.