Skip to content

Chapter 3 — Linear Algebra

Part I — Mathematical Foundations · 8–12 weeks

What you will learn

This chapter covers vectors, matrices, and the matrix factorizations built on them, in both their algebraic and their geometric interpretation. A large language model (LLM) represents each token (a unit of text) as a vector called an embedding, measures the relevance of one token to another with dot products (the attention scores of Chapter 12), and applies its weights by matrix multiplication, so every later chapter uses this material. Orthogonality, projections, and least squares underlie the regression methods of Chapter 8, and eigenvalues and positive definite matrices describe the curvature of the loss functions studied in Chapter 5. The singular value decomposition (SVD) and low-rank approximation are the basis of principal component analysis (PCA) and of the low-rank fine-tuning methods of Chapter 17.

Topics

  • Vectors, dot products, norms, and angles (cosine similarity)
  • Linear combinations, span, and linear independence
  • Systems of linear equations and elimination (the LU factorization)
  • Matrices as linear transformations
  • Matrix multiplication and inverses
  • Vector spaces, bases, dimension, and rank; the four fundamental subspaces
  • Orthogonality, projections, and least squares
  • Gram-Schmidt and the QR factorization
  • Determinants
  • Eigenvalues, eigenvectors, and diagonalization
  • Symmetric and positive definite matrices
  • The singular value decomposition
  • Low-rank approximation and principal component analysis

Resources

Suggested path. Watch Essence of Linear Algebra first, then follow 18.06SC from start to finish, with Strang's Introduction to Linear Algebra or the free text by Boyd and Vandenberghe as the written reference; the Khan Academy videos give slower derivations of individual topics where needed. Read chapters 2–4 and 10 of Mathematics for Machine Learning near the end, before the milestone. When time is short, skip the 18.06SC sessions on differential equations, Markov matrices and Fourier series, and complex matrices and the fast Fourier transform, and leave 18.065, Axler, and the other optional and advanced items for a second pass.

University courses

  • MIT OpenCourseWare — 18.06SC Linear Algebra by Gilbert Strang (Fall 2011; free; lecture and recitation videos, problem sets and exams with solutions; start here: the OpenCourseWare Scholar version of 18.06, arranged for independent study; elimination, the four fundamental subspaces, orthogonality, determinants, eigenvalues, and the SVD; the original course page is 18.06 Spring 2010).
  • MIT OpenCourseWare — 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning by Gilbert Strang (Spring 2018; free; lecture videos + problem sets; advanced; the follow-on course to 18.06: the SVD, low-rank approximation, and PCA, followed by optimization and neural networks; Chapter 5 uses lectures 21–25).
  • MIT OpenCourseWare — RES.18-010 A Vision of Linear Algebra by Gilbert Strang (Spring 2020, with later additions; free; optional; short videos that summarize the subject through the factorization A = CR, the four fundamental subspaces, and the SVD; a review after 18.06SC).

Online courses (MOOCs)

Books

  • Book: Gilbert Strang, Introduction to Linear Algebra (6th ed., Wellesley-Cambridge Press, 2023) — official page (paid; sample sections and selected solutions free; start here: the textbook of 18.06; chapters 1–7 cover every topic of this chapter, through the SVD).
  • Book: Stephen Boyd and Lieven Vandenberghe, Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares (Cambridge University Press, 2018) — official page (free PDF, with Julia and Python companions, slides, and lecture videos from Stanford ENGR108; an applied, data-oriented introduction to norms, clustering, and least squares; does not cover eigenvalues or the SVD).
  • Book: Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, Mathematics for Machine Learning (Cambridge University Press, 2020) — official page (free PDF; chapters 2, "Linear Algebra", 3, "Analytic Geometry", and 4, "Matrix Decompositions", then chapter 10 on PCA, the subject of the milestone).
  • Book: David C. Lay, Steven R. Lay, and Judi J. McDonald, Linear Algebra and Its Applications (6th ed., Pearson, 2020) — publisher page (paid; optional; a widely assigned first-course textbook in US universities; computational, with many applications).
  • Book: Jim Hefferon, Linear Algebra (4th ed., 2020) — official page (free PDF, with a free book of worked answers to all exercises; optional; a proof-oriented first course that prepares for Axler).
  • Book: Sheldon Axler, Linear Algebra Done Right (4th ed., Springer, 2024) — official page (free PDF, open access under CC BY-NC; advanced; a standard second course, which develops the theory through linear maps and introduces determinants only near the end).
  • Book: Gilbert Strang, Linear Algebra and Learning from Data (Wellesley-Cambridge Press, 2019) — official page (paid; advanced; the textbook of 18.065: the SVD, low-rank approximation, optimization, and neural networks).

Lectures, papers and articles

  • Lecture series: Grant Sanderson (3Blue1Brown), Essence of Linear Algebra (2016; free; 16 lessons; start here: matrices as transformations of space, determinants as scaling factors of area, and eigenvectors, all shown geometrically; watch before or alongside 18.06SC).

Milestone

Implement matrix multiplication and the transpose with plain Python loops and verify them against the NumPy operators @ and .T. Compute the dominant eigenvector of a small symmetric matrix by power iteration and verify it against numpy.linalg.eigh. Then perform PCA on a small dataset such as Iris in two ways — from the eigenvectors of the covariance matrix and from the SVD of the centered data — and show that both give the same two-dimensional projection up to sign.

Estimated time

8–12 weeks.