LLM for Dummies¶
This book is a self-study course that starts at college algebra and ends at building, training, operating, and improving open-source large language models (LLMs). Each chapter ends with a milestone: a practical, hands-on exercise that is used to check progress before the next chapter begins.
All required material is free or low-cost, and every resource is linked directly. The university courses listed publish their lecture videos, slides, and assignments openly, even when enrollment for credit is restricted.
Two ways in
Straight through, in chapter order: the book is built for it, and each chapter assumes the ones before it. Start at Chapter 1.
From a goal: the six learning paths each name a destination — understand how LLMs work, pretrain one, improve an existing model, serve and operate one, or build an application on open weights — and give the chapters that lead there in order, the prerequisites they assume down to the section, and the milestone that proves you arrived. Beginning from zero, take Path 0 — Foundations first.
This book is released under the MIT License. That license covers the book and the code written for its milestones, and nothing else: every course, textbook, paper, video, dataset, and model linked from the chapters keeps the terms of its own author or publisher.
How to use this book¶
The book has six parts and 24 chapters. The chapters are designed to be followed in order, because each one assumes the material of the chapters before it. The exception is Part II (programming and tools), which runs in parallel with Part I: Chapter 6 can start in the first week. Every chapter has the same structure:
- What you will learn — the scope of the chapter and how the rest of the course uses it.
- Topics — the syllabus, in study order.
- Resources — a suggested path, followed by university courses, online courses (MOOCs), books, and lectures, papers, and articles. The recommended starting point in each group is marked "start here"; optional and advanced items are labeled as such.
- Milestone — a hands-on exercise that applies the main ideas of the chapter and produces a result that can be verified.
- Estimated time — based on about 10 hours of study per week.
A reader who already knows a topic can skim the chapter and use the milestone as a diagnostic test. The milestones are the course's mechanism for checking progress, so completing each one before moving on is recommended.
The estimated total is 24–36 months at 10 hours per week, or 12–18 months at 20 hours per week. Part I accounts for a large share of that time because every later part depends on its mathematics.
Suggested first week
- Watch Andrej Karpathy's Intro to Large Language Models (1 hour) for an overview of the subject.
- Start Khan Academy College Algebra (Chapter 1): 30–45 minutes per day.
- Start CS50P (Chapter 6): 45–60 minutes per day.
- At the end of the week, re-read this page and set a weekly time budget. A fixed weekly schedule is easier to sustain over a multi-year course than irregular long sessions.
Map of goals to chapters¶
Each row below is the core of a learning path: a page that states the goal, the chapters that lead to it in order, what each one contributes, and the milestone that proves you got there. Start with Path 0 - Foundations first if you are beginning from zero.
| Goal | Relevant chapters |
|---|---|
| Understand how LLMs work internally | 13, 14, 15, 16 |
| Pretrain an open-source LLM | 14, 15, 16, 17 |
| Improve an existing model (training and research) | 16, 17, 18, 19, 20, 24 |
| Serve, deploy, and operate an LLM | 21, 22, 23 |
| Build applications on top of open models | 21, 22, 24 |
The core bookshelf¶
The books below are referenced in several chapters. Local PDFs of the freely distributable titles are kept in the
repository books/free/ folder; books/free/README.md is the manifest (source, license, and the relevant
chapters of each title). The links point to the official pages, and free titles are marked.
| Book | Where it fits | Access |
|---|---|---|
| OpenStax — Precalculus (2e, 2021) | Chapter 1 | Free PDF and web |
| Strang — Calculus (3rd ed.) | Chapter 2 | Free (MIT OCW) |
| Deisenroth, Faisal, Ong — Mathematics for Machine Learning | Chapters 2–5 | Free PDF |
| Strang — Introduction to Linear Algebra (6th ed.) | Chapter 3 | Companion to MIT 18.06 |
| Axler — Linear Algebra Done Right | Chapter 3, deeper rigor | Free online |
| Blitzstein & Hwang — Introduction to Probability | Chapter 4 | Free |
| Downey — Think Python (3rd ed.) | Chapter 6 | Free notebooks |
| Goodfellow, Bengio, Courville — Deep Learning | Chapters 5, 9, 10 | Free |
| James, Witten, Hastie, Tibshirani, Taylor — An Introduction to Statistical Learning | Chapter 8 | Free |
| Prince — Understanding Deep Learning | Chapters 8–10, 12 | Free |
| Zhang, Lipton, Li, Smola — Dive into Deep Learning | Chapters 5, 9, 11 | Free |
| Nielsen — Neural Networks and Deep Learning | Chapters 9, 10 | Free |
| Jurafsky & Martin — Speech and Language Processing, 3rd ed. | Chapters 11, 13 | Free |
| Raschka — Build a Large Language Model (From Scratch) | Chapters 12–15, 17, 23 | Official page, free code and videos |
| Alammar & Grootendorst — Hands-On Large Language Models | Chapters 13, 14, 17, 22 | Site + free code |
| Tazi, Mom, et al. (Hugging Face) — The Ultra-Scale Playbook | Chapters 7, 16 | Free |
| Ben Allal et al. (Hugging Face) — The Smol Training Playbook | Chapters 15, 23 | Free |
| Lambert — Reinforcement Learning from Human Feedback | Chapters 17–19 | Free online, paid print |
| Raschka — Build a Reasoning Model (From Scratch) | Chapter 19 | Hub + free videos |
| Kamath et al. — Large Language Models: A Deep Dive | Reference | Springer |
| Huyen — AI Engineering | Chapters 7, 17, 20–22 | Chapter summaries |
The course shelf¶
| Course | Provider | Chapter | Access |
|---|---|---|---|
| College Algebra / Pre-calculus | Khan Academy | 1 | Free, khanacademy.org |
| 18.01 / 18.02 Calculus | MIT OCW | 2 | Free, ocw.mit.edu |
| 18.06 Linear Algebra | MIT OCW | 3 | Free, ocw.mit.edu |
| Statistics 110 | Harvard | 4 | Free, YouTube |
| 18.065 Matrix Methods in Data Analysis, Signal Processing, and ML | MIT OCW | 5 | Free, ocw.mit.edu |
| CS50P / Python for Everybody | Harvard / PY4E | 6 | Free |
| CS229 Machine Learning | Stanford | 8 | Free |
| 6.S191 Introduction to Deep Learning | MIT | 9 | Free |
| Practical Deep Learning for Coders | fast.ai | 9 | Free (optional) |
| CS231n Deep Learning for Computer Vision | Stanford | 10 | Free |
| Zero to Hero | Karpathy | 10, 12, 13, 14, 16 | Free |
| CS224N NLP with Deep Learning | Stanford | 11, 12 | Free (videos) |
| CS25 Transformers United | Stanford | 12 | Free (videos, optional) |
| CS336 Language Modeling from Scratch | Stanford | 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23 | Free |
| The LLM Course | Hugging Face | 12, 13, 14, 17, 19 | Free, huggingface.co/learn |
| CS324 Foundation Models | Stanford | 15 | Free (optional) |
| 11-868 Large Language Model Systems | CMU | 16, 21 | Free, syllabus online |
| 11-667 LLM Methods and Applications | CMU | 18 | Free, syllabus online |
| The Post-Training Course | Nathan Lambert | 18, 19 | Free |
| smol course | Hugging Face | 17, 18, 20, 23 | Free |
| Efficiently Serving LLMs | DeepLearning.AI | 21 | Free |
| LLM Bootcamp | Full Stack Deep Learning | 22 | Free |
| CS329S Machine Learning Systems Design | Stanford | 22 | Free |