Skip to content

Chapter 14 — Build an LLM from Scratch

Part IV — Transformers and LLMs · 6–8 weeks

What you will learn

This chapter assembles the components of Chapters 12 and 13 into a complete GPT-style large language model (LLM) and trains it. It covers the model definition (embedding tables, a stack of Transformer blocks, and the output layer that predicts the next token), the cross-entropy training objective, and a training loop with the AdamW optimizer, learning-rate warmup and cosine decay, gradient clipping, and periodic evaluation on held-out data. It then covers text generation with greedy decoding, temperature, and top-k and top-p sampling. The implementation is verified by loading the released GPT-2 weights into it, and the trained model is saved and published on the Hugging Face Hub. Chapters 15 and 16 scale the same procedure up in data, model size, and hardware. An optional closing section looks inside the trained model — the residual stream, what its attention heads compute, and the induction heads that perform in-context copying — as a first step into mechanistic interpretability.

Topics

  • The GPT model: token and positional embeddings, Transformer blocks, final normalization, and the output layer
  • Parameter counting, weight tying between the embedding table and the output layer, and weight initialization
  • The next-token prediction objective, the cross-entropy loss, and the training/validation split
  • Batches of token sequences: context length and batch size
  • The training loop: AdamW, weight decay, learning-rate warmup and cosine decay, and gradient clipping
  • Monitoring of training and validation loss, overfitting, and generated samples
  • Text generation: the autoregressive loop, greedy decoding, temperature, and top-k and top-p sampling
  • Saving and loading checkpoints (model and optimizer state)
  • Loading the released GPT-2 weights into the implementation as a correctness check
  • Publishing a model on the Hugging Face Hub
  • An optional first look inside the trained model: the residual stream, attention heads as circuits, and induction heads

Resources

Suggested path. Raschka Chapters 4 and 5 with Appendix D are the main text of this chapter: type and run the code chapter by chapter, with the companion videos as a second explanation. Karpathy's "Let's build GPT" was watched in Chapter 12 and is not repeated here; continue with the first section of "Let's reproduce GPT-2 (124M)" for the model definition and the weight-loading check, and with its optimizer section for AdamW, gradient clipping, and the learning-rate schedule, with nanoGPT open as a reference implementation. Read Holtzman et al. and "How to generate text" before writing the sampling code, the TinyStories paper before choosing the milestone corpus, and Chapter 4 of the Hugging Face LLM Course before publishing the checkpoint. CS336 lectures 2–3 and the Transformer part of Assignment 1 are a stricter, test-driven route to the same result; when time is short, omit them together with Hands-On Large Language Models, the GPT-3 paper, nanochat, and llm.c. The interpretability items — Chapter 10 of Jurafsky and Martin, ARENA Chapter 1, the Transformer Circuits article, and TransformerLens — form an optional closing section after the milestone; omit them when time is short.

University courses

  • Stanford — CS336: Language Modeling from Scratch by Tatsunori Hashimoto and Percy Liang (free; Spring 2026 lecture videos and assignments; start here: lecture 2 "PyTorch (einops), resource accounting" and lecture 3 "Architectures, hyperparameters", first assigned in Chapter 12; in Assignment 1 (Basics) the Transformer language model, cross-entropy loss, AdamW, training loop, and decoding are written without the ready-made torch.nn layers and checked by unit tests; the tokenizer part was covered in Chapter 13).

Online courses (MOOCs)

  • Andrej Karpathy — Let's reproduce GPT-2 (124M) (free; 4 hours; the sequel to Lecture 7 of Neural Networks: Zero to Hero, with step-by-step code in build-nanogpt; start here: section 1, about the first 80 minutes, defines the GPT-2 model, loads the released weights, and writes the training loop; the section on AdamW, gradient clipping, and the learning-rate schedule starts at about 2h15m; the sections on speed and multi-GPU training belong to Chapter 16).
  • Hugging Face — LLM Course, Chapter 4: Sharing models and tokenizers (free; about 2 hours; the Hugging Face Hub, uploading a checkpoint, and writing a model card, as required by the milestone; Chapter 2 "Using Transformers" covers loading and saving models).
  • ARENA — ARENA curriculum by Callum McDougall et al. (free, open source; Colab notebooks and web pages with solutions; optional: Chapter 1 "Transformer Interpretability", whose first two exercise sets build a GPT-2-style Transformer and then locate induction heads in it with TransformerLens; the later sets (indirect object identification, superposition, sparse autoencoders) are a longer second project; Chapter 0 repeats the material of Chapters 9 and 10 and can be skipped).

Books

  • Book: Sebastian Raschka, Build a Large Language Model (From Scratch) (Manning, 2024) — official page (paid; free code at rasbt/LLMs-from-scratch and free companion videos; start here: Chapter 4 "Implementing a GPT Model from Scratch to Generate Text", Chapter 5 "Pretraining on Unlabeled Data", which includes decoding strategies and loading the GPT-2 weights, and Appendix D on warmup, cosine decay, and gradient clipping; Chapters 2 and 3 were used in Chapters 13 and 12).
  • Book: Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024) — official page (paid; optional; free notebooks on GitHub; Chapter 3 "Looking Inside Transformer LLMs" follows one forward pass of a released model, including the decoding strategy and the key-value cache).
  • Book: Dan Jurafsky and James H. Martin, Speech and Language Processing (3rd ed. draft, August 2026 release) — official page (free PDF; Chapter 7 "Transformers and Pretraining", sections 7.6–7.8: decoding, pretraining with next-token prediction, and the sizes, data, and compute of current models; sections 7.1–7.5 were read in Chapter 12; Chapter 10 "Interpretability" — probing, the logit lens, sparse autoencoders, causal techniques, and induction heads — belongs to the optional closing section of this chapter).

Lectures, papers and articles

Tools and hands-on

  • Code: Andrej Karpathy, nanoGPT (start here: a GPT model definition of about 300 lines and a training loop of about 300 lines; the README now points to nanochat as the successor, but the code remains a compact reference for this chapter).
  • Code: Andrej Karpathy, nanochat (2025; advanced: tokenizer, pretraining, fine-tuning, evaluation, and inference for a small chat model, designed to run on one node of eight H100 GPUs).
  • Code: Andrej Karpathy, llm.c (2024; optional: GPT-2 training written in plain C and CUDA without PyTorch; shows every forward and backward kernel explicitly).
  • Code: TransformerLens (free, open source, MIT license; optional: loads GPT-2 and other open models with hooks on every activation, so that attention patterns can be inspected and individual heads ablated; the library used by the ARENA exercises).

Milestone

Load the released GPT-2 (124M) weights into the from-scratch implementation and verify that its logits match those of the Hugging Face GPT2LMHeadModel on the same input. Then train a decoder-only model of 10–50M parameters on a public corpus such as TinyStories, plot the training and validation loss, and compare samples generated with greedy decoding, temperature, top-k sampling, and top-p sampling. Publish the checkpoint on the Hugging Face Hub with a model card that states the data, the hyperparameters, and the final validation loss.

Estimated time

6–8 weeks.