Skip to content

Chapter 19 — Reasoning Models and Reinforcement Learning

Part V — Post-Training · 5–7 weeks

What you will learn

Reasoning models such as OpenAI o1 and DeepSeek-R1 solve difficult mathematics and coding problems by generating a long chain of thought before the final answer, and this chapter covers how such models are built. It begins with the methods that require no training: prompting a large language model (LLM) for step-by-step answers, and inference-time scaling, which spends more compute per question by sampling many answers and selecting one by majority vote or with a verifier. It then develops the reinforcement learning (RL) foundations that Chapter 18 used at a conceptual level — Markov decision processes (MDPs), policy gradients, and proximal policy optimization (PPO) — and applies them to language models with rewards that a program can verify, such as a correct final answer or passing unit tests, using group relative policy optimization (GRPO), the algorithm used to train DeepSeek-R1. It also covers the distillation of a large reasoning model into a small one. The milestone trains a small model with RL on a single graphics processing unit (GPU) and measures the change in its reasoning accuracy.

Topics

  • Chain-of-thought prompting and the effect of intermediate steps on accuracy
  • Evaluating reasoning: answer extraction, exact-match checkers, and pass@k (the probability that at least one of k sampled answers is correct)
  • Inference-time scaling (self-consistency, best-of-N sampling, verifier-guided search)
  • Outcome and process reward models as verifiers
  • RL foundations: MDPs, returns, value functions, and advantages
  • Policy gradients: REINFORCE, baselines, actor-critic, and the clipped objective of PPO
  • RL with verifiable rewards (RLVR) and its differences from reinforcement learning from human feedback (RLHF, Chapter 18)
  • Reward design for math and code (answer and format rewards, unit tests) and reward hacking
  • GRPO: group-relative advantages without a value network
  • Reasoning models: DeepSeek-R1-Zero, DeepSeek-R1, and o1-style models
  • Distillation of reasoning traces into small models

Resources

Suggested path. Read "Understanding Reasoning LLMs" by Sebastian Raschka for an overview, then divide each week between two tracks. The first track is RL foundations: the three-part introduction of Spinning Up, chapters 3 and 13 of Sutton and Barto, and the policy-gradient lectures of CS 285 (or David Silver's course, which proceeds at a slower pace). The second track is reasoning models: Build a Reasoning Model (From Scratch) is the main text of this chapter; read the chain-of-thought and self-consistency papers alongside its chapters 4–5, and the DeepSeekMath and DeepSeek-R1 papers, CS336 lecture 16, and chapter 12 of the Hugging Face LLM Course alongside its chapters 6–7. When time is short, skip David Silver's course, the remaining CS 285 lectures, and the process-reward and test-time-compute papers, and complete the milestone with the Hugging Face Transformer Reinforcement Learning (TRL) library or Unsloth instead of a hand-written training loop.

University courses

  • Stanford — CS336 Language Modeling from Scratch (Spring 2026) by Tatsunori Hashimoto and Percy Liang (free; videos + assignments; start here: lecture 16, "Post-training - RLVR", on policy gradients for language models, GRPO, and the DeepSeek-R1 recipe; advanced: assignment 5 implements supervised fine-tuning (SFT), expert iteration, and GRPO for math reasoning and is sized for a data-center GPU).
  • UC Berkeley — CS 185/285 Deep Reinforcement Learning by Sergey Levine (free; Spring 2026 slides and homework on the course site and Fall 2023 videos, whose numbering differs; the relevant lectures in the 2026 schedule are 4–6 on RL basics, policy gradients, and actor-critic, 9–10 on advanced policy gradients, and 14, "LLM RL").
  • UCL and DeepMind — Reinforcement Learning course (2015) by David Silver (free; videos + slides; optional: a ten-lecture introduction at a slower pace than CS 285; the relevant lectures are 1–3 on MDPs and value functions and 7 on policy gradients).

Online courses (MOOCs)

  • Hugging Face — LLM Course, Chapter 12: Open R1 for Students (free; a few hours; start here: covers RL for language models, a reading of the DeepSeek-R1 paper, GRPO in TRL, and a GRPO training exercise).
  • OpenAI — Spinning Up in Deep RL (free; 2018; a concise introduction to RL terminology and policy optimization, with reference implementations of vanilla policy gradient and PPO; read "Introduction to RL" parts 1–3).

Books

  • Book: Sebastian Raschka, Build a Reasoning Model (From Scratch) (Manning, 2026) — official hub (paid; free code in the GitHub repo; start here: chapter 3 on evaluating with verifiers, chapters 4–5 on inference-time scaling, chapters 6–7 on RLVR and GRPO, and chapter 8 on distillation, all implemented in PyTorch on a small Qwen3 model).
  • Book: Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction (2nd ed., MIT Press, 2018) — official page (free PDF; the standard RL textbook; chapter 3, "Finite Markov Decision Processes", and chapter 13, "Policy Gradient Methods", provide the terminology and the mathematics underlying PPO and GRPO).
  • Book: Nathan Lambert, Reinforcement Learning from Human Feedback (Manning, 2026) — official page (free online, paid print; chapter 6, "Reinforcement Learning", derives the policy-gradient algorithms from REINFORCE to PPO and GRPO as they are applied to language models, and chapter 7, "Reasoning and Inference-Time Scaling", surveys RLVR and the reasoning-model recipes; lecture 5 of the companion Post-Training Course covers chapter 7).

Lectures, papers and articles

Tools and hands-on

  • Hugging Face — TRL documentation (free, open source; start here: the GRPOTrainer page documents a widely used open-source GRPO implementation, its custom reward functions, and the logged reward and completion length that the milestone plots; optional: the Unsloth RL guide wraps the same trainer, explains reward-function design, and links free Colab notebooks that train models of up to 1.5B parameters with low-rank adaptation (LoRA) in about 5 GB of GPU memory).

Milestone

Train a small instruction-tuned model (Qwen2.5-0.5B-Instruct or Qwen3-0.6B; with LoRA the run fits a free Colab or Kaggle T4 GPU, and a 1.5B-parameter model fits a 24 GB GPU) with GRPO — the TRL GRPOTrainer or a hand-written loop based on chapters 6–7 of Build a Reasoning Model (From Scratch) — on the training set of GSM8K, a dataset of grade-school math word problems, with a rule-based reward for the answer format and for an exact-match final answer. Plot the mean reward, the mean completion length, and the accuracy on a fixed 200-question slice of the GSM8K test set against training steps. Compare the final accuracy with that of the model before RL training under greedy decoding and under self-consistency with 8 samples, and state which of the two gains required more compute.

Estimated time

5–7 weeks.