Training a language model involves dozens of choices nobody can derive from first principles: the learning rate, the batch size, how long to warm up, how much weight decay, which optimizer, how to scale any of it to a bigger model. Practitioners mostly know these things the way an alchemist knew a recipe. Stanford’s CS 312, Deep Learning Alchemy (Tatsunori Hashimoto and Suhas Kotha) takes a stricter line: you understand a training phenomenon when you can predict the outcome of an experiment before running it. Its grade is 85% prediction quizzes about one small reference model. This companion follows that course unit by unit, in its own words, with interactive figures, sealed predictions you lock before reading the answer, and a ledger that measures how well calibrated you are.
It has one extra purpose the original does not. If you direct an agent (a person or an LLM) that runs training experiments for you, the hard part is not running them; it is knowing which conclusions to trust. Every unit ends with audit drills: short, fictional research logs with a conclusion in them, and the question “what would you challenge before accepting this?”
How to use it. Read Unit 0 once. Then, for each unit: lock every prediction before you scroll past it, do the drills, and check your ledger at the end. The ledger lives in your browser only; export it if you switch devices.
Prediction as understanding, the reference model everything is measured on, and a protocol for directing experiments you did not run yourself.
How hyperparameters behave, how much a single run can tell you, and why optimizers, batch sizes and model sizes each move the best learning rate.
Each unit is written once its lectures and quiz are out, so the debrief can use the course’s measured answers.
Your ledger: every prediction you locked, your hit rate, and your calibration at 50%, 70% and 90% confidence.
CS 312 releases slides and a prediction quiz roughly every two weeks. This companion follows one step behind, so the quiz debriefs can rely on the course’s own measured answers rather than on guesses.
| Date (2026) | CS 312 | Here |
|---|---|---|
| 22–24 Sep | Introduction, LLMs 101, lecture 2: basics | Units 0 and 1 |
| 29 Sep | Quiz 1: basics | Unit 1 debrief |
| 1 & 6 Oct | Lectures 3–4: optimization I | Units 2a, 2b |
| 8 Oct | Quiz 2: hyperparameter scaling | Unit 2 debriefs |
| 13 & 15 Oct | Lectures 5–6: loss landscapes, sharpness, edge of stability | Unit 3 in preparation |
| 20 Oct | Quiz 3 | Unit 3 |
| 22 & 27 Oct · 29 Oct | Architecture I · quiz | Unit 4 |
| 5 Nov · 10 Nov | Architecture II · quiz | Unit 5 |
| 12 & 17 Nov · 19 Nov | Generalization · quiz | Unit 6 |
| 1 Dec · 3 Dec | Exotic topics · quiz | Unit 7 |