When a training experiment fails, the only thing you know for sure is that at least one setting was bad. This unit is about telling which. It starts with the most consequential setting, the learning rate: final loss is roughly a parabola in its logarithm, and the bottom of the parabola moves when you change the batch size, the model size or the optimizer, so any comparison that holds the rate fixed is partly a comparison of how well that rate suits each arm. Then noise: two runs that differ only in their random seed end about 0.003 apart in validation loss on the course’s reference model, and a single changed token in the training data moves the result about half that much, so a difference between two single runs means little until you know the spread. Then schedules, where mid-run loss is mostly a picture of the schedule, and debugging, where what matters is knowing what normal looks like. Eleven predictions use CS 312’s measured quiz-1 answers; three fictional research logs ask you to find what their authors missed.
A hyperparameter is anything chosen by hand rather than learned: the learning rate and its schedule, the batch size, the optimizer’s βs and ε, weight decay, initialization scales, warmup length, gradient clipping, dropout. CS 312’s first assignment lists about twenty for d8 alone. They matter because the conclusion of an experiment is conditional on all of them. “Method X does not help” really means “method X, at these twenty settings, did not help”, and if one of the twenty was poor for X, the experiment measured the setting.
Three cases from the lecture show the size of the effect:
Because the loss is not an arbitrary function of the twenty. Most hyperparameters barely matter near a sensible default; a few (learning rate first, then batch size, weight decay, warmup) matter a lot; and the important ones interact in structured ways (the best rate moves smoothly with the batch and the model size). So in practice you sweep one or two at a time around a known-good point, and use rules for how the optimum moves (unit 2) instead of re-searching everything at every scale.
Run the same model at several learning rates, each a factor of 2 apart, and plot the final loss against the logarithm of the rate. Almost always you get a rounded valley: too small a rate and the model does not get far in its budget; too large and the updates overshoot, the loss is noisy, and past some point training diverges. Near the bottom the curve is roughly a parabola, which gives a useful rule of thumb: missing the best rate by a factor of 2 costs about 0.01 in final loss at d8’s scale, and a factor of 4 about four times as much.
StepFun’s large study of this surface (Li et al., 2025, “Step Law”) trained thousands of models across learning rates and batch sizes at many sizes. Three findings matter for us: the surface is smooth and roughly quadratic in log learning rate and log batch size; the best learning rate moves with batch size; and both optimal values move with model size and data size.
You tuned the learning rate for d8 at batch 64. Now you train d16 (8× the parameters, same data and batch). Where is d16’s best learning rate?
Lower. With standard initialization, a weight matrix with more inputs sums more update contributions in the same direction, so the same per-entry step moves its outputs further; the stable step shrinks as the width grows. Unit 2b derives this and shows how µP removes it. Published fits find the optimum falling roughly like N−0.7 in parameters (Step Law), and CS 312’s assignment 2 measures it falling roughly as 1/width when only the width changes. How fast it moves depends on the architecture and the optimizer; that it moves does not. A rate that is right for d8 runs hot at d16.
Mechanism: unit 2b. Direction: Step Law and CS 312 lecture 2.
The shape (a valley in log learning rate that runs diagonally across batch sizes and moves left as the model grows) is what Step Law and CS 312 report. The numbers are an illustrative model, written in assets/fig-surface.js: loss = best(model) + 0.01 × (octaves from the best rate)2 + 0.003 × (octaves from batch 64)2, with the best rate growing as the square root of the batch. Its anchors are measured: the default recipe gives 2.926 at d8, d9 is 0.41 better than d4, and a rate of 0.03 costs +0.023 at d4, +0.056 at d6 and +0.122 at d16 over 0.003 (CS 312 quiz 1, Q7); given the 0.01 rule, those three gaps put d8’s best rate near 0.0045 and d16’s near 0.0027. Real valleys are steeper on the high side than the low side (instability), and warmup shifts them; the model ignores both.
The film’s batch-512 question is answered by the model’s square-root rule, a common rule for Adam: eight times the batch gives √8 ≈ 2.8 times the rate, and the nearest step of the grid is 0.012. The linear rule (8× the rate, 0.048) is the usual one for SGD. Which rule real runs follow depends on the optimizer and on how far below the critical batch you are (unit 2a).
Act 3 of the figure is the trap to remember. Comparing batch sizes at a single rate reads the side of the valley wall: at the borrowed rate, batch 512 looks clearly worse than batch 64; with each batch at its own best rate the gap is smaller. The same is true of any comparison at a fixed rate: optimizers, architectures, model sizes. Which brings us to ladders.
Big runs are expensive, so we study small ones and extrapolate. A ladder is a series of models of increasing size (d4, d6, d8, …) trained the same way; a scaling law is a curve fitted through it, usually a power law in parameters, data or compute. Scaling laws are the cheap proxy that makes large-model research possible: Kaplan et al. (2020) and Hoffmann et al. (2022, “Chinchilla”) predicted the loss of models 100× larger than any they fitted on.
They are also easy to bend. Kaplan and Chinchilla disagreed by a large factor on how to split compute between parameters and data; later work traced much of the gap to how the ladders were run (learning-rate schedules not matched to each run’s length, hyperparameters not retuned per size, and how parameters and FLOPs were counted). The Marin project’s public notes describe a ladder that looked healthy until the large run behaved differently from the small ones. A ladder is an experiment like any other: its conclusion inherits every hyperparameter held fixed along it.
Train d4 to d16 at the same rate, as a quick ladder would, and compare with each rung at its own best rate. Illustrative model from Fig. 1, anchored to CS 312’s measured gaps illustrative
Try the extremes. A rate tuned on the smallest rung (about 0.007) makes the big rungs run hot, and the ladder flattens: “width stops paying”. A rate too low for everyone does the opposite and flatters scale. Only per-rung tuning, or a parametrization that keeps the optimum in place (µP, unit 2b), measures what size buys.
CS 312’s assignment 1 ran four recipes on the d4–d9 ladder (the default; a constant rate with no decay; dropout 0.2; a rate of 0.03) and then asked what happens at d16. At the two smallest rungs, measured as the gap to the default recipe at the same size:
| Recipe | d4 | d6 | d16 |
|---|---|---|---|
| constant rate | +0.125 | +0.187 | ? |
| dropout 0.2 | +0.156 | +0.161 | ? |
| rate 0.03 | +0.023 | +0.056 | ? |
At d16 (16 layers, width 1,024, the same 614.4M tokens), how much worse than the default recipe is the run at rate 0.03?
+0.122 measured. The gap grows with size (0.023, 0.056, 0.122) because the best rate falls as the model grows, so 0.03 is further above it at each rung. It did not diverge: QK-norm and gradient clipping keep d8’s architecture stable well above its optimum.
CS 312 quiz 1, Q7(a).
And dropout 0.2, which costs about +0.16 at both d4 and d6: what is its gap at d16?
+0.058 measured. Dropout is a regularizer, and every run here sees each token once, so there is no overfitting for it to prevent; it costs capacity instead. A bigger model has capacity to spare, so the cost shrinks with size. The constant-rate recipe went the other way, growing to +0.244 at d16: without a decay at the end, the bigger model is left further from the bottom of its valley.
CS 312 quiz 1, Q7(a). The lesson for ladders: two recipes that look equally bad at small scale can move in opposite directions.
With the default recipe, d9 ends 0.41 below d4. Now remove QK-norm and use a rate of 0.006 (2× the default). What is d9 minus d4?
+0.608 measured: the ladder inverts. Without QK-norm the attention logits grow during training, and the bigger model, at a rate already too high for its size, becomes unstable while d4 is fine. A ladder run this way would conclude that bigger models are worse. The other intervention in the same question, repeating 75,000 sequences for 8 epochs, shrank d9’s advantage from −0.41 to −0.226: with repeated data the bigger model starts to memorize.
CS 312 quiz 1, Q7(b).
Training is random in several ways: the initial weights, the order of the data, and (unless you force determinism) the order of floating-point additions on the GPU. Rerun the same recipe with a different seed and the final loss changes. How much it changes sets the resolution of every experiment you run.
The default d8 recipe is trained ten times with seeds 4 to 13 (each seed sets both the initialization and the data order), on the same type of GPU. The standard deviation of the ten final validation losses is closest to:
0.0029 measured. This is the number to carry around: on d8, differences below about 0.005 between single runs are within noise, and 0.01 (the cost of a factor-of-2 learning-rate error) is about three standard deviations. Note how small it is compared with the step-to-step noise of the training loss curve, about 0.04 at batch 64 (the next question): the curve jitters ten times more than the endpoint varies.
CS 312 quiz 1, Q1(a). The mean step-to-step jitter over the last 2,000 steps was 0.0399, Q1(b).
Define a run’s jitter as the standard deviation of the training loss around its own running mean, late in training. At batch 64 it is about 0.04. Train the same 600,000 sequences at batch 256 (so a quarter of the steps). The jitter is about:
×1/2 measured: 0.0769, 0.0404 and 0.0198 at batches 16, 64 and 256. Each step’s loss is an average over the batch, and the standard deviation of an average of B independent samples falls as 1/√B: four times the batch, half the jitter.
The jitter of the curve is mostly measurement noise of a single batch, not a property of the model. Two runs can look very different step by step and end within 0.003.
CS 312 quiz 1, Q2. The dashed line is 1/√B through the batch-64 point.
Where does the seed-to-seed spread come from? Two mental models compete. In one, SGD is like Brownian motion: small perturbations keep you close, so changing a seed moves the result a little. In the other, training is chaotic: an infinitesimal perturbation grows until the two runs are as different as two random seeds. Even full-batch gradient descent, with no sampling at all, shows sudden spikes in its loss, a sign of chaotic dynamics rather than of noise.
With full determinism enabled, train d8 twice with the same seed, except that one token (of 614.4M) in the first training batch is changed. Let σ be the seed-to-seed standard deviation from above (0.0029). Averaged over five such pairs, the final validation losses differ by:
0.57σ on average measured. One token is enough to send the run on a different path that ends about as far away as half a seed change. In the course’s higher-rate variant (0.009), the per-step training loss of the two runs differed by up to 0.095 at its peak.
Two consequences. First, “same seed” does not mean “same run” once anything changes, so pairing two arms by seed removes only part of the noise. For two independent runs of spread σ, the expected gap is about 1.13σ; a gap of 0.57σ corresponds to a correlation of about 0.75 between the paired runs derived, assuming normal noise. A real method change perturbs far more than one token, so expect less. Second, any change you make, however tiny, buys you a fresh draw of the noise, which is why one run per arm is never enough.
CS 312 quiz 1, Q8.
With the spread known, the arithmetic is standard. If each arm has n runs with standard deviation σ, the difference of the two means has a standard error of σ√(2/n), reduced by a factor √(1−ρ) if the arms are paired with correlation ρ. Two less obvious facts are worth seeing in numbers: how often a design picks the wrong winner, and how little a handful of runs tells you about σ itself.
Arm B is truly better than arm A by Δ. Seed noise is d8’s σ = 0.0029 measured; the rest is exact Gaussian arithmetic derived
Read the last box with two runs per arm: the true spread could be anywhere from 0.45× to 32× what the two runs show. A “noise floor” measured from two seeds is almost no information, and a decision rule built on it (“anything above 2σ is real”) inherits that uncertainty. Ten runs narrow it to 0.69×–1.83×. On d8 the course has done this for you; on a new model or a new pipeline, measuring σ with 4–10 seeds is one of the cheapest and most valuable experiments you can run.
Pairing matters even more for evaluation than for seeds. Two models evaluated on the same validation sequences share most of their per-sequence variation, because most of it is about how hard each sequence is, not about the model. A paired comparison (per-sequence differences, then their mean and standard error) is often far more sensitive than comparing two independent averages whose error bars overlap. The lecture’s point: overlapping error bars do not mean an insignificant difference when the samples are paired.
d8’s default schedule warms the rate up linearly over the first 1% of steps, then decays it linearly to zero. Both ends do work. Warmup protects the first steps, when Adam’s second-moment estimates are poor and the network is far from any stable region; the decay at the end removes the noise of the last steps, letting the parameters settle into the bottom of the valley they have been bouncing around in.
Remove the warmup from the default recipe (start at the peak rate 0.003 on step one) and change nothing else. Final validation loss changes by:
About +0.1 measured (+0.097, +0.111, +0.094 over three seed pairs), more than 30σ, from 94 steps of warmup (1% of training). The damage happens in the first few hundred steps and is never repaired.
CS 312 quiz 1, Q3.
A popular alternative to decaying all the way is warmup–stable–decay (WSD): hold the peak rate for most of training and decay only at the end. Its appeal is that the stable phase can be extended or branched, so one long run gives checkpoints for many training lengths.
WSD with the default warmup, then decaying linearly to zero over only the last fraction X of steps. What is the smallest X that ends within 0.02 of the default linear schedule?
20% measured. The gaps to linear decay were 0.085, 0.051, 0.030 and 0.009 for decay fractions of 2%, 5%, 10% and 20%.
A short decay is not enough to settle from the peak rate. And note what this means for comparisons: two runs compared during their stable phase are compared before most of their loss improvement has happened, and the improvement each gets from its decay need not be equal. Compare after decay, or decay both from the same point.
CS 312 quiz 1, Q4.
Triple the peak rate to 0.009 and lengthen the warmup to 5%. The final validation loss (default: 2.926) is:
2.9165 measured, better than the default. The default 0.003 is not quite d8’s best rate, and a longer warmup lets a higher one work. At 0.027 the same 5% warmup gave 2.9745, and 20% warmup brought it back to 2.9563: the higher the peak, the longer the warmup it needs. Warmup and the learning rate interact, so tuning one at the other’s default can give the wrong answer for both.
CS 312 quiz 1, Q5.
When a run is worse than expected, the loss curve tells you less than you would like. The lecture’s debugging kit:
Four d8 runs cross the schedule (linear decay or constant rate) with weight decay (0.1 or 0). Each logs the total parameter norm every step. In which runs does the parameter norm end below its peak, rather than at it?
Only linear decay with weight decay 0.1, which ends 3–15% below its peak measured; the other three end at their maximum. Without weight decay, nothing pulls the norm down. With weight decay, the norm heads for an equilibrium where the growth from the updates balances the decay’s pull, and that equilibrium scales like the square root of the learning rate over the weight decay (unit 2a derives it). At a constant rate the equilibrium stays put and the norm approaches it from below, so it ends at its maximum. With a decaying rate the equilibrium itself sinks during training, and the norm follows it down. And in none of the four did the gradient norm double over training, so a slowly rising gradient norm is not by itself a sign of trouble at this scale.
CS 312 quiz 1, Q6. Unit 2a explains the equilibrium.
Each log below is fictional, written for this course, with a conclusion in it. Tick every objection you would raise before accepting the conclusion, then check. Some options are real problems; some sound reasonable but are not what is wrong here.
What would you challenge before accepting the recommendation?
Three independent mechanisms each bend a ladder’s top in the same direction, so the observed flattening is what you would expect even if width kept paying. The repair: tune the rate at two or three rungs (or use µP), scale tokens with parameters or state the data budget explicitly in the conclusion, and put two or three seeds on the top rungs.
What would you challenge?
Here nothing in the log says the result is wrong; it says the design cannot tell. The cheapest fix also has the most value: six more baseline seeds give a usable σ for every later decision, and pairing seeds across arms removes part of the noise of each comparison.
What would you challenge?
A smoke run is a harness check, not an experiment: it answers “does it run”, not “which is better”. To test warmup, run the full schedule (or a shorter one that still decays to zero) for both arms, at the peak rate you intend to use, and compare final validation loss.
CS 312’s first quiz is online as an interactive page that grades itself in your browser; the answer slides are on the course page. Take it cold before or after this unit. Every question is covered above:
| Q | About | Measured answer | Here |
|---|---|---|---|
| 1 | Seed spread; step-to-step jitter | 0.0029; 0.040 | §4 |
| 2 | Jitter at batch 16 and 256 | ×2; ×½ | §4 |
| 3 | No warmup | +0.1 | §6 |
| 4 | WSD decay fraction | 20% | §6 |
| 5 | Rate and warmup together | 2.917 / 2.975 / 2.956 | §6 |
| 6 | Parameter and gradient norms | only linear + WD falls | §7 |
| 7 | Recipes at d16; ladder benders | +0.12 / +0.06 / +0.24; +0.61 | §3 |
| 8 | One changed token | 0.57σ | §4 |
Set up as in unit 0 on the a1_release branch. The assignment’s seven problems each have a launcher in experiments/a1_basics/, run with module syntax, for example uv run python -m experiments.a1_basics.p3_measuring_variation. The README’s first exercise is a learning-rate sweep: uv run python -m experiments.lr_tuning.launch_lr_tuning, then uv run python -m experiments.lr_tuning.plot_lr_tuning.
If you run only one thing, make it problem 3 (measuring variation, about 25 d8 runs, around $12 at the course’s estimate): it gives you your own σ on your own hardware, which every later comparison needs. None of this has been run for this page.
Each method has its own best rate, and the loss rises roughly quadratically in log rate away from it. At one shared rate you measure how far each method is from its own optimum as much as how good it is.
The rate is too high for the bigger rungs (the optimum falls with size); a fixed token budget gives bigger rungs fewer tokens per parameter; and the top rungs’ differences are small enough for seed noise to matter.
Jitter is mostly the sampling noise of one batch (it falls as 1/√B), about 0.04 on d8 at batch 64; the seed-to-seed spread of final validation loss is about 0.003. They measure different things.
Very little: with two runs the 95% interval for the true σ runs from about 0.45× to 32× the sample value. Measure with 4–10 seeds.
Training is chaotic: even one changed token sends the run on a different path, ending about 0.57σ from its twin on d8. Pairing keeps some correlation (about 0.75 for a one-token change, by that measurement), less for a larger change.
Most of the improvement arrives during the decay, and how much each run gains from it need not be the same. Compare after decay, or decay both from the same point.
The norm of a weight matrix whose output is normalized: scaling it does not change the function, so its growth only changes the effective learning rate, not the model’s outputs.