CS 312 starts from a claim borrowed from the history of science: you understand a system when you can predict what it will do in an experiment you have not run yet. So the course grades almost nothing except predictions, all about one small language model trained on one dataset. This unit sets up the three things every later unit relies on. First, the method: what a good prediction looks like, and why calibration (being right 70% of the time when you say 70%) matters as much as being right. Second, the reference model, d8, with its sizes and budget written out, so every later number has something to attach to. Third, a protocol for directing experiments run by someone else, human or agent: write the plan as a prediction question, seal both sides’ predictions before the run, check the harness against a known result, and audit every conclusion with five questions. It ends with how to run d8 yourself on your own Modal account.
Most deep-learning knowledge is held as recipes: use AdamW with β2 = 0.95, warm up for a few percent of training, decay the learning rate to zero, put weight decay on matrices but not on norm gains. Recipes work until you change something. The moment you swap the optimizer, double the batch, or train a model four times bigger, you need to know which parts of the recipe still hold, and a recipe cannot tell you.
CS 312’s answer is to treat training the way astronomy treated the sky: a system you are expected to forecast. Each quiz describes experiments on the same small transformer (“we train the default model with the residual connection mixed half from the previous layer and half from two layers back; what is the loss difference after tuning the learning rate for each?”) and asks for a number or a ranking, with a rationale. The assignments that prepare for the quizzes count for nothing on their own; you earn back up to 75% of a missed question by citing an experiment you ran. The design is honest about what it wants: a working model of training in your head, built by running things.
CS 312 published how the first quiz went for people who had not run the assignment’s experiments. About what fraction of the questions did the course’s own instructors get right without running anything?
About 70%. A frontier LLM answering without experiments scored about the same, around 70%. The same LLM with the assignment’s experiments and report in hand scored close to 100%.
Two lessons sit in that table. Experience gets you most of the way, and the last 30% is exactly the part that only an experiment reveals: magnitudes, crossovers, which of two plausible effects wins. And an agent that can run experiments answers these questions very well, so the scarce skill for someone directing an agent is not answering. It is choosing the question and checking that the experiment behind the answer measures what it claims to.
measured CS 312 quiz 1 answer slides, “how did people do”.
A useful prediction has four parts, and the quizzes force all four:
Every unit here has sealed predictions in that format. You pick an answer and a confidence, lock it (it cannot be changed afterwards), then reveal. Everything goes into your ledger, which shows your calibration: how often you were right at each confidence level. Five revealed predictions per level is the minimum before the reading means anything.
Numbers on these pages carry a tag. measured means someone ran it, usually CS 312’s staff for their answer slides. derived means it follows by arithmetic from stated facts, and you can check it. illustrative means a toy model chosen to show the shape of an effect; its exact values mean nothing.
Everything in CS 312 happens on one small model, so that predictions can be checked cheaply and compared across the class. It is a standard modern decoder-only transformer, the same design as the open-weight models of 2024–2026 shrunk to fit a 15-minute run.
Architecture. Pre-norm blocks with RMSNorm, rotary position embeddings (RoPE), QK-norm (the queries and keys are normalized before the dot product, which keeps attention logits bounded), a SwiGLU MLP with hidden size 3.5× the width, and separate (untied) input and output embeddings.
Data. Web text from DCLM, tokenized with a 4,096-token vocabulary, pre-packed into sequences of 1,024. A run reads each sequence once.
Default recipe. AdamW with β = (0.9, 0.95) and ε = 10−8; peak learning rate 0.003; warmup over the first 1% of steps, then linear decay to zero; batch 64 sequences (65,536 tokens), so 9,375 steps; weight decay 0.1 on the weight matrices only (norm gains and the like are masked out); gradient clipping at norm 1.0. On one H100 a run takes 10–15 minutes and costs about half a dollar on Modal at the course’s estimate.
d8 has 30.4M non-embedding parameters and trains on 614.4M tokens. Compared with the “compute-optimal” ratio from the Chinchilla paper (about 20 tokens per parameter), how is d8 trained?
614.4M / 30.4M ≈ 20.2 tokens per parameter derived. d8 sits at the Chinchilla point: for its compute budget, neither a bigger nor a smaller model would reach a much lower loss. The point matters later. Many effects (how much weight decay helps, whether repeating data hurts, how the best learning rate moves) depend on whether a model is under- or over-trained, and a conclusion drawn at 20 tokens per parameter need not hold at 200, where most production small models now live.
Training compute ≈ 6·N·D = 6 × 30.4M × 614.4M ≈ 1.1 × 1017 FLOPs derived, before attention.
Many questions are about scale: does an effect grow or shrink as the model grows? CS 312 uses a one-knob family, dX: X layers, width 64·X, X heads of 64. Depth and width grow together, so the shape stays the same. Each layer holds about 4·w2 parameters in attention (queries, keys, values, output) and 3 × 3.5·w2 = 10.5·w2 in the SwiGLU MLP, so 14.5·w2 per layer.
| Model | Layers | Width | Non-embedding params | × d8 |
|---|---|---|---|---|
| d4 | 4 | 256 | 3.8M | 0.125 |
| d6 | 6 | 384 | 12.8M | 0.42 |
| d8 | 8 | 512 | 30.4M | 1 |
| d12 | 12 | 768 | 102.6M | 3.4 |
| d16 | 16 | 1,024 | 243.3M | 8 |
Parameters grow as X3 (X layers of size X2) derived: doubling X multiplies the model by 8. At a fixed token count the compute grows the same way, so d16 at d8’s data costs about 8 d8 runs, and d16 at its own Chinchilla data (about 4.9B tokens) about 64.
6 × (0.125 + 1 + 8) ≈ 55 d8 runs, around $27 at the course’s price. The d16 rung is 88% of the bill. This arithmetic is why scale experiments are where shortcuts get taken (one learning rate for every rung, one seed), and why the next unit spends time on what those shortcuts do to the conclusion.
Suppose an agent (a research assistant, a colleague, an LLM with a GPU budget) runs your experiments and reports back. CS 312’s own table says such an agent can answer prediction questions very well once it has run the experiments. What it cannot do reliably is check its own experimental design: when it reviews its own work, the review shares the blind spots of the design. A confound it did not think of while planning is a confound it will not think of while reviewing. Your job as director moves from “run the experiment” to three things.
Write the plan as a prediction question. Before any run, the plan states the setup, the quantity, how each arm is tuned, how many seeds, and the decision rule, in a form a CS 312 quiz could print. Vague plans (“see if Muon helps”) produce conclusions that cannot be wrong, and so cannot be checked.
Seal two predictions. You and the agent each write down the expected result and a confidence before launch, without seeing the other’s. Afterwards you have two calibration records. When the agent’s 90% predictions keep failing in one area, you know where to stop taking its word. When yours do, you know where to read.
Calibrate the harness on something known. Before a new pipeline answers a new question, it must reproduce an answer you already know: the default d8 loss within the seed noise, the learning-rate curve with its optimum where the course found it, a known bad change (no warmup at a high learning rate) coming out bad. A harness that cannot reproduce known results will produce unknown ones with the same confidence.
Here is a plan template that bakes the three together. Paste it into the instructions you give an agent and require it filled in before launch.
EXPERIMENT PLAN (filled in and sealed before any run)
Question: Does X change final validation loss of MODEL at BUDGET tokens?
Arms: A = baseline exactly as-is. B = baseline + X. List every other difference (should be none).
Tuning: learning rate swept for EACH arm over the same grid (factor-of-2 steps,
at least 4 points, optimum inside the grid, not at an edge).
Noise: sigma from >= 4 seeds of A; arms paired (same data order and init seed).
Readout: final val loss after the full decay; plus each arm's LR curve.
Predictions: director: delta = ... (confidence ..%) agent: delta = ... (confidence ..%)
Decision rule: adopt X if delta < -2*sigma_paired at each arm's best LR,
and the sign holds at 2x and 0.5x that LR.
Harness check: first run reproduces KNOWN RESULT within NOISE.
The numbers in it (four seeds, a factor-of-2 grid, two standard deviations) are defaults, not laws. Unit 1 shows where each one comes from and when to change it.
Most wrong conclusions in training experiments come from a short list of causes, and the course’s first two quizzes are built on them. When a result arrives, ask:
Which objections would you raise before accepting the promotion and the next step?
The repair is cheap: tune the baseline’s learning rate (4–6 runs), then add the three changes one at a time at each arm’s tuned rate, each with paired seeds. The self-review missed it because the agent was checking whether the effect was large, which it was, and not whether the design could say where it came from.
Everything in this companion can be read without running anything: the debriefs use the course’s measured answers. If you want to run the experiments, the course repository is public and uses Modal for GPUs. Enrolled students read the data from a class volume; you need your own copy.
The steps below follow the course repository’s README and its modal_utils.py as of October 2026. Nobody has run them for this page yet. The data-volume step uses a helper written for this companion. Expect to adjust something.
a1_release, a2_release, …) of deep-learning-alchemy/assignments. Install uv, then:
git clone -b a1_release https://github.com/deep-learning-alchemy/assignments
cd assignments
uv python install 3.10
uv syncuv run modal setup (browser login), then uv run modal config set-environment main (or your environment’s name), and put the same name in CONFIG_MODAL_ENVIRONMENT near the top of utils.py.hard-dl-dclm-v1 from the class environment, which you cannot read. Fill your own with fill_volume.py (about 20 GB from Hugging Face), then point the launchers at it:
uv run modal run /path/to/fill_volume.py
export DL_ALCHEMY_MODAL_SHARED_DATA_ENVIRONMENT=main
export DL_ALCHEMY_MODAL_SHARED_DATA_VOLUME_NAME=dl-alchemy-datadl-alchemy-wandb; offline runs do not. Check whether the launcher you use sets wandb_online.uv run python -m experiments.smoke.modal_smoke_train. It submits a detached job and prints a link to the logs. About 10–15 minutes on an H100.Because predicting an outcome before running it requires a working model of how training behaves, and that model is what transfers to new problems. Running an experiment only requires following instructions. The assignments still matter: they are where the model gets built, and citing them earns back partial credit.
A precise setup (including how each arm is tuned), a quantity, a confidence, and a mechanism.
The review is done with the same assumptions that produced the design, so a confound missed in planning is usually missed in review. Self-review is good at checking execution (did the run finish, are the numbers what the log says) and poor at checking the question.
About 103M outside the embeddings: 12 layers of 14.5 × 7682 ≈ 8.6M each. Parameters grow as X3 along the ladder, so d12 is (12/8)3 ≈ 3.4 times d8.
Reproduce a known result: the default model’s loss within seed noise, or a known learning-rate curve. That calibrates the harness, not the hypothesis.
modal_utils.py, data.py): the default recipe, data layout and Modal setup.