ML Systems & GPU Performance

How the machine actually executes your code, and how to tell what limits a kernel. Everything after this rests on it.

1. 01The job, the machine, and your bench~45 min · 1 widget · lab 0
2. 02The execution model: grids, blocks, warps~65 min · 6 widgets · film · lab 1
3. 03The memory hierarchy: coalescing, shared memory, banks~55 min · 5 widgets · lab 2
4. 04The roofline: compute-bound, memory-bound, overhead-bound~45 min · 2 widgets · lab 3
5. 05Profiling with Nsight: reductions and transposes~55 min · 3 widgets · lab 4
6. 05bInterlude (optional): the hardware underneath — pipes, DRAM and clocks~50 min · 2 figures · lab 4b

The handful of kernels that make up a transformer, and just enough machine learning to know what they compute.

1. 06Neural networks as a workload~50 min · 3 widgets · lab 5
2. 07Matmul I: tiling and shared memory~60 min · 2 widgets · lab 6
3. 08Matmul II: register blocking, warp tiling, tensor cores~70 min · 2 widgets · lab 7
4. 09Softmax, layernorm, online softmax and fusion
5. 10The transformer from the GPU’s point of view
6. 11Lab: a GPT-2 forward pass in CUDA

The single most optimized operation in modern ML, and the two phases of inference that stress it differently.

1. 12Triton: kernels at the block level
2. 13Inference arithmetic: FLOPs, bytes, KV cache, prefill vs decode
3. 14FlashAttention 1 → 2
4. 15Decode attention: split-K, FlashDecoding, KV-cache kernels

From one fast forward pass to a server that handles many users at once without blowing its latency budget.

1. 16Quantization: INT8, INT4, FP8, FP4 and dequant fusion
2. 17Anatomy of an inference engine: paged KV, continuous batching, CUDA graphs
3. 18Speculative decoding and benchmarking methodology
4. 19Reading vLLM and llama.cpp; your first PR

How kernels get written when nobody writes them by hand, and how the fastest hand-written ones are built.

1. 20PyTorch internals and torch.compile / Inductor
2. 21ML compilation: IRs, scheduling, autotuning
3. 22CUTLASS and CuTe
4. 23Datacenter GPUs: TMA, WGMMA, warp specialization

When one GPU is not enough: moving tensors between GPUs, and splitting a model across them.

1. 24Collectives and NCCL
2. 25Parallelism for inference and disaggregated serving
3. 26Capstone, and where to go next