The five moves — the heartbeat of training
From Chapter 25 of Large Language Models from the Ground Up. Every training run — from a toy bigram to a frontier model — is the same short loop repeated: sample a batch, forward, zero the gradients, backward, step. The book names these "the five moves" (introduced in Chapter 23) and calls the repeating loop the heartbeat. Walk it once, then let it run and watch the loss fall.
One lap of the loop = one training step
Step walks the highlight one move around the cycle; a full lap commits a training step and drops a point on the loss curve. Play runs the heartbeat on its own — the same 3,000-step run from notebook 09.
Draw 32 random context windows from the training text.
Dashed lines mark the 4.17 random-guessing baseline (ln 65) and the 2.45 bigram ceiling. The real run lands at train 1.38 / val 1.59 — the curve here is calibrated to notebook 09's printed losses.
The loop never changes shape. As the book puts it, the heartbeat is "get_batch → forward (which returns the loss) → zero_grad → backward (autograd computes every gradient) → step (nudge all 824,897 knobs downhill)." Order matters: clear, compute, step — in that order, every time. Watch the loss crash from 4.17 (pure guessing among 65 characters) down through the 2.45 bigram ceiling and level off near 1.38. That plateau isn't the loop wearing out — it's the model reaching the limit of what this size and this data can learn. Same five moves run 3,000 times; that is the entire act of "training" a GPT.