Overfitting — homework score vs. exam score
From Chapter 25 of Large Language Models from the Ground Up. During training you watch two losses: the train loss, measured on text the model studies, and the validation loss, measured on held-out text it never sees. If both fall together, the model is genuinely learning the language. If train keeps dropping while validation stalls and creeps back up, the model has started memorizing its textbook. That failure is called overfitting, and the gap between the two numbers is how you spot it.
Train a model, watch the two curves
Pick a model size and how much training data it gets, then press Play to run the steps. A big model on little data is the classic recipe for overfitting — watch the validation curve bottom out and bend upward while the train curve keeps sliding down.
Loss starts at 4.17 (random guessing among 65 characters, ln 65). A healthy run — like the book's real 3,000-step run — lands near train 1.38 / val 1.59, "with validation sitting a little above train." That thin gap is normal. A widening gap is not.
"A student who studies past exam papers can do two different things: learn the subject, or memorize the answer sheets." The validation set is the exam — a paper the model has never seen. When the two curves stay close, the model learned the subject. When they open like scissors — val bending upward while train keeps falling — the model is "reciting its textbook," and more training makes it worse at everything except the pages it memorized. The book's fix is blunt: stop, or get more data, or raise the dropout. Try a Large model on Small data to force the scissors open; then widen the data and watch the gap shrink and the divergence get pushed off the chart.