The grading loop
From Chapter 10 of Large Language Models from the Ground Up (with the loss formula from Chapter 9). Every guess is graded by one number: the model puts a probability on the true next word, and the loss is −ln(p) — the natural log of that one probability, nothing else. Flip between model states and watch what the grade rewards, and what it punishes savagely.
Grade a next-word guess
The true next word is Paris. Pick a model state, or drag the slider to set how much probability lands on the truth. The loss and the −ln curve update live.
Line the four states up by their grade. Untrained spreads its guess evenly over all 50,257 words, so p(Paris) = 1/50,257 and the loss is ln(50,257) ≈ 10.82 — exactly the “loss around 10 or 11” the book quotes for hour zero. Hedging gives Paris 0.10 for a loss of 2.30; getting confident & right (Paris 0.90) drops it to 0.11, a 20-fold smaller penalty. But confident & wrong — 0.90 on London, just 0.01 on Paris — is walloped with 4.61: twice the penalty of honest hedging. That steep wall near zero is the whole personality of the loss. It teaches the model, above everything else, never to be certain and wrong — hedging is survivable, confident error is not.