The learning-rate playground
From Chapter 9 of Large Language Models from the Ground Up. Training is a hiker in fog: feel the slope, step downhill, repeat. Here the whole model is a single dial w and the loss is the book's own valley, L(w) = (w − 3)². Pick a step size and watch the exact update w ← w − lr × 2(w − 3) either glide to the floor, zig-zag, bounce forever, or explode.
Roll the ball down the valley
The ball starts at w = 0 (the chapter's starting point). Press Step to take one gradient-descent step, or Play to animate. The dotted trail is every setting the dial has visited.
Set lr = 0.1 and Step three times: the dial marches 0 → 0.6 → 1.08 → 1.464 and the loss falls 9 → 5.76 → 3.69 → 2.36 — the exact figures from the chapter, and the steps shrink on their own as the floor flattens. Now try lr = 0.5: one perfect leap straight to w = 3. Try lr = 1.0: the hiker soars over the valley and bounces 0 → 6 → 0 → 6 forever, never touching bottom. Nudge it to 1.05 and each leap lands higher than the last — the loss climbs and training explodes. Too small crawls; too big diverges; the whole art is the knob in between.