Attention learns to look
From Chapter 25 of Large Language Models from the Ground Up. While notebook 09's GPT trained, we froze its attention maps at eleven checkpoints — the real post-softmax weights of all sixteen heads, on one probe line. At step 0 every head stares blankly at everything it is allowed to see. Scrub to step 3,000 and meet your employees: as the loss falls, the heads sharpen and take jobs nobody assigned.
At step 0 every row of every map is flat — a token with i visible characters gives each roughly 1/(i+1) of its attention, because random weights have no opinions (that flatness is the 4.17 = ln 65 loss). Watch layer 2 as you scrub: by step 3,000 its four heads have visibly specialized. Head 1 anchors on the line-opening "O" — nearly every character sends it 30–40% of its attention, a line-start head. Head 2 is the sharpest in the model: inside "Romeo" it locks onto the capital R that opens the current word with up to ~90% weight — a word-start head. Head 3 hunts punctuation, gripping the last "," or "!" — a clause-boundary head. Head 4 hugs the diagonal, blending the current and just-previous characters. This is Chapter 25's story made visible: nobody assigned those jobs — heads that look somewhere useful predict the next character better, so the falling loss curve below and the sharpening squares above are the same event seen twice.