Reward hacking — the judge gets played
From Chapter 36 of Large Language Models from the Ground Up. In RLHF the reward model is "a lossy, imperfect compression of human taste" — a stand-in judge. Optimize against it hard enough and the policy stops improving the thing and starts optimizing the measurement. Drag the training pressure right and watch the measured reward keep climbing while the real quality peaks, then falls — through bloat, tics, and gibberish.
Push the policy away from the base model
The slider is optimization pressure — how far RL has dragged the policy from the frozen reference model, measured as KL divergence. Watch both curves and the sample output at the current point. Then toggle the KL penalty — the book's leash — and see the collapse held off.
Goodhart's law: "when a measure becomes a target, it ceases to be a good measure." The reward model is a copy of human judgment, and every copy has flaws; the optimizer is a flaw-finding machine that reinforces whatever scores highest, with no opinion about why. So the reward climbs to record highs while the model can forget how to write — first bloating answers ("ask for the time, receive a treatise"), then pressing in verbal tics ("Great question!", "I hope this helps!"), then collapsing into fluent gibberish "that the broken judge adores." In practice teams watch three curves — reward, KL, and generation quality on held-out prompts (the only curve that can't lie). The fix is never a smarter optimizer; it's a constraint. The β·KL leash anchors the policy to "the land of sensible language" near the reference, where the reward model's scores are still trustworthy — capping drift keeps you near the quality peak instead of chasing the reward off a cliff.