GRPO group scoring — Book Labs

35 Book labs · Post-training

GRPO: the group is the baseline

From Chapter 38 of Large Language Models from the Ground Up. To decide which attempts to reinforce, older RL kept a whole extra value network to estimate "par." GRPO deletes it: sample a group of answers to one problem, and let the group's own mean score be par. Each answer's advantage is how far above or below par it landed.

Interactive

Score a group of reasoning traces

The policy samples G answers to one verifiable problem. A checker stamps each reward 1 if the final answer is right, 0 if wrong. Then GRPO computes A_i = (r_i − mean) / std — the same number for every token of that attempt. Green traces get pushed up, red pushed down.

Problem (verifiable): Compute 23 × 47. Answer key: 1081

Group size G = 4

Group mean (par)

–

Group std

–

Pushed up

–

Pushed down

–

Degenerate group — zero learning signal. Every reward equals the mean, so every advantage is exactly 0: no token moves. As the book puts it, an all-correct or all-wrong group teaches nothing — all learning signal comes from problems the model sometimes solves, which is why these pipelines curate difficulty and feed problems at the edge of the model's ability.

1 model

No learned value function. The group of samples is its own baseline, so GRPO keeps only one network in play — versus PPO's four (policy, reference, reward, value). That deleted network is the entire point: "checker score minus group mean" stands exactly where "reward-model score minus value estimate" used to.

What to notice

This mirrors the book's worked example exactly. Rewards 1, 0, 0, 1 give mean 0.5, and advantages +0.5, −0.5, −0.5, +0.5; dividing by the group std (here 0.5) normalizes them to +1, −1, −1, +1 — the ranking is unchanged, and it's these signs that decide push-up vs push-down. Watch the two edge cases: a 1,1,1,1 group has mean 1 and every advantage 0; a 0,0,0,0 group is the same from the other side. Both flatten to no signal. That is why GRPO can drop a full network — the group mean is the baseline — and why "the frontier is where the gradient is": training lives on the problems the model gets right only some of the time.

← Back to all labs