Speculative decoding: a small model proposes
From Chapter 40 of Large Language Models from the Ground Up. Decode is memory-bound — the big model reads all its weights to make one token, wasting 99% of its arithmetic. Speculative decoding fills that idle silicon: a cheap draft model guesses k tokens, the big target model verifies all k in one wide pass, and the agreeing prefix is kept. Same output distribution, more than twice the speed.
Draft, verify, accept — one target pass at a time
Hit Step (or Play). The draft proposes k grey tokens; the target verifies them in a single stamp. The agreeing prefix turns green; the first mismatch turns red and is replaced by the target's own token; the rest are discarded. Every pass yields at least one real token.
This is the book's worked example, live. At the defaults k = 4, p = 80% the chapter computes expected accepted drafts = 0.8 + 0.64 + 0.512 + 0.4096 = 2.36, so tokens per target pass = 2.36 + 1 = 3.36 (the +1 is the target's own token, a correction or a bonus — every pass yields it). With draft steps costing a tenth of a big step, cost per pass = 1 + 4×0.1 = 1.4, giving speedup ≈ 2.4×. Let the run go long and the empirical tokens / pass above converges on 3.36. Now drag p down: acceptance collapses, and near p = 0 the chapter's arithmetic gives ~0.7× — slower than not speculating, because the draft's steps still cost real time. The step count never gets worse, but the wall-clock can — which is why real servers watch the acceptance rate and shrink k or switch speculation off. And the punchline the chapter stresses: the output is exactly the big model's distribution — speed with a quality price of precisely zero.