Continuous batching: the bus vs the taxi rank
A serving GPU has a fixed number of "seats" — sequences it can decode at once. The question is what happens when riders finish at different times. Static batching is a bus: everyone boards together and nobody leaves until the last passenger reaches the end of the line. Continuous batching is a taxi rank: the moment a seat empties, the next waiting request takes it. Watch the wasted seats pile up on one side and vanish on the other.
Static (bus)
Continuous (taxi)
Same request stream feeds both simulations (seeded — hit Reseed for a new arrival pattern). Requests arrive at random steps with answer lengths skewed short with the occasional long grinder, exactly the mix that makes the bus waste seats. Backlog waiting: 0.
What to notice
- On the bus, once a short answer finishes its cell turns grey — the seat is held hostage until the longest rider in the batch reaches the end of the line. Whole columns go grey while one seat grinds on.
- On the taxi rank, a finished seat is coral again next step with a new rider. Grey almost never appears.
- The utilisation gap is the whole point: the bus typically lands near 40–55%, the taxi rank above 90% — and utilisation is throughput, so the taxi produces roughly twice the tokens per second on the very same silicon.
- This is why every serious serving stack in 2026 (vLLM, SGLang) rebuilds the batch every single token step.
This is the continuous-batching idea from Chapter 40 of Large Language Models from the Ground Up. ← Back to all labs