The pipeline bubble
From Chapter 46 of Large Language Models from the Ground Up. Deal a model's layers out to S GPUs like an assembly line and one batch keeps all but one station idle. The fix is many micro-batches fed nose-to-tail — but the ramp-up and drain leave an unavoidable idle wedge, the pipeline bubble. Its size is (S−1) / (M+S−1).
GPipe schedule: all forwards down, then all backwards up
Set the number of pipeline stages S and micro-batches M, then Play. Forward tiles flow diagonally down the stages; backward tiles flow back up. Hatched cells are the bubble — a GPU with nothing to do. The counted idle fraction (idle cells ÷ all cells) is checked against the analytic formula live.
With S = 4 stages, the chapter's numbers fall straight out of the Gantt: M = 1 micro-batch is 75% idle — the do-nothing disaster; M = 8 drops to ≈27%; M = 32 reaches ≈8.6%, the regime real runs target. The idle wedge is the ramp-up staircase plus its mirror at the drain — pure overhead where some station always waits. More micro-batches dilute that fixed wedge across more useful work, which is exactly why frontier runs use enormous global batches: they need enough micro-batches to keep long pipelines full. The counted-vs-analytic check confirms the drawn schedule really does obey (S−1)/(M+S−1) — the two numbers agree to the digit.