The vocabulary seesaw
From Chapter 2 of Large Language Models from the Ground Up. The same sentence, chopped at four granularities. Slide from single characters up to whole words and watch the seesaw tip: a tiny menu means long sequences; a huge menu means short sequences but a fortune spent on rare words. Every split below is computed live by greedy longest-match over a hand-built vocabulary — no faking.
Chop a sentence four ways
Pick a sentence, then drag the granularity slider (or click a stop). Tokens recompute live; the leading space that travels with each word is shown as ·.
Sequence length
Menu size & rare-word risk
Slide to Characters and read the count: "The cat sat on the mat." becomes 23 tokens — exactly the figure in Chapter 2, and it means 23² = 529 attention comparisons for six words of meaning, versus 7² = 49 as whole words. Slide to Large subword and the grandmother sentence lands on the book's own nine-token split — My · grandmother · unbeliev · ably · tweeted… — common words kept whole, the rare "unbelievably" and "cryptocurrency" assembled from fragments, and every word still spellable. That's the compromise every modern LLM makes: not the shortest sequence, not the smallest menu, but the row where both problems are bearable at once.