The BPE merge machine
From Chapter 18 (and Chapter 28) of Large Language Models from the Ground Up. Byte-Pair Encoding builds a vocabulary bottom-up: it starts from single characters and, greedily, fuses the most frequent adjacent pair into one new token — again and again. Watch it happen.
Train a tokenizer, one merge at a time
Edit the corpus, then press Merge next pair. The most frequent adjacent pair lights up everywhere, fuses into a colored token, and joins the vocabulary. Every merge shrinks the total token count.
| Pair | Count |
|---|
Frequent letter-teams become single tokens. "l" + "o" → "lo" first (it appears in every word), then "lo" + "w" → "low". Common words end up as one token; rare words stay in pieces. That is exactly why a real tokenizer spells strawberry as a couple of chunks — the model never sees the letters, so it can't count them. Watch the token count fall with each merge: fewer, bigger tokens is the whole point.
Tie-break: when two pairs are equally frequent, the one seen first (left-to-right, top-to-bottom) wins — the same deterministic rule a real BPE trainer uses.