A model only does math. Step one: turn text into integers.
Every LLM starts by chopping text into tokens and mapping each to an integer id. We build the simplest scheme -- one id per character -- on the real dataset, then look at what production models do.
A neural network only does arithmetic; it cannot read the letter "h" directly. So the first step of every language model is to translate text into numbers. Each small piece of text is given a token id, which is simply a whole number that stands in for it. A useful image is a cloakroom: every distinct item is given a numbered ticket, the same item always receives the same number, and the ticket can later be exchanged back for the item.
We will start with the simplest scheme, one id per character, and then look briefly at what production models do, a method called subword BPE that is explained later in this notebook.
Char-level tokenizer
Type anything. Each character becomes one token (␣ = space, \n = newline).
# Colab setup -- fetch the files this notebook needs.
# (Does nothing when run locally in the course folder.)
import os, urllib.request
BASE = ("https://raw.githubusercontent.com/waze"
"emlabs/llm-book-code/main/")
for f in ['data/input.txt']:
if not os.path.exists(f):
d = os.path.dirname(f)
if d: os.makedirs(d, exist_ok=True)
urllib.request.urlretrieve(BASE + f, f)
print("downloaded", f)
Line by line: what each line does
import os, urllib.request: two standard-library toolboxes: checking files on disk, and downloading from the web.BASE = (...): the web address of this course's folder on GitHub, split over two lines (Python glues adjacent strings together).for f in [...]: loop over the file names this notebook needs.if not os.path.exists(f): only download what is missing -- running locally, everything already exists, so nothing happens.os.makedirs(d, exist_ok=True): create the folder for the file if it has one (exist_okmeans don't complain if it's already there).urllib.request.urlretrieve(...): download the file and save it under the same name here.
from pathlib import Path
text = Path("data/input.txt").read_text()
print("dataset characters:", len(text))
print("----- first 200 chars -----")
print(text[:200])Line by line: what each line does
from pathlib import Path: borrow just thePathtool (for working with files) from Python's standard library.text = Path("data/input.txt").read_text(): open that file and hand back its entire contents as one long string.textnow holds all ~1.1 million characters of Shakespeare.len(text): count the characters in the string.text[:200]: slice the first 200 characters (positions 0 up to, but not including, 200) for a quick peek.
A character-level tokenizer
The vocabulary is just the list of distinct characters that appear in the text (here there are 65 of them: the letters, the space, punctuation, and the newline). We number them and build two small lookup tables:
stoi, short for "string to int": give it a character and it returns that character's id.itos, short for "int to string": the reverse, turning an id back into its character.
encode runs a whole string through stoi to produce a list of ids, and decode runs ids back through itos to rebuild the string. The assert line is a self-check that the round trip is lossless, meaning that encoding and then decoding returns exactly what you started with. For example, encode("hi there") becomes [46, 47, 1, 58, 46, 43, 56, 43], where the 1 in the middle is the id for the space.
chars = sorted(set(text))
vocab_size = len(chars)
print("vocab size:", vocab_size)
print("vocab:", "".join(chars))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)
print("encode('hi there') ->", encode("hi there"))
print("round-trip:", decode(encode("hi there")))
assert decode(encode("First Citizen")) == "First Citizen"
print("round-trip OK")Line by line: what each line does
chars = sorted(set(text)):set(text)throws away duplicates, leaving the distinct characters;sortedputs them in a fixed order. That ordered list of 65 characters is the vocabulary.vocab_size = len(chars): how many distinct characters there are: 65.stoi = {c: i for i, c in enumerate(chars)}: build a lookup table ("string to int") in one line:enumeratewalks the list handing out (position, character) pairs, and each character is filed under its position. Sostoi['a']gives a's id.itos = {i: c for c, i in stoi.items()}: the reverse table ("int to string"), built by flipping every (char, id) pair.encode = lambda s: [stoi[c] for c in s]: a one-line function (lambda): walk a string, look each character up instoi, collect the ids into a list.decode = lambda ids: "".join(itos[i] for i in ids): the reverse: turn each id back into its character and glue them into one string ("".joinconcatenates with nothing between).encode("hi there"): shows the ids; the1in the result is the space.assert decode(encode("First Citizen")) == "First Citizen": a tripwire: if encode-then-decode ever fails to return the original, the notebook stops loudly right here instead of going wrong silently later.
import numpy as np
data = np.array(encode(text), dtype=np.int64)
print("encoded dataset:", data.shape, data.dtype)
print("first 30 ids:", data[:30])
# Hold out the last 10% as a validation set (to check we're not just memorizing).
n = int(0.9 * len(data))
train_data, val_data = data[:n], data[n:]
print("train:", len(train_data), " val:", len(val_data))Line by line: what each line does
import numpy as np: the array toolbox again.data = np.array(encode(text), dtype=np.int64): encode the whole book into ids and store them as a NumPy array of 64-bit integers (dtype= what kind of number).data.shape/data[:30]: confirm it's one long sequence of ~1.1M ids, and peek at the first 30.n = int(0.9 * len(data)): the index 90% of the way through.int(...)chops it to a whole number.train_data, val_data = data[:n], data[n:]: slice into two parts: everything beforento train on, everything fromnonward held back as a validation set (the practice exam).
What real models do: subword (BPE) tokenization
Character-level tokenizing is simple but wasteful. The word "Tokenization" becomes 12 separate ids, and a single letter carries very little meaning on its own, so the model has to work harder over longer sequences.
Production language models instead use Byte-Pair Encoding, usually shortened to BPE. The idea is straightforward: scan the text, find the most common neighbouring pair of pieces, glue that pair into one new token, and repeat this thousands of times. Common chunks such as " the" or "ing" end up as a single id each. The vocabulary grows to roughly 50,000 to 100,000 tokens, but any given sentence becomes far shorter.
You can see the benefit below: the same sentence is 37 character-ids but only 7 GPT-2 tokens, and the pieces ("Token", "ization", " splits", and so on) line up with how a person would naturally break the sentence up. We will keep using character-level tokens for the rest of the course because they are clearer to follow, but this is what the production systems do.
import tiktoken
enc = tiktoken.get_encoding("gpt2") # GPT-2's actual BPE tokenizer
sample = "Tokenization splits text into pieces."
print("char-level ids :", len(encode(sample)), "tokens")
print("GPT-2 BPE ids :", len(enc.encode(sample)), "tokens ->", enc.encode(sample))
print("BPE pieces :", [enc.decode([t]) for t in enc.encode(sample)])Line by line: what each line does
import tiktoken: OpenAI's real tokenizer library (this needs installing, which is why this cell runs in the notebook, not the browser).enc = tiktoken.get_encoding("gpt2"): load GPT-2's actual trained BPE vocabulary.len(encode(sample)): our character tokenizer's count for the sentence (37).enc.encode(sample): GPT-2's ids for the same sentence: only 7, because it merged common chunks into single tokens.[enc.decode([t]) for t in enc.encode(sample)]: decode each id individually to reveal the text chunk it stands for. Note[t]:decodewants a list, so each id is wrapped in one.
Context windows and batches
We never feed the whole book in at once. It is too large, and the model only needs the recent context to predict the next character. So we cut the text into fixed-length chunks of block_size tokens. Such a chunk is called the context window: the stretch of text the model can see at one time.
There is a neat point about the labels. Within a single chunk, the target at every position is simply the next token. So one 8-character chunk quietly contains 8 training examples at once: given t predict h, given th predict a space, given th predict s, and so on. The unrolled printout below shows exactly this.
That one relationship, context in and next token out, is the entire training signal. Nobody hand-labels anything; the text itself is the answer key. This is why language models can be trained on essentially all the text in the world.
A batch is just several of these chunks stacked together (batch_size of them), so the computer can process many examples at the same time. That is all get_batch does: it picks a few random starting points, slices out the context as x, and slices out the same span shifted one step to the right as the targets y.
One chunk, eight examples
The book is too big to feed in at once, so we cut it into fixed block_size chunks. The neat part: because every position's target is just the next character, a single 8-character chunk quietly holds 8 training examples — the context grows one step at a time.
def get_batch(split, block_size=8, batch_size=4, seed=None):
data = train_data if split == "train" else val_data
rng = np.random.default_rng(seed)
ix = rng.integers(0, len(data) - block_size, size=batch_size)
x = np.stack([data[i:i+block_size] for i in ix]) # (batch, block) context
y = np.stack([data[i+1:i+1+block_size] for i in ix]) # (batch, block) the next token at each step
return x, y
xb, yb = get_batch("train", block_size=8, batch_size=4, seed=1)
print("x batch shape:", xb.shape, " y batch shape:", yb.shape)
print("\nWhat one row means (context -> next token), unrolled:")
for t in range(8):
ctx = xb[0, :t+1]
print(f" given {repr(decode(ctx)):20} predict {repr(decode([yb[0, t]]))}")Line by line: what each line does
def get_batch(split, block_size=8, batch_size=4, seed=None):: grab a training batch; the defaults mean you can just callget_batch("train").data = train_data if split == "train" else val_data: a one-line if/else picking which dataset to draw from.rng = np.random.default_rng(seed): a random-number generator; passing a seed makes the picks repeatable.ix = rng.integers(0, len(data) - block_size, size=batch_size): choose 4 random starting positions, kept far enough from the end that a full 8-long chunk fits.x = np.stack([data[i:i+block_size] for i in ix]): slice out an 8-character chunk at each start and stack the 4 chunks into a 4×8 grid. This is the context.y = np.stack([data[i+1:i+1+block_size] for i in ix]): the same chunks shifted one step right: at every position,yholds the character that comes next. That's the answer key.for t in range(8): ... print(...): unroll one chunk to show the eight (context → next-character) lessons hiding inside it.repr(decode(ctx)): show the decoded text with quotes and escape characters visible, so spaces and newlines are unmistakable.
Recap
The text is now a stream of whole-number ids, split into a training portion and a held-back validation portion, so that later we can check the model is genuinely learning the language rather than memorizing the book. We can draw (context, next-token) batches on demand. Next, notebook 03 uses these to train your first real language model.
02_tokenization.ipynb