{ "cells": [ { "cell_type": "markdown", "id": "48f58381", "metadata": {}, "source": "# 02 - Tokenization: turning text into numbers\n\nA neural network only does arithmetic; it cannot read the letter \"h\" directly. So the first step of every language model is to translate text into numbers. Each small piece of text is given a **token id**, which is simply a whole number that stands in for it. A useful image is a cloakroom: every distinct item is given a numbered ticket, the same item always receives the same number, and the ticket can later be exchanged back for the item.\n\nWe will start with the simplest scheme, one id per character, and then look briefly at what production models do, a method called subword **BPE** that is explained later in this notebook." }, { "cell_type": "code", "execution_count": null, "id": "colab-setup-02", "metadata": {}, "outputs": [], "source": [ "# Colab setup -- fetch the files this notebook needs.\n", "# (Does nothing when run locally in the course folder.)\n", "import os, urllib.request\n", "BASE = (\"https://raw.githubusercontent.com/waze\"\n", " \"emlabs/llm-book-code/main/\")\n", "for f in ['data/input.txt']:\n", " if not os.path.exists(f):\n", " d = os.path.dirname(f)\n", " if d: os.makedirs(d, exist_ok=True)\n", " urllib.request.urlretrieve(BASE + f, f)\n", " print(\"downloaded\", f)\n" ] }, { "cell_type": "markdown", "id": "colab-setup-md-02", "metadata": {}, "source": [ "
Line by line: what each line does\n\n
" ] }, { "cell_type": "code", "execution_count": 1, "id": "5899fece", "metadata": { "execution": { "iopub.execute_input": "2026-06-11T07:12:05.595507Z", "iopub.status.busy": "2026-06-11T07:12:05.595303Z", "iopub.status.idle": "2026-06-11T07:12:05.602665Z", "shell.execute_reply": "2026-06-11T07:12:05.601877Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dataset characters: 1115394\n", "----- first 200 chars -----\n", "First Citizen:\n", "Before we proceed any further, hear me speak.\n", "\n", "All:\n", "Speak, speak.\n", "\n", "First Citizen:\n", "You are all resolved rather to die than to famish?\n", "\n", "All:\n", "Resolved. resolved.\n", "\n", "First Citizen:\n", "First, you\n" ] } ], "source": [ "from pathlib import Path\n", "text = Path(\"data/input.txt\").read_text()\n", "print(\"dataset characters:\", len(text))\n", "print(\"----- first 200 chars -----\")\n", "print(text[:200])" ] }, { "id": "f6041896", "cell_type": "markdown", "metadata": {}, "source": "
Line by line: what each line does\n\n
" }, { "cell_type": "markdown", "id": "08a3f883", "metadata": {}, "source": "## A character-level tokenizer\n\nThe **vocabulary** is just the list of distinct characters that appear in the text (here there are 65 of them: the letters, the space, punctuation, and the newline). We number them and build two small lookup tables:\n\n- `stoi`, short for \"string to int\": give it a character and it returns that character's id.\n- `itos`, short for \"int to string\": the reverse, turning an id back into its character.\n\n`encode` runs a whole string through `stoi` to produce a list of ids, and `decode` runs ids back through `itos` to rebuild the string. The `assert` line is a self-check that the round trip is lossless, meaning that encoding and then decoding returns exactly what you started with. For example, `encode(\"hi there\")` becomes `[46, 47, 1, 58, 46, 43, 56, 43]`, where the `1` in the middle is the id for the space." }, { "cell_type": "code", "execution_count": 2, "id": "972b8cf8", "metadata": { "execution": { "iopub.execute_input": "2026-06-11T07:12:05.604617Z", "iopub.status.busy": "2026-06-11T07:12:05.604479Z", "iopub.status.idle": "2026-06-11T07:12:05.617565Z", "shell.execute_reply": "2026-06-11T07:12:05.617149Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "vocab size: 65\n", "vocab: \n", " !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\n", "encode('hi there') -> [46, 47, 1, 58, 46, 43, 56, 43]\n", "round-trip: hi there\n", "round-trip OK\n" ] } ], "source": [ "chars = sorted(set(text))\n", "vocab_size = len(chars)\n", "print(\"vocab size:\", vocab_size)\n", "print(\"vocab:\", \"\".join(chars))\n", "\n", "stoi = {c: i for i, c in enumerate(chars)}\n", "itos = {i: c for c, i in stoi.items()}\n", "encode = lambda s: [stoi[c] for c in s]\n", "decode = lambda ids: \"\".join(itos[i] for i in ids)\n", "\n", "print(\"encode('hi there') ->\", encode(\"hi there\"))\n", "print(\"round-trip:\", decode(encode(\"hi there\")))\n", "assert decode(encode(\"First Citizen\")) == \"First Citizen\"\n", "print(\"round-trip OK\")" ] }, { "id": "cca9dadd", "cell_type": "markdown", "metadata": {}, "source": "
Line by line: what each line does\n\n
" }, { "cell_type": "code", "execution_count": 3, "id": "e9a9c636", "metadata": { "execution": { "iopub.execute_input": "2026-06-11T07:12:05.619039Z", "iopub.status.busy": "2026-06-11T07:12:05.618918Z", "iopub.status.idle": "2026-06-11T07:12:05.693624Z", "shell.execute_reply": "2026-06-11T07:12:05.693200Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "encoded dataset: (1115394,) int64\n", "first 30 ids: [18 47 56 57 58 1 15 47 58 47 64 43 52 10 0 14 43 44 53 56 43 1 61 43\n", " 1 54 56 53 41 43]\n", "train: 1003854 val: 111540\n" ] } ], "source": [ "import numpy as np\n", "data = np.array(encode(text), dtype=np.int64)\n", "print(\"encoded dataset:\", data.shape, data.dtype)\n", "print(\"first 30 ids:\", data[:30])\n", "\n", "# Hold out the last 10% as a validation set (to check we're not just memorizing).\n", "n = int(0.9 * len(data))\n", "train_data, val_data = data[:n], data[n:]\n", "print(\"train:\", len(train_data), \" val:\", len(val_data))" ] }, { "id": "9970e8a4", "cell_type": "markdown", "metadata": {}, "source": "
Line by line: what each line does\n\n
" }, { "cell_type": "markdown", "id": "3590dc1f", "metadata": {}, "source": "## What real models do: subword (BPE) tokenization\n\nCharacter-level tokenizing is simple but wasteful. The word \"Tokenization\" becomes 12 separate ids, and a single letter carries very little meaning on its own, so the model has to work harder over longer sequences.\n\nProduction language models instead use **Byte-Pair Encoding**, usually shortened to **BPE**. The idea is straightforward: scan the text, find the most common neighbouring pair of pieces, glue that pair into one new token, and repeat this thousands of times. Common chunks such as `\" the\"` or `\"ing\"` end up as a single id each. The vocabulary grows to roughly 50,000 to 100,000 tokens, but any given sentence becomes far shorter.\n\nYou can see the benefit below: the same sentence is 37 character-ids but only **7** GPT-2 tokens, and the pieces (`\"Token\"`, `\"ization\"`, `\" splits\"`, and so on) line up with how a person would naturally break the sentence up. We will keep using character-level tokens for the rest of the course because they are clearer to follow, but this is what the production systems do." }, { "cell_type": "code", "execution_count": 4, "id": "cf4a4a5e", "metadata": { "execution": { "iopub.execute_input": "2026-06-11T07:12:05.695054Z", "iopub.status.busy": "2026-06-11T07:12:05.694934Z", "iopub.status.idle": "2026-06-11T07:12:18.125298Z", "shell.execute_reply": "2026-06-11T07:12:18.124697Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "char-level ids : 37 tokens\n", "GPT-2 BPE ids : 7 tokens -> [30642, 1634, 30778, 2420, 656, 5207, 13]\n", "BPE pieces : ['Token', 'ization', ' splits', ' text', ' into', ' pieces', '.']\n" ] } ], "source": [ "import tiktoken\n", "enc = tiktoken.get_encoding(\"gpt2\") # GPT-2's actual BPE tokenizer\n", "sample = \"Tokenization splits text into pieces.\"\n", "print(\"char-level ids :\", len(encode(sample)), \"tokens\")\n", "print(\"GPT-2 BPE ids :\", len(enc.encode(sample)), \"tokens ->\", enc.encode(sample))\n", "print(\"BPE pieces :\", [enc.decode([t]) for t in enc.encode(sample)])" ] }, { "id": "0c7cc304", "cell_type": "markdown", "metadata": {}, "source": "
Line by line: what each line does\n\n
" }, { "cell_type": "markdown", "id": "ab1daba5", "metadata": {}, "source": "## Context windows and batches\n\nWe never feed the whole book in at once. It is too large, and the model only needs the recent context to predict the next character. So we cut the text into fixed-length chunks of `block_size` tokens. Such a chunk is called the **context window**: the stretch of text the model can see at one time.\n\nThere is a neat point about the labels. Within a single chunk, the target at every position is simply the next token. So one 8-character chunk quietly contains 8 training examples at once: given `t` predict `h`, given `th` predict a space, given `th ` predict `s`, and so on. The unrolled printout below shows exactly this.\n\nThat one relationship, context in and next token out, is the entire training signal. Nobody hand-labels anything; the text itself is the answer key. This is why language models can be trained on essentially all the text in the world.\n\nA **batch** is just several of these chunks stacked together (`batch_size` of them), so the computer can process many examples at the same time. That is all `get_batch` does: it picks a few random starting points, slices out the context as `x`, and slices out the same span shifted one step to the right as the targets `y`." }, { "cell_type": "code", "execution_count": 5, "id": "d1e121b7", "metadata": { "execution": { "iopub.execute_input": "2026-06-11T07:12:18.126639Z", "iopub.status.busy": "2026-06-11T07:12:18.126554Z", "iopub.status.idle": "2026-06-11T07:12:18.144466Z", "shell.execute_reply": "2026-06-11T07:12:18.143984Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "x batch shape: (4, 8) y batch shape: (4, 8)\n", "\n", "What one row means (context -> next token), unrolled:\n", " given 't' predict 'h'\n", " given 'th' predict ' '\n", " given 'th ' predict 's'\n", " given 'th s' predict 't'\n", " given 'th st' predict 'o'\n", " given 'th sto' predict 'l'\n", " given 'th stol' predict \"'\"\n", " given \"th stol'\" predict 'n'\n" ] } ], "source": [ "def get_batch(split, block_size=8, batch_size=4, seed=None):\n", " data = train_data if split == \"train\" else val_data\n", " rng = np.random.default_rng(seed)\n", " ix = rng.integers(0, len(data) - block_size, size=batch_size)\n", " x = np.stack([data[i:i+block_size] for i in ix]) # (batch, block) context\n", " y = np.stack([data[i+1:i+1+block_size] for i in ix]) # (batch, block) the next token at each step\n", " return x, y\n", "\n", "xb, yb = get_batch(\"train\", block_size=8, batch_size=4, seed=1)\n", "print(\"x batch shape:\", xb.shape, \" y batch shape:\", yb.shape)\n", "print(\"\\nWhat one row means (context -> next token), unrolled:\")\n", "for t in range(8):\n", " ctx = xb[0, :t+1]\n", " print(f\" given {repr(decode(ctx)):20} predict {repr(decode([yb[0, t]]))}\")" ] }, { "id": "c351bfed", "cell_type": "markdown", "metadata": {}, "source": "
Line by line: what each line does\n\n
" }, { "cell_type": "markdown", "id": "a4b7382f", "metadata": {}, "source": "## Recap\n\nThe text is now a stream of whole-number ids, split into a training portion and a held-back validation portion, so that later we can check the model is genuinely learning the language rather than memorizing the book. We can draw `(context, next-token)` batches on demand. Next, notebook 03 uses these to train your first real language model." } ], "metadata": { "kernelspec": { "display_name": "Python 3 (.venv)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }