{ "cells": [ { "cell_type": "markdown", "id": "04bfe466", "metadata": {}, "source": "# 00 - Build an LLM from scratch: the roadmap\n\nWelcome. By the end of this course you will have built a working **GPT**, the kind of model behind tools like ChatGPT, and trained it until it produces Shakespeare-flavored text. Nothing is left as a black box; you build every piece yourself.\n\nA reassurance before we start: you do not need to be a mathematician. Every idea is introduced first with a plain-English picture and a small worked example, and the heaviest arithmetic you will meet is \"multiply some numbers together and add them up.\" Some ability to read Python, a popular and fairly readable programming language, is the only real prerequisite.\n\nTwo words you will see often are worth defining now. A **neural network** is a computer model that learns patterns from examples instead of being given fixed rules. A **GPT** is one particular design of neural network (the letters stand for \"generative pre-trained transformer\"), and it is the design that today's language models are built on.\n\nThe course covers everything twice.\n\n1. **First by hand, using NumPy** (Part 1). NumPy is a free, widely used add-on for Python that does fast arithmetic on long lists of numbers. Writing attention, the gradients, and the training loop with nothing but NumPy lets you see exactly how each one works, rather than trusting a ready-made library (a library is simply reusable code that other people have written for you).\n2. **Then in PyTorch** (Part 2). PyTorch is the standard free toolkit for building and training neural networks. You rebuild the same ideas with this professional tool, then make the model larger and train it on a GPU, the fast graphics chip found in many computers. (The terms *Apple MPS* and *CUDA* that appear later simply name the two common kinds of graphics chip, made by Apple and NVIDIA.)" }, { "cell_type": "markdown", "id": "003ba07c", "metadata": {}, "source": "## The one idea behind every LLM\n\nA language model, or **LLM** (short for \"large language model\"), does exactly one thing: given a sequence of **tokens**, it predicts the next one. A token is a small piece of text, roughly a word or part of a word; the next notebook explains tokens properly.\n\n> \"To be or not to be, that is the ___\"\n\nIf you train a large enough model on enough text to fill in that blank well, then, in order to keep reducing its mistakes, it is gradually forced to pick up grammar, facts, and a degree of reasoning. Everything else these models appear to do, whether holding a conversation, writing code, or translating between languages, is this same next-token prediction repeated. This single idea runs through the entire course." }, { "cell_type": "code", "execution_count": 1, "id": "07b2961a", "metadata": { "execution": { "iopub.execute_input": "2026-06-11T07:11:59.262691Z", "iopub.status.busy": "2026-06-11T07:11:59.262471Z", "iopub.status.idle": "2026-06-11T07:12:00.327108Z", "shell.execute_reply": "2026-06-11T07:12:00.326567Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "python : 3.12.12\n", "numpy : 2.4.6\n", "torch : 2.12.0\n", "device : mps (this is what we'll train on in Part 2)\n" ] } ], "source": [ "# Environment check - make sure the tools are here before we start.\n", "import sys, numpy as np, torch\n", "print(\"python :\", sys.version.split()[0])\n", "print(\"numpy :\", np.__version__)\n", "print(\"torch :\", torch.__version__)\n", "dev = \"mps\" if torch.backends.mps.is_available() else (\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "print(\"device :\", dev, \"(this is what we'll train on in Part 2)\")" ] }, { "id": "f3458c5e", "cell_type": "markdown", "metadata": {}, "source": "
Line by line: what each line does\n\n
" }, { "id": "8270cd45", "cell_type": "markdown", "metadata": {}, "source": "## How to read the Python in this course\n\nYou do not need to *write* Python to follow this course; you only need to **read** it. The twelve patterns below cover about 95% of every code cell, so skim them now and come back whenever a line looks unfamiliar. Every code cell also has a **\"New to Python? Read this cell line by line\"** panel directly beneath it.\n\n- **Variables**: `x = 5` stores a value in a labelled box. `x = x - 1` updates it (the right-hand side is worked out first, then stored back).\n- **Lists**: `xs = [10, 20, 30]` is an ordered collection. `xs[0]` is the first item (counting starts at 0), `xs[-1]` is the last, and `xs[1:3]` takes items 1 and 2 (the end position is excluded).\n- **Dictionaries**: `d = {\"a\": 1, \"b\": 2}` is a lookup table; `d[\"a\"]` fetches the value stored under `\"a\"`.\n- **Loops**: `for c in chars:` repeats the indented lines once for each item. `range(5)` counts 0, 1, 2, 3, 4. `enumerate(chars)` counts as it goes, handing you both the position and the item.\n- **Functions**: `def f(x): return x*2` defines a reusable recipe, so calling `f(3)` gives 6. A `lambda x: x*2` is the same idea written as a short, nameless function.\n- **Building a list in one line**: `[f(x) for x in xs]` means \"apply `f` to every item of `xs` and collect the results.\" The dictionary version, `{c: i for i, c in enumerate(chars)}`, builds a lookup table the same way.\n- **f-strings**: `f\"loss {x:.3f}\"` inserts the value of `x` into the text, and `:.3f` means \"show three decimal places.\"\n- **Imports**: `import numpy as np` loads a toolkit and gives it the short name `np`, after which `np.zeros(...)` calls one of its tools.\n- **NumPy arrays**: grids of numbers. `a.shape` reports the grid's size, so `(2, 3)` means 2 rows by 3 columns. Arithmetic on an array applies to every number at once, with no loop needed, and `@` means matrix multiply.\n- **Indexing grids**: `W[3]` takes row 3; `W[:, 1]` takes column 1 (the `:` means \"all of this direction\"); `data[i:i+8]` takes 8 consecutive items starting at position `i`.\n- **Classes**: `class Neuron:` bundles data and functions together into one object, and inside it `self` means \"this particular object.\" Classes first appear in notebook 04, where they are explained in full.\n- **Comments**: anything after a `#` is a note for humans, which Python ignores." }, { "cell_type": "markdown", "id": "aedd48b7", "metadata": {}, "source": "## The map\n\n**Part 0 - Foundations**\n- `00` this roadmap\n- `01` the small amount of maths you actually need: vectors, matrix multiply, softmax, cross-entropy, and gradients\n\n**Part 1 - Build the pieces by hand (NumPy)**\n- `02` tokenization: turning text into numbers\n- `03` your first language model (a \"bigram\") and the full training loop\n- `04` autograd: how the gradients are worked out automatically\n- `05` self-attention, built up from the intuition\n- `06` the Transformer block\n\n**Part 2 - Rebuild and scale up in PyTorch**\n- `07` PyTorch and autograd (the same ideas, far less code)\n- `08` assembling the full GPT\n- `09` training it and generating text\n- `10` controlling how it writes, and a tour of what separates this from a frontier model\n\nWork through the notebooks **in order**, because each one builds directly on the one before. And try to do more than read: change a number, see what breaks, and run the cell again. That habit is what makes the ideas stick." } ], "metadata": { "kernelspec": { "display_name": "Python 3 (.venv)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }