Welcome. By the end of this course you will have built a working GPT, the kind of model behind tools like ChatGPT, and trained it until it produces Shakespeare-flavored text. Nothing is left as a black box; you build every piece yourself.

A reassurance before we start: you do not need to be a mathematician. Every idea is introduced first with a plain-English picture and a small worked example, and the heaviest arithmetic you will meet is "multiply some numbers together and add them up." Some ability to read Python, a popular and fairly readable programming language, is the only real prerequisite.

Two words you will see often are worth defining now. A neural network is a computer model that learns patterns from examples instead of being given fixed rules. A GPT is one particular design of neural network (the letters stand for "generative pre-trained transformer"), and it is the design that today's language models are built on.

The course covers everything twice.

1. First by hand, using NumPy (Part 1). NumPy is a free, widely used add-on for Python that does fast arithmetic on long lists of numbers. Writing attention, the gradients, and the training loop with nothing but NumPy lets you see exactly how each one works, rather than trusting a ready-made library (a library is simply reusable code that other people have written for you).
2. Then in PyTorch (Part 2). PyTorch is the standard free toolkit for building and training neural networks. You rebuild the same ideas with this professional tool, then make the model larger and train it on a GPU, the fast graphics chip found in many computers. (The terms Apple MPS and CUDA that appear later simply name the two common kinds of graphics chip, made by Apple and NVIDIA.)

The one idea behind every LLM

A language model, or LLM (short for "large language model"), does exactly one thing: given a sequence of tokens, it predicts the next one. A token is a small piece of text, roughly a word or part of a word; the next notebook explains tokens properly.

"To be or not to be, that is the ___"

If you train a large enough model on enough text to fill in that blank well, then, in order to keep reducing its mistakes, it is gradually forced to pick up grammar, facts, and a degree of reasoning. Everything else these models appear to do, whether holding a conversation, writing code, or translating between languages, is this same next-token prediction repeated. This single idea runs through the entire course.

How to read the Python in this course

You do not need to write Python to follow this course; you only need to read it. The twelve patterns below cover about 95% of every code cell, so skim them now and come back whenever a line looks unfamiliar. Every code cell also has a "New to Python? Read this cell line by line" panel directly beneath it.

• Variables: x = 5 stores a value in a labelled box. x = x - 1 updates it (the right-hand side is worked out first, then stored back).
• Lists: xs = [10, 20, 30] is an ordered collection. xs[0] is the first item (counting starts at 0), xs[-1] is the last, and xs[1:3] takes items 1 and 2 (the end position is excluded).
• Dictionaries: d = {"a": 1, "b": 2} is a lookup table; d["a"] fetches the value stored under "a".
• Loops: for c in chars: repeats the indented lines once for each item. range(5) counts 0, 1, 2, 3, 4. enumerate(chars) counts as it goes, handing you both the position and the item.
• Functions: def f(x): return x*2 defines a reusable recipe, so calling f(3) gives 6. A lambda x: x*2 is the same idea written as a short, nameless function.
• Building a list in one line: [f(x) for x in xs] means "apply f to every item of xs and collect the results." The dictionary version, {c: i for i, c in enumerate(chars)}, builds a lookup table the same way.
• f-strings: f"loss {x:.3f}" inserts the value of x into the text, and :.3f means "show three decimal places."
• Imports: import numpy as np loads a toolkit and gives it the short name np, after which np.zeros(...) calls one of its tools.
• NumPy arrays: grids of numbers. a.shape reports the grid's size, so (2, 3) means 2 rows by 3 columns. Arithmetic on an array applies to every number at once, with no loop needed, and @ means matrix multiply.
• Indexing grids: W[3] takes row 3; W[:, 1] takes column 1 (the : means "all of this direction"); data[i:i+8] takes 8 consecutive items starting at position i.
• Classes: class Neuron: bundles data and functions together into one object, and inside it self means "this particular object." Classes first appear in notebook 04, where they are explained in full.
• Comments: anything after a # is a note for humans, which Python ignores.

The map

Part 0 - Foundations

• 00 this roadmap
• 01 the small amount of maths you actually need: vectors, matrix multiply, softmax, cross-entropy, and gradients

Part 1 - Build the pieces by hand (NumPy)

• 02 tokenization: turning text into numbers
• 03 your first language model (a "bigram") and the full training loop
• 04 autograd: how the gradients are worked out automatically
• 05 self-attention, built up from the intuition
• 06 the Transformer block

Part 2 - Rebuild and scale up in PyTorch

• 07 PyTorch and autograd (the same ideas, far less code)
• 08 assembling the full GPT
• 09 training it and generating text
• 10 controlling how it writes, and a tour of what separates this from a frontier model

Work through the notebooks in order, because each one builds directly on the one before. And try to do more than read: change a number, see what breaks, and run the cell again. That habit is what makes the ideas stick.