A hands-on course
Build a GPT from scratch —
and understand every line.
Start from "what is a vector" and finish with a working Transformer that writes Shakespeare. Each lesson below is a notebook shown in full: the explanations, the code, and the real output it produced. The early lessons (01 to 06) use NumPy, a Python toolkit for number-crunching, and run right here in your browser. The later ones use PyTorch, the standard software for building neural networks, and download as a notebook file so you can train the model on a GPU (the graphics chip in many computers). Nothing is hidden, and all the maths is shown.
Start the course →The hands-on half of How LLMs Work — the friendly, no-code companion course. New to all this? Start there first.
Part 0 · Foundations
00The roadmap
Next-token prediction — the single move behind every LLM — and the plan.
01The math you actually need runs in-pageVectors, matmul, the dot product, softmax, cross-entropy, gradient descent.
Part 1 · Build it by hand (NumPy)
02Tokenization runs in-page
Turning text into the integers a model consumes.
03Your first language model runs in-pageA bigram trained on the full dataset, with the gradient by hand — and its ceiling, computed.
04Autograd from scratch runs in-pageBuild the engine behind
05Self-attention runs in-page.backward() in ~40 lines.The mechanism that makes LLMs work, built from the intuition up.
06The Transformer block runs in-pageMulti-head attention, positional encoding, LayerNorm, residuals.
Part 2 · Rebuild & scale (PyTorch)
07PyTorch & autograd
The same ideas, a tenth of the code, ready for a GPU.
08Build the GPTAssemble the full model — a guided tour of every class in
09Train the GPTgpt.py.Train on your GPU; watch the loss fall and Shakespeare appear.
10Sampling & beyondTemperature, top-k/p, and the road to a frontier model.
Reference