Every term, in one place.

A quick cheat-sheet for the whole course. Each entry is a plain-English definition of a term you built or used.

Token

A chunk of text the model works with — one character (here) or one subword (real models). The model reads and predicts tokens, not words.

Vocabulary

The fixed set of distinct tokens the model knows. Each gets an integer id.

Embedding

The vector of numbers that represents a token's meaning. Nothing assigns these by hand: training nudges them until tokens used in similar ways get similar vectors.

Logits

The raw, unnormalized scores the model outputs — one per vocabulary token — before softmax turns them into probabilities.

Softmax

Turns a vector of logits into a probability distribution: all positive, summing to 1, bigger logits getting exponentially more weight.

Cross-entropy

The loss we minimize: the negative log of the probability the model gave the correct token. Confident-and-right → near 0; confident-and-wrong → large. Think of it as the model's surprise at the truth.

Perplexity

e raised to the loss — roughly "how many options is the model still torn between?" Random guessing on 65 characters: perplexity 65. Our trained GPT: about 5.

Dot product

Multiply two vectors slot-by-slot and sum. Large when they point the same way — a similarity score, and the core of attention.

Gradient

For each parameter, the direction (and steepness) to nudge it to lower the loss.

Gradient descent

The learning rule: take a small step downhill along the gradient, repeat. The learning rate sets the step size.

Chain rule

How sensitivities multiply along a chain: if nudging a moves b, and b moves the loss, the rates multiply — like currency exchange rates. The one rule all of backprop uses.

Backpropagation

Computing the gradient of the loss for every parameter in one backward pass, by applying the chain rule through the computation graph.

Autograd

Automatic differentiation: the machinery (PyTorch's .backward()) that does backprop for you. You build the forward pass; it derives the backward.

Parameter (weight)

One of the model's learnable numbers. Training is the search for parameter values that make the loss small.

Bigram model

The simplest language model: predict the next token from only the current one, via a lookup table of logits. Its loss floor (~2.45 here) is set by its one-token memory — no training can beat it.

Self-attention

The mechanism that lets each token gather information from earlier tokens, weighted by relevance.

Query / Key / Value

Three vectors each token produces — like a search engine: the query is what I'm looking for, the key is what I contain, the value is what I hand over. The attention weight from token i to j is query_i · key_j; the output is a weighted sum of values.

Causal mask

Blocks attention to future tokens, so position t can only use tokens up to t. What makes the model able to generate left-to-right.

Multi-head attention

Several attention heads run in parallel, so the model can attend to different kinds of relationships at once.

Scaling (1/√d)

Dividing attention scores by the square root of the head size, so softmax stays soft and gradients flow.

Positional encoding

Information added to each token so the model knows word order — attention alone is order-blind.

LayerNorm

Normalizes each token vector to zero mean and unit variance, keeping activations stable in a deep stack.

Residual connection

Adding a sub-layer's input back to its output (x = x + sublayer(x)) — keep the notes, add an edit — giving gradients a clean path through depth.

Feed-forward network

The per-token MLP (expand, nonlinearity, project back) after attention. Where most parameters live.

Transformer block

Attention + feed-forward, each wrapped in pre-norm and a residual. The unit you stack to make a GPT.

GPT (decoder-only Transformer)

Embeddings → a stack of Transformer blocks → final norm → a head producing next-token logits.

Context window (block size)

The maximum number of tokens the model can look back over at once.

Tensor

A multi-dimensional array (like NumPy's) that can live on a GPU and track gradients — PyTorch's basic object.

Optimizer (AdamW)

The rule that applies gradients to update parameters. AdamW is a smarter, adaptive gradient descent.

Batch / step / epoch

A batch is a handful of examples processed together; a step is one update; an epoch is one pass over the data.

Validation set

A held-out slice of data the model never trains on — its practice exam. If val loss tracks train loss, the model is learning the language, not memorizing the text.

Overfitting

When train loss keeps falling but val loss stalls or rises: the model is memorizing the training data instead of learning patterns that carry over.

Dropout

Randomly switching off a fraction of signals during training so the model can't lean on any single path — a standard guard against overfitting.

Temperature

Divides logits before softmax. Low = sharp and safe; high = flat and creative.

Greedy / top-k / top-p

Ways to pick the next token: always the most likely (greedy), sample from the k most likely (top-k), or from the smallest set summing to probability p (top-p).

BPE (Byte-Pair Encoding)

The subword tokenizer real models use: merge frequent character pairs into tokens, giving shorter, more meaningful sequences.

KV-cache

Caching past tokens' keys and values during generation so each new token is cheap — the main inference speed-up.

Scaling laws

The finding that loss falls predictably as parameters, data, and compute grow together.

Pretraining / SFT / RLHF

The three stages from raw model to assistant: predict next token (pretraining), fine-tune on instruction pairs (SFT), tune toward human preferences (RLHF/DPO).

MPS / CUDA

The GPU backends PyTorch trains on — Metal Performance Shaders on Apple silicon, CUDA on NVIDIA.