.backward()Stop deriving gradients by hand. Build the machine that does it.
Automatic differentiation -- the machinery behind loss.backward() -- is small enough to build in ~40 lines and run in your browser.
In notebook 03 we worked out the gradient of a one-layer model by hand. That approach does not scale, because a real network is a deep chain of thousands of operations, and differentiating all of it by hand is hopeless. The solution is automatic differentiation, usually shortened to autograd. It is the machinery behind the loss.backward() call in PyTorch.
The idea is small and rather elegant: build an expression out of tiny operations, have each operation remember how it was computed, then sweep backward through the chain applying a single rule, the chain rule, to obtain the gradient of everything at once. We will build a working engine for ordinary numbers in about 40 lines. After this, .backward() will no longer feel mysterious.
The chain rule, the only rule
Everything below rests on one fact from calculus, and it has a friendly picture.
Think of exchange rates. Suppose 1 dollar is worth 3 euros, and 1 euro is worth 2 pesos. How many pesos is a dollar worth? You multiply the rates: 3 * 2 = 6. The chain rule is that same move applied to "sensitivities." If nudging a changes b at a rate written db/da, and nudging b changes the loss L at a rate written dL/db, then nudging a changes L at the product of the two:
$$\frac{dL}{da} = \frac{dL}{db}\cdot\frac{db}{da}$$
(The notation dL/da simply means "the rate at which L changes when a changes," which is the slope, or derivative, from notebook 01.) So to find how the loss responds to some number buried deep in the network, you multiply the small rates along the path from that number up to the loss.
How autograd uses this: every operation knows its own small rate, called its local derivative, with respect to its inputs. The code comments spell each one out; for a + b the rate is 1, and for a * b it is the other input. Backprop starts at the output with dL/dL = 1 and walks backward, and at each step it multiplies the gradient coming in from above by that operation's local rate, handing the correct gradient down to its inputs. That per-operation step is stored as a small _backward function.
One subtlety: gradients add up. If a value is used in two places, nudging it affects the loss through both paths, so its total gradient is the sum of what comes back from each. This is why every grad starts at 0 and we always add to it, and why backward() first sorts the operations into order (a "topological order"), so that each one is finalized only after everything that depends on it has already contributed its gradient.
import math
class Value:
"""A scalar that records how it was computed, so we can backprop through it."""
def __init__(self, data, _children=(), _op=''):
self.data = data
self.grad = 0.0 # dL/d(self), filled in during backward()
self._backward = lambda: None # how to send gradient to our inputs
self._prev = set(_children)
self._op = _op
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), '+')
def _backward(): # d(a+b)/da = 1, d(a+b)/db = 1
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), '*')
def _backward(): # d(a*b)/da = b, d(a*b)/db = a
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def __pow__(self, k): # only constant powers
out = Value(self.data ** k, (self,), f'**{k}')
def _backward(): # d(a**k)/da = k * a**(k-1)
self.grad += k * self.data ** (k - 1) * out.grad
out._backward = _backward
return out
def tanh(self):
t = math.tanh(self.data)
out = Value(t, (self,), 'tanh')
def _backward(): # d(tanh)/dx = 1 - tanh^2
self.grad += (1 - t ** 2) * out.grad
out._backward = _backward
return out
def backward(self):
# topological order so every node is processed after the things that use it
topo, visited = [], set()
def build(v):
if v not in visited:
visited.add(v)
for child in v._prev: build(child)
topo.append(v)
build(self)
self.grad = 1.0 # dL/dL = 1
for v in reversed(topo):
v._backward()
# niceties so we can write normal math
def __neg__(self): return self * -1
def __radd__(self, o): return self + o
def __sub__(self, o): return self + (-o)
def __rsub__(self, o): return o + (-self)
def __rmul__(self, o): return self * o
def __truediv__(self, o): return self * o ** -1
def __repr__(self): return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"Line by line: what each line does
class Value:: our first class: a bundle of data plus the functions that work on it. EachValuewraps one number and remembers how it was made, so we can later trace gradients back through it.def __init__(self, data, ...):: the constructor, run when you writeValue(2.0).selfmeans "this particular object";self.data = datastores the number on it.self.grad = 0.0: a slot for "how much does the final loss care about me?", starting at 0 and filled in later bybackward().self._backward = lambda: None: a placeholder function that does nothing; each operation will swap in its own gradient-passing rule. (The leading underscore is a convention for "internal -- not the public face.")self._prev = set(_children): remember which Values fed into this one, so we can walk the graph backwards.def __add__(self, other):: the double-underscore names hook into Python's operators: defining__add__is what makesa + bwork on our objects (likewise__mul__for*,__pow__for**).- Inside each operation: build the result
out, then define a tiny_backward()that knows this operation's local slope. For+the slope is 1, so gradient passes straight through to both inputs. For*the slope toward one input is the other input's value. Fortanhit's1 - tanh². - Every rule multiplies the incoming
out.grad(the chain rule) and adds into the input's.gradwith+=(gradients accumulate when a value is used in several places). - These inner functions are closures: they capture
self,other,outat the moment they're created, so calling them later still knows exactly which numbers were involved. def backward(self):: the orchestrator. Thebuildhelper lists every node so each comes after everything it feeds into (a topological order). Then setself.grad = 1.0(the loss's sensitivity to itself) and call each node's_backward()in reverse: gradient flows from the loss back to every input.__neg__, __sub__, __radd__, ...: small conveniences soa - bor2 * aalso work, each written in terms of the+and*we already defined.__repr__: what a Value looks like when you print it.
A Value is a number that remembers where it came from
The class above is short but easy to read past. The one idea that makes autograd possible is that a Value stores more than a number: it keeps the operation and the input Values that produced it. Let's build a tiny expression and look.
# A Value is not just a number -- it remembers the op and inputs that made it.
n_a = Value(2.0)
n_b = Value(3.0)
expr = n_a * n_b + 1 # a small expression
print("expr.data =", expr.data, " (the forward value: 2*3 + 1)")
print("expr._op =", repr(expr._op), " (the operation that produced expr)")
print("expr._prev -- the inputs it was built from:")
for child in expr._prev:
print(f" Value(data={child.data}) from op {child._op!r}")
print("\nthat stored link from each node back to its inputs IS the graph")
print("backward() will later walk in reverse.")Line by line: what each line does
n_a = Value(2.0); n_b = Value(3.0): two leaf values, the inputs.expr = n_a * n_b + 1: builds two nodes -- a*node (data 6) and then a+node (data 7) sitting on top of it.expr._op: the label of the last operation, here'+'.expr._prev: the set of inputs that+combined -- the*node and the constant 1. Following these links from the top reaches every value in the expression.- This is the whole trick: each node knows its inputs, so a single reverse walk can apply the chain rule everywhere.
Does it work? Check against numerical gradients
The honest test of an autograd engine is whether it produces the same gradients as brute force. We build a small expression, L = (a*b + b**2) * tanh(c), and call .backward() once, which fills in a.grad, b.grad, and c.grad by the chain rule. We then compare these against the numerical slope from notebook 01: nudge each input by a tiny amount, see how much L moves, and divide the change by the distance covered. The two agree to five decimal places, so the engine is correct. (These are the same three numbers PyTorch will reproduce in notebook 07.)
The whole graph, alive
The exact expression checked above — L = (a*b + b**2) · tanh(c) — really is this little graph our engine built. Drag a, b, c and every .data (top number) updates: that is the forward pass. Press Run backward and watch each .grad (bottom number) fill in, right to left — the chain rule, one node at a time.
# L = (a*b + b**2) * tanh(c)
a, b, c = Value(2.0), Value(-3.0), Value(0.5)
L = (a*b + b**2) * c.tanh()
L.backward()
print("autograd grads: a=%.5f b=%.5f c=%.5f" % (a.grad, b.grad, c.grad))
def numeric(fn, x, h=1e-6):
return (fn(x + h) - fn(x - h)) / (2*h)
da = numeric(lambda x: (x*(-3.0) + (-3.0)**2) * math.tanh(0.5), 2.0)
db = numeric(lambda x: (2.0*x + x**2) * math.tanh(0.5), -3.0)
dc = numeric(lambda x: (2.0*(-3.0) + (-3.0)**2) * math.tanh(x), 0.5)
print("numerical grads: a=%.5f b=%.5f c=%.5f" % (da, db, dc))
print("match ->", all(abs(x-y) < 1e-4 for x, y in [(a.grad,da),(b.grad,db),(c.grad,dc)]))Line by line: what each line does
a, b, c = Value(2.0), Value(-3.0), Value(0.5): three tracked numbers (three assignments on one line).L = (a*b + b**2) * c.tanh(): ordinary-looking math, but every+,*,**secretly builds a node that remembers its inputs.L.backward(): one call; afterwardsa.grad,b.grad,c.gradhold how much L moves per tiny nudge of each input.def numeric(fn, x, h=1e-6):: the brute-force slope check from notebook 01.da = numeric(lambda x: ..., 2.0): estimate each gradient by wiggling that one input while the other two stay fixed as constants.all(abs(x-y) < 1e-4 for ...): True only if every pair agrees to four decimals: the engine matches reality.
The whole graph, as a table
.backward() filled in a .grad on every node in the expression, not just the three inputs. Here is the entire graph of L = (a*b + b**2) * tanh(c) laid out -- each node's operation, its forward data, and its grad, which is dL/dnode. Read it bottom-up: L.grad starts at 1, and the chain rule carries a gradient into every node below it.
# Walk the graph L was built from and print every node's data and grad.
names = {id(a): "a", id(b): "b", id(c): "c"}
topo, seen = [], set()
def walk(v):
if id(v) not in seen:
seen.add(id(v))
for ch in v._prev:
walk(ch)
topo.append(v)
walk(L)
print(f"{'node':>6} {'op':>6} {'data':>10} {'grad = dL/dnode':>18}")
for v in topo:
tag = names.get(id(v), "L" if v is L else "")
print(f"{tag:>6} {(v._op or 'input'):>6} {v.data:>+10.4f} {v.grad:>+18.4f}")Line by line: what each line does
names = {id(a): "a", ...}: a lookup so the three input nodes print with friendly names; the rest are labelled by their operation.walk(v): the same topological walkbackward()uses -- it lists every node with children before parents.- The loop prints each node's
data(its forward value) andgrad(how muchLmoves if you nudge that node). Lsits at the bottom withgrad = 1.0000(nudging L changes L one-for-one), and every node above gets a nonzero grad -- that is the chain rule reaching the whole graph, not just the inputs.
A tiny neural net, trained with our engine
Now we put the engine to real use. We will stack Value objects into a small neural network and train it on a toy problem, with no PyTorch, using only the autograd we just wrote.
The building blocks:
- a neuron takes its inputs, multiplies each by a learned weight, adds the results together along with one extra learned number called a bias (so the whole step is a weighted vote), and then passes the result through
tanh, a standard "squashing" function that gently forces any number into the range -1 to 1. The weights and the bias are the knobs the neuron learns. - a layer is simply several neurons side by side, each looking at the same inputs.
- an MLP, short for multi-layer perceptron, stacks layers, feeding the outputs of one layer in as the inputs of the next.
MLP(2, [8, 8, 1]) means 2 inputs, then a layer of 8 neurons, then another 8, then a single output. That comes to 105 individual knobs (the parameters) for the engine to tune. Training is the same loop as always: a forward pass to predict, a measure of how far off we are (the squared error this time, which is the gap between prediction and target, squared so that every miss counts as a positive amount, since these outputs are plain numbers rather than probabilities), then loss.backward() to fill in every knob's .grad, then the update -- the line that actually changes the knobs. Repeat, and the loss falls until the predictions settle onto the +1 and -1 targets.
The update step, and what "SGD" means. All the learning happens in one line: p.data -= 0.05 * p.grad. Each knob's .grad says which way to move it to raise the loss, so we step the opposite way -- subtract a small slice of the gradient -- to lower it. The size of that slice is the learning rate (here 0.05): too small and training crawls, too large and it overshoots the bottom and bounces. Doing this over and over is gradient descent -- the same rolling-downhill picture from notebook 01, now running on all 105 knobs at once.
The code comment calls it an SGD step, short for stochastic gradient descent. The only extra idea in the word "stochastic" is randomness: real training does not measure the gradient on the whole dataset each step, it uses a fresh random handful of examples (a mini-batch) -- far cheaper, and the little bit of noise even helps it generalize. Our toy here has just 8 points, so we use all of them every step (technically full-batch gradient descent), but the update line is identical. The bigram in notebook 03 and the GPT in notebook 09 both take the stochastic route, drawing a new random batch on every step.
import random
random.seed(42)
class Neuron:
def __init__(self, nin):
self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
self.b = Value(0.0)
def __call__(self, x):
act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
return act.tanh()
def parameters(self): return self.w + [self.b]
class Layer:
def __init__(self, nin, nout): self.neurons = [Neuron(nin) for _ in range(nout)]
def __call__(self, x):
outs = [n(x) for n in self.neurons]
return outs[0] if len(outs) == 1 else outs
def parameters(self): return [p for n in self.neurons for p in n.parameters()]
class MLP:
def __init__(self, nin, nouts):
sizes = [nin] + nouts
self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(nouts))]
def __call__(self, x):
for layer in self.layers: x = layer(x)
return x
def parameters(self): return [p for layer in self.layers for p in layer.parameters()]
model = MLP(2, [8, 8, 1]) # 2 inputs -> 8 -> 8 -> 1 output
print("parameters:", len(model.parameters()))Line by line: what each line does
import random; random.seed(42): plain-Python randomness for the starting weights, seeded for repeatability.class Neuron:: one neuron: a list of weightsself.w(one Value per input, started random) plus a biasself.b.def __call__(self, x):: defining__call__lets you use the object like a function, writingn(x).act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b):zippairs each weight with its input; multiply and add them all up, starting the sum from the bias. A weighted vote.return act.tanh(): squash that vote into the range -1..1. Every step here is a tracked Value op, so gradients will flow through.class Layer:: a row of neurons that all read the same inputs; returns their outputs as a list.class MLP:: chains layers:for layer in self.layers: x = layer(x)feeds each layer's output into the next.def parameters(self):(at each level) -- gather every weight and bias into one flat list, so the trainer can reach all 105 knobs.model = MLP(2, [8, 8, 1]): build the network: 2 inputs → 8 neurons → 8 → 1 output.
Where the 105 knobs live, and one neuron by hand
MLP(2, [8, 8, 1]) reports 105 parameters. That number is not magic -- it is just every weight and bias added up. Let's break it down layer by layer, then compute a single neuron the long way so "a weighted vote, then tanh" stops being words.
One neuron: multiply each input by a weight, add them with a bias, then squash with tanh — out = tanh(w₁x₁ + w₂x₂ + b). That is 3 numbers for this 2-input neuron (2 weights + 1 bias). Stack 8 of them for layer 1, feed those into 8 more, then into 1 — and you get the 105 weights the engine tunes.
# How the 105 parameters are distributed, and one neuron computed by hand.
total = 0
for li, layer in enumerate(model.layers):
nout = len(layer.neurons); nin = len(layer.neurons[0].w)
pc = nout * (nin + 1); total += pc
print(f"layer {li}: {nout} neurons x ({nin} weights + 1 bias) = {pc} params")
print(f"total = {total} (equals len(model.parameters()) = {len(model.parameters())})")
nrn = model.layers[0].neurons[0]; xin = [1.0, 1.0] # one neuron, first layer
s = sum(w.data * xi for w, xi in zip(nrn.w, xin)) + nrn.b.data
print(f"\nneuron[0][0] on input {xin}:")
print(f" weights = {[round(w.data, 3) for w in nrn.w]}, bias = {nrn.b.data}")
print(f" weighted vote = {s:+.3f} -> tanh = {math.tanh(s):+.3f} (the neuron's output)")Line by line: what each line does
- The loop reads each layer straight off the model: number of neurons times (one weight per input + one bias).
24 + 72 + 9 = 105: exactly whatlen(model.parameters())reported -- no hidden knobs.nrn = model.layers[0].neurons[0]: pull out a single neuron from the first layer.s = sum(w.data * xi ...) + nrn.b.data: its weighted vote -- each input times its weight, plus the bias.math.tanh(s): squash that vote into (-1, 1). That one number is the neuron's output, and 105 of these knobs are what training tunes.
# toy dataset: two interleaved blobs (label +1 / -1)
xs = [[ 1.0, 1.0], [ 1.5, 0.5], [ 0.5, 1.5], [ 2.0, 1.0],
[-1.0, -1.0], [-1.5, -0.5], [-0.5, -1.5], [-2.0, -1.0]]
ys = [1.0]*4 + [-1.0]*4
losses = []
for step in range(120):
preds = [model(x) for x in xs] # forward
loss = sum((p - y)**2 for p, y in zip(preds, ys)) # sum of squared errors
for p in model.parameters(): p.grad = 0.0 # reset grads
loss.backward() # backprop (our engine!)
for p in model.parameters(): p.data -= 0.05 * p.grad # SGD step
losses.append(loss.data)
print("loss: %.4f -> %.4f" % (losses[0], losses[-1]))
print("predictions:", [round(model(x).data, 2) for x in xs])
print("targets :", ys)Line by line: what each line does
xs = [[1.0, 1.0], ...]; ys = [1.0]*4 + [-1.0]*4: eight 2-number points and their labels: the first four should output +1, the last four -1.for step in range(120):: train for 120 rounds.preds = [model(x) for x in xs]: run the network on all eight points.loss = sum((p - y)**2 for p, y in zip(preds, ys)): squared error: (prediction minus target)², summed. Squaring makes every miss positive and punishes big misses extra.for p in model.parameters(): p.grad = 0.0: reset every knob's gradient (they accumulate, so stale ones must be cleared each round).loss.backward(): our engine fills in all 105.gradslots in one sweep.for p in model.parameters(): p.data -= 0.05 * p.grad: the update -- the moment of learning. Each.gradpoints uphill (toward more loss), so subtract a small fraction of it to step every knob downhill; that fraction0.05is the learning rate. This is gradient descent. The comment's SGD (stochastic gradient descent) is the same rule, just with the gradient measured on a fresh random mini-batch each step -- as the bigram does in notebook 03 -- instead of on all the data at once.round(model(x).data, 2): after training, the predictions hug the +1 / -1 targets.
import matplotlib.pyplot as plt
plt.figure(figsize=(6,3)); plt.plot(losses)
plt.title("training a neural net on a hand-written autograd engine"); plt.xlabel("step"); plt.ylabel("loss"); plt.show()Line by line: what each line does
import matplotlib.pyplot as plt: the plotting toolbox.plt.plot(losses): with a single list, matplotlib uses 0,1,2,... for the x-axis: loss per step, falling.- Title, axis labels, and
plt.show(): the standard plotting finish.
See what it learned: the decision boundary
The loss curve tells you training worked, but not what the net now believes. So feed a whole grid of points through the trained model and colour each region by the sign of the output. The eight training points fall on the right sides of a boundary the engine bent to fit them -- learned with nothing but the autograd we wrote at the top of this notebook.
The neural network you just built
This is MLP(2, [8, 8, 1]) after the 120 training steps above — 2 inputs → 8 → 8 → 1 output. Its 105 real learned weights are the edges (blue = positive, red = negative, thickness = strength). Drag inside the square to move the test point: the net runs a forward pass, every neuron lights up with its tanh activation, and the readout says which class. The shaded regions are the actual boundary it learned.
Circles are the 4 points labelled +1, squares the 4 labelled −1. The net bends a smooth boundary between them — trained entirely by the ~40-line engine you wrote, no PyTorch.
# Feed a grid of points through the trained net; colour each by the sign of its output.
xr = [-3 + 6 * i / 43 for i in range(44)]
yr = [-2.5 + 5 * j / 35 for j in range(36)]
Z = [[model([X, Y]).data for X in xr] for Y in yr] # 1584 forward passes
plt.figure(figsize=(5.2, 4.2))
plt.contourf(xr, yr, Z, levels=[-100, 0, 100], colors=["#f4b7b7", "#b7c9f4"])
plt.contour(xr, yr, Z, levels=[0], colors=["#333"], linewidths=1)
plt.scatter([p[0] for p in xs[:4]], [p[1] for p in xs[:4]], c="#1c4fd6", edgecolors="k", s=90, label="+1")
plt.scatter([p[0] for p in xs[4:]], [p[1] for p in xs[4:]], c="#d61c1c", edgecolors="k", s=90, marker="s", label="-1")
plt.legend(); plt.title("the decision boundary our engine learned")
plt.xlabel("x1"); plt.ylabel("x2"); plt.show()Line by line: what each line does
xr, yr: a grid of x and y coordinates covering the plot area.Z = [[model([X, Y]).data ...]]: run the trained net on every grid point -- 1584 forward passes -- and keep each output number.plt.contourf(..., levels=[-100, 0, 100]): paint the plane in two colours, split at output 0 -- the model's decision.plt.contour(..., levels=[0]): draw the boundary line itself, where the net switches from -1 to +1.plt.scatter(...): the eight training points on top; each sits in the correctly coloured region, so the net separated the two classes.
Recap
You have built autograd: a graph of operations that differentiates itself by multiplying local rates along the chain, and you trained a real, if tiny, neural network on top of it. PyTorch's .backward() is exactly this idea, generalized from single numbers to whole tensors (a tensor is just a grid of numbers, as in notebook 01) and written in fast, low-level code for speed. From here on we let the framework handle the bookkeeping, but you now know precisely what it is doing underneath.
Next, notebook 05 covers self-attention, the mechanism that lets a token look back over the whole context rather than just one character.
04_micrograd.ipynb