Notebook 04 · The engine behind .backward()

Stop deriving gradients by hand. Build the machine that does it.

Automatic differentiation -- the machinery behind loss.backward() -- is small enough to build in ~40 lines and run in your browser.

In notebook 03 we worked out the gradient of a one-layer model by hand. That approach does not scale, because a real network is a deep chain of thousands of operations, and differentiating all of it by hand is hopeless. The solution is automatic differentiation, usually shortened to autograd. It is the machinery behind the loss.backward() call in PyTorch.

The idea is small and rather elegant: build an expression out of tiny operations, have each operation remember how it was computed, then sweep backward through the chain applying a single rule, the chain rule, to obtain the gradient of everything at once. We will build a working engine for ordinary numbers in about 40 lines. After this, .backward() will no longer feel mysterious.

The chain rule, the only rule

Everything below rests on one fact from calculus, and it has a friendly picture.

Think of exchange rates. Suppose 1 dollar is worth 3 euros, and 1 euro is worth 2 pesos. How many pesos is a dollar worth? You multiply the rates: 3 * 2 = 6. The chain rule is that same move applied to "sensitivities." If nudging a changes b at a rate written db/da, and nudging b changes the loss L at a rate written dL/db, then nudging a changes L at the product of the two:

$$\frac{dL}{da} = \frac{dL}{db}\cdot\frac{db}{da}$$

(The notation dL/da simply means "the rate at which L changes when a changes," which is the slope, or derivative, from notebook 01.) So to find how the loss responds to some number buried deep in the network, you multiply the small rates along the path from that number up to the loss.

How autograd uses this: every operation knows its own small rate, called its local derivative, with respect to its inputs. The code comments spell each one out; for a + b the rate is 1, and for a * b it is the other input. Backprop starts at the output with dL/dL = 1 and walks backward, and at each step it multiplies the gradient coming in from above by that operation's local rate, handing the correct gradient down to its inputs. That per-operation step is stored as a small _backward function.

One subtlety: gradients add up. If a value is used in two places, nudging it affects the loss through both paths, so its total gradient is the sum of what comes back from each. This is why every grad starts at 0 and we always add to it, and why backward() first sorts the operations into order (a "topological order"), so that each one is finalized only after everything that depends on it has already contributed its gradient.

a = 1 dollarthe input we wiggle
▼
3 euros per dollarlocal rate db/da = 3
▼
2 pesos per eurolocal rate dL/db = 2
▼
a is worth 6 pesosdL/da = 3 × 2 = 6 — the chain rule just multiplies the rates along the path

Step 1: A number that remembers where it came from

We build the engine in five small steps and run each one before taking the next. One example runs through all of them:

python
a = 2,  b = -3,  c = 10
e = a * b        (so e = -6)
L = e + c        (so L = 4)

L plays the loss. The question: nudge a, b or c, and how fast does L move?

A plain Python number forgets where it came from: -6 cannot say it was made by multiplying 2 and -3. The first version of Value fixes that and nothing else. It does no calculus; it only records.

python · runnable
class Value:
    """A number that remembers how it was made."""
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self._prev = _children
        self._op = _op

    def __add__(self, other):
        out = Value(self.data + other.data,
                    (self, other), '+')
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data,
                    (self, other), '*')
        return out

    def __repr__(self):
        return f"Value(data={self.data})"
Line by line: what each line does
  • class Value:: the blueprint. It holds a number plus its history. The triple-quoted line is a docstring, a note for human readers.
  • def __init__(self, data, _children=(), _op=''):: runs when you write Value(2.0). The last two parameters default to an empty tuple and an empty string, which is right for a leaf: a number you typed in, with no inputs.
  • self._prev = _children: the inputs that produced this value. They are called children because these diagrams are often drawn loss-at-the-top, like a family tree, with inputs hanging underneath.
  • self._op = _op: which operation produced it, e.g. '+' or '*'. Only for inspection.
  • def __add__(self, other):: runs when Python meets x + y, with self as x and other as y.
  • out = Value(self.data + other.data, (self, other), '+'): add the numbers, then wrap the answer in a new Value that records its inputs (self, other) and its label '+'.
  • __mul__: the same shape, with * in place of +.
  • __repr__: what a Value looks like when printed.
python · runnable
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
e = a * b
L = e + c
print(L)
print(L._op, L._prev)
print(e._op, e._prev)
Line by line: what each line does
  • a, b, c: three leaves.
  • e = a * b: looks like arithmetic, but Python runs a.__mul__(b), which builds a node holding -6.0 that remembers a and b.
  • L = e + c: the same with +: L remembers e and c.
  • print(L._op, L._prev): read the receipt back: made by '+' from -6.0 and 10.0. Together these back-links form the computation graph. No gradient has been computed yet.
saved output · press Run to reproduce liveValue(data=4.0) + (Value(data=-6.0), Value(data=10.0)) * (Value(data=2.0), Value(data=-3.0))

Step 2: Sweep the graph with a pencil

First, the backward pass on paper, using the chain rule section's two facts: + forwards the incoming gradient untouched (rate 1), and * scales it, for each input, by the partner input's value.

  1. At L: its gradient with respect to itself is 1.
  2. The + in L = e + c forwards that 1: e.grad = 1, c.grad = 1.
  3. The * in e = a * b receives 1: a.grad = -3 x 1 = -3, b.grad = 2 x 1 = 2.

Check by nudging: move a to 2.01 and L becomes -6.03 + 10 = 3.97. A drop of 0.03 per 0.01 of push: rate -3, as predicted.

Every node did one identical thing: scale its own gradient by a local rate and add the result to each input. The local rate is fixed by the operation and its inputs, which are known when the node is built, so each node can be given its rule right then. That is Step 3.

Step 3: Give every node its own backward rule

Each node's rule will be a tiny function. Three Python facts make this possible; here they are in isolation.

A function can be stored like a number.

python · runnable
def say_hi():
    print("hi")

greet = say_hi    # store the function
greet()           # run it
Line by line: what each line does
  • greet = say_hi: no brackets: this stores the function under a second name. It does not run it.
  • greet(): the brackets run it, printing hi. With or without brackets is the difference between storing a rule and running it.
saved output · press Run to reproduce livehi

A function made inside a function remembers what it saw. Such a function is called a closure.

python · runnable
def make_rule(rate):
    def rule(arriving):
        return rate * arriving
    return rule

times_three = make_rule(3)
print(times_three(10))
print(times_three(0.5))
Line by line: what each line does
  • def rule(arriving):: a function defined inside make_rule; it multiplies whatever arrives by rate.
  • return rule: hand back the inner function without running it.
  • times_three = make_rule(3): make_rule runs once with rate = 3 and finishes. We keep rule.
  • times_three(10): rule still knows rate is 3, so this prints 30, and times_three(0.5) prints 1.5. A stored rate that multiplies what arrives: one step of the chain rule.
saved output · press Run to reproduce live30 1.5

A function reads a value when it runs, not when it was written.

python · runnable
box = Value(1.0)

def peek():
    return box.data

box.data = 5.0
print(peek())
Line by line: what each line does
  • def peek():: written while box.data is 1.0.
  • box.data = 5.0: change the number before peek ever runs.
  • print(peek()): prints 5.0: peek looks the value up when it runs. A backward rule is written when its node is made and runs later, once the gradient has arrived; it reads that gradient as it is then.
saved output · press Run to reproduce live5.0

The class, second version

Two new slots, grad and _backward, and a rule inside __add__ and __mul__. The changes from Step 1 are marked # new.

python · runnable
class Value:
    """A number that remembers how it was made."""
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0.0                  # new
        self._backward = lambda: None    # new
        self._prev = _children
        self._op = _op

    def __add__(self, other):
        out = Value(self.data + other.data,
                    (self, other), '+')
        def _backward():                 # new
            self.grad  += out.grad       # new
            other.grad += out.grad       # new
        out._backward = _backward        # new
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data,
                    (self, other), '*')
        def _backward():
            self.grad  += other.data * out.grad
            other.grad += self.data  * out.grad
        out._backward = _backward
        return out

    def __repr__(self):
        return (f"Value(data={self.data:.4f}, "
                f"grad={self.grad:.4f})")
Line by line: what each line does
  • self.grad = 0.0: where the sweep will leave this node's gradient; zero until then.
  • self._backward = lambda: None: this node's backward rule. A leaf has no inputs, so its rule does nothing; every operation replaces it.
  • def _backward():: the rule for this one operation, defined inside it, so it is a closure that remembers self, other and out.
  • self.grad += out.grad: for + the local rate is 1, so both inputs get out.grad as it is. out.grad is read when the rule runs.
  • +=: add to what is already there, so a value used in two places collects a share from each.
  • out._backward = _backward: no brackets: store the rule on the result node for later. _backward() would run it now, while out.grad is still 0.
  • self.grad += other.data * out.grad: for *, an input's share is its partner's value multiplied by out.grad.
  • __repr__: now prints the gradient too, with four digits after the point.

Run the rules by hand

No sweep exists yet, so trigger the rules manually, in the same order as on paper: set L.grad to 1, then L._backward(), then e._backward().

python · runnable
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
e = a * b
L = e + c
print("before :", a.grad, b.grad, c.grad)
L.grad = 1.0
L._backward()
print("after L:", e.grad, c.grad)
e._backward()
print("after e:", a.grad, b.grad)
Line by line: what each line does
  • a = Value(2.0) ... L = e + c: a fresh graph built from the new class: all gradients zero, every node carrying a rule.
  • L.grad = 1.0: start the sweep: dL/dL is 1.
  • L._backward(): runs the rule __add__ stored on L: 1 to e and to c.
  • e._backward(): runs the rule __mul__ stored on e: -3 to a, 2 to b. The pencil's numbers exactly.
saved output · press Run to reproduce livebefore : 0.0 0.0 0.0 after L: 1.0 1.0 after e: -3.0 2.0

The order is not a detail

What if e._backward() goes first, before anything has set e.grad? Try it on a freshly built graph.

python · runnable
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
e = a * b
L = e + c
L.grad = 1.0
e._backward()      # wrong: e before L
L._backward()
print("a.grad =", a.grad, " b.grad =", b.grad)
print("e.grad =", e.grad)
Line by line: what each line does
  • e._backward(): runs while e.grad is still 0, so it hands -3 x 0 and 2 x 0 to a and b.
  • L._backward(): fills in e.grad correctly, but too late. No error, no warning: two zeros that read as "this input does not matter." So a node must wait until all of its users have handed their shares down before it hands on its own.
saved output · press Run to reproduce livea.grad = 0.0 b.grad = 0.0 e.grad = 1.0

Step 4: Let the computer find the order

Put the nodes in a list where inputs always appear before whatever they feed (socks before shoes), then read the list backwards. Such a list is a topological order. build makes it by recursion: before listing a node, list its inputs.

python · runnable
def topo_order(root):
    order, visited = [], set()
    def build(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build(child)
            order.append(v)
    build(root)
    return order
Line by line: what each line does
  • order, visited = [], set(): a list to fill and a set to remember which nodes are already handled.
  • def build(v):: the procedure for one node, a closure over order and visited.
  • if v not in visited: visited.add(v): handle each node once, even when two paths reach it.
  • for child in v._prev: build(child): the recursion: list every input first.
  • order.append(v): only then list v itself.
  • build(root); return order: start at the final node and return the finished list.
python · runnable
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
e = a * b
L = e + c
print([v.data for v in topo_order(L)])
Line by line: what each line does
  • topo_order(L): gives a, b, e, c, L: inputs always before the node they feed.
  • [v.data for v in ...]: a list comprehension printing each node's number so you can recognize it.
saved output · press Run to reproduce live[2.0, -3.0, -6.0, 10.0, 4.0]
python · runnable
L.grad = 1.0
for v in reversed(topo_order(L)):
    v._backward()
print("a.grad =", a.grad, " b.grad =", b.grad,
      " c.grad =", c.grad)
Line by line: what each line does
  • reversed(topo_order(L)): visits L, c, e, b, a, so users always come before the nodes they use.
  • v._backward(): L's and e's rules do the work; the leaves' rules do nothing. The pencil's answers, and nobody chose the order by hand.
saved output · press Run to reproduce livea.grad = -3.0 b.grad = 2.0 c.grad = 1.0

Step 5: Finish the engine and check it

New in this version: backward() becomes a method, operations accept bare numbers like 1 or 2, there are rules for ** and tanh, and - and / are defined through +, * and **, which already carry rules. This is the class the rest of the notebook uses.

python · runnable
import math

class Value:
    """A number that remembers how it was made."""
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = _children
        self._op = _op

    def __add__(self, other):
        if not isinstance(other, Value):     # new
            other = Value(other)             # new
        out = Value(self.data + other.data,
                    (self, other), '+')
        def _backward():
            self.grad  += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        if not isinstance(other, Value):     # new
            other = Value(other)             # new
        out = Value(self.data * other.data,
                    (self, other), '*')
        def _backward():
            self.grad  += other.data * out.grad
            other.grad += self.data  * out.grad
        out._backward = _backward
        return out

    def __pow__(self, k):
        out = Value(self.data ** k, (self,), f'**{k}')
        def _backward():
            # rate = k * a**(k-1)
            self.grad += k * self.data**(k-1) * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,), 'tanh')
        def _backward():
            # rate = 1 - t**2
            self.grad += (1 - t**2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        self.grad = 1.0
        for v in reversed(topo_order(self)):
            v._backward()

    def __neg__(self):
        return self * -1
    def __radd__(self, o):
        return self + o
    def __sub__(self, o):
        return self + (-o)
    def __rsub__(self, o):
        return o + (-self)
    def __rmul__(self, o):
        return self * o
    def __truediv__(self, o):
        return self * o ** -1
    def __repr__(self):
        return (f"Value(data={self.data:.4f}, "
                f"grad={self.grad:.4f})")
Line by line: what each line does
  • import math: Python's math toolbox, for tanh.
  • if not isinstance(other, Value): other = Value(other): bare numbers such as the 1 in a + 1 get wrapped so every node in the graph is a Value.
  • def __pow__(self, k):: a ** k for a plain number k. Its local rate is the power rule, k * a**(k-1). (self,) is a one-item tuple: the comma makes the tuple.
  • def tanh(self):: squashes any number into -1..1. Its local rate is 1 - t**2; the rule reuses the remembered t rather than recomputing it.
  • def backward(self):: the three sweep lines from Step 4 as a method, using topo_order, which keeps the ordering code in a single spot.
  • __neg__, __sub__, __truediv__: -a is a * -1, a - b is a + (-b), a / b is a * b**-1: built from operations that already have rules.
  • __radd__, __rsub__, __rmul__: the reflected versions: Python asks the right operand when the left one, a plain number, cannot handle a Value. Without them 2 * a fails while a * 2 works.

Why __rmul__ and friends exist

The class defines __radd__, __rsub__ and __rmul__ alongside the ordinary ones. The r stands for reflected, and they exist because of the order Python resolves an operator in. Meeting 2 * a, it asks the left operand first: "int, can you multiply yourself by this thing?" An int has never heard of a Value, so it declines, and only then does Python turn to the right-hand operand and ask the mirrored question through __rmul__. Leave them out and a * 2 works while 2 * a fails.

The operand order inside __rsub__ matters too. There, self is the Value and o is the number on the left, so 2 - a has to come out as 2 + (-a). The other way round quietly computes a - 2 — the negation of the right answer.

python · runnable
a = Value(3.0)
print("a * 2 uses __mul__  ->", (a * 2).data)
print("2 * a uses __rmul__ ->", (2 * a).data)
print("2 - a should be -1.0")
print("  o + (-self) gives ->", (2 - a).data)
print("  self - o would be ->", (a + (-2)).data)
print("sum([a, a]) uses __radd__ ->",
      sum([a, a]).data)
Line by line: what each line does
  • (a * 2).data and (2 * a).data: both give 6.0, but by different routes: the first through __mul__, the second through __rmul__ after int declined.
  • (2 - a).data: -1.0, which is what 2 - 3 should be.
  • (a + (-2)).data: 1.0 — what the wrong operand order would have produced. Both look like valid answers and neither raises an error, but only -1.0 is correct.
  • sum([a, a]): sum begins with a plain 0, and 0 + a can only be handled by __radd__. The neural net below adds up lists of Values this way.
saved output · press Run to reproduce livea * 2 uses __mul__ -> 6.0 2 * a uses __rmul__ -> 6.0 2 - a should be -1.0 o + (-self) gives -> -1.0 self - o would be -> 1.0 sum([a, a]) uses __radd__ -> 6.0

One backward() call on the running example reproduces the paper sweep:

python · runnable
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
L = a * b + c
L.backward()
print(a.grad, b.grad, c.grad)
Line by line: what each line does
  • L.backward(): seeds L with 1 and runs every rule in the order topo_order gives. Prints -3.0 2.0 1.0, the pencil's numbers.
saved output · press Run to reproduce live-3.0 2.0 1.0

A value used twice

Why += and not =? Because of values like this, used in two places:

python · runnable
z = Value(3.0)
M = z * z
print(M._prev[0] is M._prev[1])
M.backward()
print("z.grad =", z.grad)
Line by line: what each line does
  • M._prev[0] is M._prev[1]: is asks "are these the very same object?" True: both inputs of this multiplication are z.
  • M.backward(): topo_order lists z once, thanks to visited, so M's rule runs once.
  • print("z.grad =", z.grad): that one rule adds 3 for self and 3 more for other, both z: 6.0, the slope of z² at z = 3. With = instead of += the second would overwrite the first and leave 3.0. visited keeps the walk to once per node; += counts every use.
saved output · press Run to reproduce liveTrue z.grad = 6.0

When one value is used twice

M = z * z is the smallest case, but it hides something: both paths leave from the same operation. Real fan-out looks like this, where a feeds two different operations that meet again at the end:

python
      b = a * 2
    /           \
a                 L = b * c
    \           /
      c = a + 5

Nudging a moves L along both routes, so a.grad has to be the sum of what comes back along each.

python · runnable
a = Value(3.0)
b = a * 2           # route 1
c = a + 5           # route 2
L = b * c
L.backward()

print("forward:  b =", b.data, " c =", c.data, " L =", L.data)
print("b.grad =", b.grad, "  c.grad =", c.grad)
print("route 1 into a:", b.grad, "x 2 =", b.grad * 2)
print("route 2 into a:", c.grad, "x 1 =", c.grad * 1)
print("a.grad =", a.grad)
Line by line: what each line does
  • b = a * 2 / c = a + 5: two different operations reading the same a. That is the fan-out; a plain chain never does this.
  • b.grad = 8.0: the gradient that reached b. L = b × c, so the local rate toward b is the other input, c = 8.
  • c.grad = 6.0: same rule at the same node: the rate toward c is b = 6.
  • route 1 into a: carry 8 back through b = a × 2, whose local rate toward a is 2: 8 × 2 = 16.
  • route 2 into a: carry 6 back through c = a + 5; addition passes a nudge through unchanged, rate 1: 6 × 1 = 6.
  • a.grad = 22.0: 16 + 6. Neither route alone is the answer, and nothing in the code picks a winner — += lets both arrive and total up.
saved output · press Run to reproduce liveforward: b = 6.0 c = 8.0 L = 48.0 b.grad = 8.0 c.grad = 6.0 route 1 into a: 8.0 x 2 = 16.0 route 2 into a: 6.0 x 1 = 6.0 a.grad = 22.0

Watch the two routes pay in

The sum above happens one step at a time. Here is the same sweep with the loop unrolled, printing a.grad after every single _backward() call, so you can see it climb from 0 to 6 to 22 as each route arrives.

python · runnable
a = Value(3.0)
b = a * 2
c = a + 5
L = b * c

label = {id(a): 'a', id(b): 'b', id(c): 'c', id(L): 'L'}

L.grad = 1.0
print('%20s :  a.grad = %s' % ('start', a.grad))
for v in reversed(topo_order(L)):
    v._backward()
    who = label.get(id(v), 'const %g' % v.data)
    print('%20s :  a.grad = %s' % (who + '._backward()', a.grad))
Line by line: what each line does
  • label = {id(a): 'a', ...}: id(v) is a number unique to that exact object, so this dictionary lets the loop print each node's name.
  • L.grad = 1.0: seed the sweep at the far end. dL/dL is 1.
  • for v in reversed(topo_order(L)): the same loop backward() runs, spelled out so we can print between steps.
  • v._backward(): one step of the sweep. Watch a.grad: 0, then 6 when the route through c lands, then 22 when the route through b does.
  • const %g: the constants 2 and 5 became Value nodes too, so they appear in the walk; they have no inputs, so their _backward does nothing.
saved output · press Run to reproduce live start : a.grad = 0.0 L._backward() : a.grad = 0.0 c._backward() : a.grad = 6.0 const 5._backward() : a.grad = 6.0 b._backward() : a.grad = 22.0 const 2._backward() : a.grad = 22.0 a._backward() : a.grad = 22.0

The order is not arbitrary. a is reached last, and that is the whole point: it is used by both b and c, so its gradient is only complete once both have run. Had the sweep reached a straight after c, it would have carried 6 onward — a number that looks plausible and is wrong. That is what topo_order protects against.

There is an interactive version of this exact graph, where you step the sweep and watch a.grad climb: https://llm-course.wazeem.com/labs/backprop-sweep.html

Does it work? Check against numerical gradients

The honest test of an autograd engine is whether it produces the same gradients as brute force. We build a small expression, L = (a*b + b**2) * tanh(c), and call .backward() once, which fills in a.grad, b.grad, and c.grad by the chain rule. We then compare these against the numerical slope from notebook 01: nudge each input by a tiny amount, see how much L moves, and divide the change by the distance covered. The two agree to five decimal places, so the engine is correct. (These are the same three numbers PyTorch will reproduce in notebook 07.)

build the graphevery +, ×, **, tanh becomes a node that remembers its inputs
▼
forward passfill each node's .data from its inputs, bottom to top, up to L
▼
seed the outputset L.grad = 1 — nudging L changes L one-for-one
▼
backward passvisit nodes in reverse; each hands grad to its inputs via its own local rule
▼
read .grad everywhereeach node now holds dL/dnode — exactly the table below
Interactive · forward & backward

The whole graph, alive

The exact expression checked above — L = (a*b + b**2) · tanh(c) — really is this little graph our engine built. Drag a, b, c and every .data (top number) updates: that is the forward pass. Press Run backward and watch each .grad (bottom number) fill in, right to left — the chain rule, one node at a time.

top = .data (forward value)bottom = .grad = dL/dnode (backward)

python · runnable
a = Value(2.0)
b = Value(-3.0)
c = Value(0.5)
L = (a*b + b**2) * c.tanh()
L.backward()
print("L      = %.5f" % L.data)
print("a.grad = %.5f" % a.grad)
print("b.grad = %.5f" % b.grad)
print("c.grad = %.5f" % c.grad)
Line by line: what each line does
  • a, b, c: three tracked numbers.
  • L = (a*b + b**2) * c.tanh(): ordinary-looking math, but every +, *, ** and tanh builds a node that remembers its inputs.
  • L.backward(): one call; afterwards a.grad, b.grad, c.grad hold how much L moves per tiny nudge of each input. b is used twice; += adds the two contributions.
  • "%.5f" % L.data: %-formatting: print with 5 decimal places.
saved output · press Run to reproduce liveL = 1.38635 a.grad = -1.38635 b.grad = -1.84847 c.grad = 2.35934
python · runnable
def numeric(fn, x, h=1e-6):
    return (fn(x + h) - fn(x - h)) / (2 * h)

f_a = lambda x: (x*(-3.0) + (-3.0)**2) * math.tanh(0.5)
f_b = lambda x: (2.0*x + x**2) * math.tanh(0.5)
f_c = lambda x: (2.0*(-3.0) + (-3.0)**2) * math.tanh(x)
da = numeric(f_a, 2.0)
db = numeric(f_b, -3.0)
dc = numeric(f_c, 0.5)
print("numeric: a=%.5f  b=%.5f  c=%.5f"
      % (da, db, dc))
print("match ->", all(abs(x - y) < 1e-4 for x, y in
      [(a.grad, da), (b.grad, db), (c.grad, dc)]))
Line by line: what each line does
  • def numeric(fn, x, h=1e-6):: the brute-force slope check from notebook 01: nudge a hair above and below x and divide the change by the distance, 2 * h.
  • f_a, f_b, f_c: the same formula with only one input free to vary; the other two are fixed numbers.
  • da, db, dc: the three measured slopes.
  • all(abs(x - y) < 1e-4 for ...): True only if every pair agrees to four decimals: the engine matches reality.
saved output · press Run to reproduce livenumeric: a=-1.38635 b=-1.84847 c=2.35934 match -> True

Every node got a gradient

.backward() filled in a .grad on every node in the expression, not only the three inputs. topo_order lists them all, in order. Start from the final row, L with gradient 1, and read upward: the chain rule has reached every node.

python · runnable
for v in topo_order(L):
    print(v._op or "input",
          round(v.data, 4), round(v.grad, 4))
Line by line: what each line does
  • for v in topo_order(L):: every node of the graph, each after the nodes that feed it.
  • v._op or "input": or gives its left value if it is non-empty, otherwise the right one. Leaves have an empty _op, so they print as input.
  • round(v.data, 4), round(v.grad, 4): each node's forward value and its gradient, dL/dnode, rounded to four places. tanh shows 3.0: in the last multiplication its partner is the sum, and the sum equals 3.
saved output · press Run to reproduce liveinput 2.0 -1.3864 input -3.0 -1.8485 * -6.0 0.4621 **2 9.0 0.4621 + 3.0 0.4621 input 0.5 2.3593 tanh 0.4621 3.0 * 1.3864 1.0

A tiny neural net, trained with our engine

Now we put the engine to real use. We will stack Value objects into a small neural network and train it on a toy problem, with no PyTorch, using only the autograd we just wrote.

The building blocks:

  • a neuron takes its inputs, multiplies each by a learned weight, adds the results together along with one extra learned number called a bias (so the whole step is a weighted vote), and then passes the result through tanh, a standard "squashing" function that gently forces any number into the range -1 to 1. The weights and the bias are the knobs the neuron learns.
  • a layer is simply several neurons side by side, each looking at the same inputs.
  • an MLP, short for multi-layer perceptron, stacks layers, feeding the outputs of one layer in as the inputs of the next.

MLP(2, [8, 8, 1]) means 2 inputs, then a layer of 8 neurons, then another 8, then a single output. That comes to 105 individual knobs (the parameters) for the engine to tune. Training is the same loop as always: a forward pass to predict, a measure of how far off we are (the squared error this time, which is the gap between prediction and target, squared so that every miss counts as a positive amount, since these outputs are plain numbers rather than probabilities), then loss.backward() to fill in every knob's .grad, then the update -- the line that actually changes the knobs. Repeat, and the loss falls until the predictions settle onto the +1 and -1 targets.

The update step, and what "SGD" means. All the learning happens in one line: p.data -= 0.05 * p.grad. Each knob's .grad says which way to move it to raise the loss, so we step the opposite way -- subtract a small slice of the gradient -- to lower it. The size of that slice is the learning rate (here 0.05): too small and training crawls, too large and it overshoots the bottom and bounces. Doing this over and over is gradient descent -- the same rolling-downhill picture from notebook 01, now running on all 105 knobs at once.

The code comment calls it an SGD step, short for stochastic gradient descent. The only extra idea in the word "stochastic" is randomness: real training does not measure the gradient on the whole dataset each step, it uses a fresh random handful of examples (a mini-batch) -- far cheaper, and the little bit of noise even helps it generalize. Our toy here has just 8 points, so we use all of them every step (technically full-batch gradient descent), but the update line is identical. The bigram in notebook 03 and the GPT in notebook 09 both take the stochastic route, drawing a new random batch on every step.

python · runnable
import random
random.seed(42)

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(0.0)
    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh()
    def parameters(self): return self.w + [self.b]

class Layer:
    def __init__(self, nin, nout): self.neurons = [Neuron(nin) for _ in range(nout)]
    def __call__(self, x):
        outs = [n(x) for n in self.neurons]
        return outs[0] if len(outs) == 1 else outs
    def parameters(self): return [p for n in self.neurons for p in n.parameters()]

class MLP:
    def __init__(self, nin, nouts):
        sizes = [nin] + nouts
        self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(nouts))]
    def __call__(self, x):
        for layer in self.layers: x = layer(x)
        return x
    def parameters(self): return [p for layer in self.layers for p in layer.parameters()]

model = MLP(2, [8, 8, 1])    # 2 inputs -> 8 -> 8 -> 1 output
print("parameters:", len(model.parameters()))
Line by line: what each line does
  • import random; random.seed(42): plain-Python randomness for the starting weights, seeded for repeatability.
  • class Neuron:: one neuron: a list of weights self.w (one Value per input, started random) plus a bias self.b.
  • def __call__(self, x):: defining __call__ lets you use the object like a function, writing n(x).
  • act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b): zip pairs each weight with its input; multiply and add them all up, starting the sum from the bias. A weighted vote.
  • return act.tanh(): squash that vote into the range -1..1. Every step here is a tracked Value op, so gradients will flow through.
  • class Layer:: a row of neurons that all read the same inputs; returns their outputs as a list.
  • class MLP:: chains layers: for layer in self.layers: x = layer(x) feeds each layer's output into the next.
  • def parameters(self): (at each level) -- gather every weight and bias into one flat list, so the trainer can reach all 105 knobs.
  • model = MLP(2, [8, 8, 1]): build the network: 2 inputs → 8 neurons → 8 → 1 output.
saved output · press Run to reproduce liveparameters: 105

Where the 105 knobs live, and one neuron by hand

MLP(2, [8, 8, 1]) reports 105 parameters. That number is not magic -- it is just every weight and bias added up. Let's break it down layer by layer, then compute a single neuron the long way so "a weighted vote, then tanh" stops being words.

x₁ x₂ × w₁ × w₂ Σ + b tanh out weighted vote squash to (−1, 1)

One neuron: multiply each input by a weight, add them with a bias, then squash with tanh — out = tanh(w₁x₁ + w₂x₂ + b). That is 3 numbers for this 2-input neuron (2 weights + 1 bias). Stack 8 of them for layer 1, feed those into 8 more, then into 1 — and you get the 105 weights the engine tunes.

python · runnable
# How the 105 parameters are distributed, and one neuron computed by hand.
total = 0
for li, layer in enumerate(model.layers):
    nout = len(layer.neurons); nin = len(layer.neurons[0].w)
    pc = nout * (nin + 1); total += pc
    print(f"layer {li}: {nout} neurons x ({nin} weights + 1 bias) = {pc} params")
print(f"total = {total}   (equals len(model.parameters()) = {len(model.parameters())})")

nrn = model.layers[0].neurons[0]; xin = [1.0, 1.0]     # one neuron, first layer
s = sum(w.data * xi for w, xi in zip(nrn.w, xin)) + nrn.b.data
print(f"\nneuron[0][0] on input {xin}:")
print(f"   weights = {[round(w.data, 3) for w in nrn.w]}, bias = {nrn.b.data}")
print(f"   weighted vote = {s:+.3f}  ->  tanh = {math.tanh(s):+.3f}   (the neuron's output)")
Line by line: what each line does
  • The loop reads each layer straight off the model: number of neurons times (one weight per input + one bias).
  • 24 + 72 + 9 = 105: exactly what len(model.parameters()) reported -- no hidden knobs.
  • nrn = model.layers[0].neurons[0]: pull out a single neuron from the first layer.
  • s = sum(w.data * xi ...) + nrn.b.data: its weighted vote -- each input times its weight, plus the bias.
  • math.tanh(s): squash that vote into (-1, 1). That one number is the neuron's output, and 105 of these knobs are what training tunes.
saved output · press Run to reproduce livelayer 0: 8 neurons x (2 weights + 1 bias) = 24 params layer 1: 8 neurons x (8 weights + 1 bias) = 72 params layer 2: 1 neurons x (8 weights + 1 bias) = 9 params total = 105 (equals len(model.parameters()) = 105) neuron[0][0] on input [1.0, 1.0]: weights = [0.279, -0.95], bias = 0.0 weighted vote = -0.671 -> tanh = -0.586 (the neuron's output)
python · runnable
# toy dataset: two interleaved blobs (label +1 / -1)
xs = [[ 1.0,  1.0], [ 1.5,  0.5], [ 0.5,  1.5], [ 2.0,  1.0],
      [-1.0, -1.0], [-1.5, -0.5], [-0.5, -1.5], [-2.0, -1.0]]
ys = [1.0]*4 + [-1.0]*4

losses = []
for step in range(120):
    preds = [model(x) for x in xs]                       # forward
    loss = sum((p - y)**2 for p, y in zip(preds, ys))    # sum of squared errors
    for p in model.parameters(): p.grad = 0.0            # reset grads
    loss.backward()                                      # backprop (our engine!)
    for p in model.parameters(): p.data -= 0.05 * p.grad # SGD step
    losses.append(loss.data)

print("loss: %.4f -> %.4f" % (losses[0], losses[-1]))
print("predictions:", [round(model(x).data, 2) for x in xs])
print("targets    :", ys)
Line by line: what each line does
  • xs = [[1.0, 1.0], ...]; ys = [1.0]*4 + [-1.0]*4: eight 2-number points and their labels: the first four should output +1, the last four -1.
  • for step in range(120):: train for 120 rounds.
  • preds = [model(x) for x in xs]: run the network on all eight points.
  • loss = sum((p - y)**2 for p, y in zip(preds, ys)): squared error: (prediction minus target)², summed. Squaring makes every miss positive and punishes big misses extra.
  • for p in model.parameters(): p.grad = 0.0: reset every knob's gradient (they accumulate, so stale ones must be cleared each round).
  • loss.backward(): our engine fills in all 105 .grad slots in one sweep.
  • for p in model.parameters(): p.data -= 0.05 * p.grad: the update -- the moment of learning. Each .grad points uphill (toward more loss), so subtract a small fraction of it to step every knob downhill; that fraction 0.05 is the learning rate. This is gradient descent. The comment's SGD (stochastic gradient descent) is the same rule, just with the gradient measured on a fresh random mini-batch each step -- as the bigram does in notebook 03 -- instead of on all the data at once.
  • round(model(x).data, 2): after training, the predictions hug the +1 / -1 targets.
saved output · press Run to reproduce liveloss: 0.0328 -> 0.0013 predictions: [0.99, 0.99, 0.99, 0.99, -0.99, -0.99, -0.99, -0.99] targets : [1.0, 1.0, 1.0, 1.0, -1.0, -1.0, -1.0, -1.0]
python · runnable
import matplotlib.pyplot as plt
plt.figure(figsize=(6,3)); plt.plot(losses)
plt.title("training a neural net on a hand-written autograd engine"); plt.xlabel("step"); plt.ylabel("loss"); plt.show()
Line by line: what each line does
  • import matplotlib.pyplot as plt: the plotting toolbox.
  • plt.plot(losses): with a single list, matplotlib uses 0,1,2,... for the x-axis: loss per step, falling.
  • Title, axis labels, and plt.show(): the standard plotting finish.
saved output · press Run to reproduce liveoutput figure

See what it learned: the decision boundary

The loss curve tells you training worked, but not what the net now believes. So feed a whole grid of points through the trained model and colour each region by the sign of the output. The eight training points fall on the right sides of a boundary the engine bent to fit them -- learned with nothing but the autograd we wrote at the top of this notebook.

Interactive · the net you trained

The neural network you just built

This is MLP(2, [8, 8, 1]) after the 120 training steps above — 2 inputs → 8 → 8 → 1 output. Its 105 real learned weights are the edges (blue = positive, red = negative, thickness = strength). Drag inside the square to move the test point: the net runs a forward pass, every neuron lights up with its tanh activation, and the readout says which class. The shaded regions are the actual boundary it learned.

2 in · 8 · 8 · 1 out
watch the signal flow input → output, layer by layer

Circles are the 4 points labelled +1, squares the 4 labelled −1. The net bends a smooth boundary between them — trained entirely by the ~40-line engine you wrote, no PyTorch.

python · runnable
# Feed a grid of points through the trained net; colour each by the sign of its output.
xr = [-3 + 6 * i / 43 for i in range(44)]
yr = [-2.5 + 5 * j / 35 for j in range(36)]
Z = [[model([X, Y]).data for X in xr] for Y in yr]     # 1584 forward passes

plt.figure(figsize=(5.2, 4.2))
plt.contourf(xr, yr, Z, levels=[-100, 0, 100], colors=["#f4b7b7", "#b7c9f4"])
plt.contour(xr, yr, Z, levels=[0], colors=["#333"], linewidths=1)
plt.scatter([p[0] for p in xs[:4]], [p[1] for p in xs[:4]], c="#1c4fd6", edgecolors="k", s=90, label="+1")
plt.scatter([p[0] for p in xs[4:]], [p[1] for p in xs[4:]], c="#d61c1c", edgecolors="k", s=90, marker="s", label="-1")
plt.legend(); plt.title("the decision boundary our engine learned")
plt.xlabel("x1"); plt.ylabel("x2"); plt.show()
Line by line: what each line does
  • xr, yr: a grid of x and y coordinates covering the plot area.
  • Z = [[model([X, Y]).data ...]]: run the trained net on every grid point -- 1584 forward passes -- and keep each output number.
  • plt.contourf(..., levels=[-100, 0, 100]): paint the plane in two colours, split at output 0 -- the model's decision.
  • plt.contour(..., levels=[0]): draw the boundary line itself, where the net switches from -1 to +1.
  • plt.scatter(...): the eight training points on top; each sits in the correctly coloured region, so the net separated the two classes.
saved output · press Run to reproduce liveoutput figure

Recap

You have built autograd: a graph of operations that differentiates itself by multiplying local rates along the chain, and you trained a real, if tiny, neural network on top of it. PyTorch's .backward() is exactly this idea, generalized from single numbers to whole tensors (a tensor is just a grid of numbers, as in notebook 01) and written in fast, low-level code for speed. From here on we let the framework handle the bookkeeping, but you now know precisely what it is doing underneath.

Next, notebook 05 covers self-attention, the mechanism that lets a token look back over the whole context rather than just one character.

⬇ Download this lesson as a notebook — 04_micrograd.ipynb