{
"cells": [
{
"cell_type": "markdown",
"id": "01e265c7",
"metadata": {},
"source": [
"# 08 - Build the GPT\n",
"\n",
"Now we assemble everything (attention, multi-head attention, the feed-forward network, LayerNorm, the residuals, and the embeddings) into a complete **GPT**. To keep it reusable, the model lives in a file called `gpt.py`, so that notebooks 09 and 10 can simply import it. This notebook is a guided tour of that file: we read each class in turn and explain what it does and why.\n",
"\n",
"Here is the whole journey a token takes, from top to bottom:\n",
"\n",
"```\n",
"token ids -> token embedding + position embedding\n",
" -> N x Transformer block (attention + feed-forward, with residuals)\n",
" -> final LayerNorm\n",
" -> linear head -> logits over the vocabulary (next-token scores)\n",
"```\n",
"\n",
"In words: turn each token id into a vector and add in *where* it sits; let the stack of blocks mix and refine those vectors; give them a final clean-up; then read out a score for every possible next token. That last list of scores is exactly what softmax and sampling turn into the next character."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "colab-setup-08",
"metadata": {},
"outputs": [],
"source": [
"# Colab setup -- fetch the files this notebook needs.\n",
"# (Does nothing when run locally in the course folder.)\n",
"import os, urllib.request\n",
"BASE = (\"https://raw.githubusercontent.com/waze\"\n",
" \"emlabs/llm-book-code/main/\")\n",
"for f in ['gpt.py', 'data/input.txt']:\n",
" if not os.path.exists(f):\n",
" d = os.path.dirname(f)\n",
" if d: os.makedirs(d, exist_ok=True)\n",
" urllib.request.urlretrieve(BASE + f, f)\n",
" print(\"downloaded\", f)\n"
]
},
{
"cell_type": "markdown",
"id": "colab-setup-md-08",
"metadata": {},
"source": [
"Line by line: what each line does\n
\n
import os, urllib.request: two standard-library toolboxes: checking files on disk, and downloading from the web.
\n
BASE = (...): the web address of this course's folder on GitHub, split over two lines (Python glues adjacent strings together).
\n
for f in [...]: loop over the file names this notebook needs.
\n
if not os.path.exists(f): only download what is missing -- running locally, everything already exists, so nothing happens.
\n
os.makedirs(d, exist_ok=True): create the folder for the file if it has one (exist_ok means don't complain if it's already there).
\n
urllib.request.urlretrieve(...): download the file and save it under the same name here.
import torch, inspect: PyTorch, plus inspect, a standard tool that can fetch a class's source code as text.
\n",
"
from gpt import GPTConfig, Head, ...: import the model's classes from the local file gpt.py, so notebooks 08-10 all share the exact same code.
\n",
"
torch.manual_seed(1337): fix the seed so the untrained sample lower down is reproducible (the same seed the other notebooks use).
\n",
"
The device line picks Apple GPU → NVIDIA → CPU, as before.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "c8df2c15",
"metadata": {},
"source": [
"## Config\n",
"\n",
"Every size of the model, gathered in one place. Think of these as the dials you turn up for a larger and more capable (but slower and more demanding) model: more **layers** (`n_layer`) makes it deeper; a wider embedding (`n_embd`) gives each token more room to represent meaning; more **heads** (`n_head`) lets it track more relationships at once; and a longer **block_size** lets it look further back. Our values are deliberately tiny, so the whole thing trains in about a minute."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8009730d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.945360Z",
"iopub.status.busy": "2026-07-01T09:18:31.945277Z",
"iopub.status.idle": "2026-07-01T09:18:31.947787Z",
"shell.execute_reply": "2026-07-01T09:18:31.947482Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"@dataclass\n",
"class GPTConfig:\n",
" vocab_size: int = 65 # number of distinct tokens\n",
" block_size: int = 128 # maximum context length (tokens the model can look back over)\n",
" n_layer: int = 4 # number of Transformer blocks\n",
" n_head: int = 4 # attention heads per block\n",
" n_embd: int = 128 # embedding / residual-stream width\n",
" dropout: float = 0.0 # dropout probability (set >0 to regularize larger runs)\n",
"\n"
]
}
],
"source": [
"print(inspect.getsource(GPTConfig))"
]
},
{
"cell_type": "markdown",
"id": "7b9b4e05",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
print(inspect.getsource(GPTConfig)): fetch and print the class's real source from gpt.py. What you see below is the actual code.
\n",
"
@dataclass: a decorator (the @ line above the class) that auto-writes the boring constructor: you just declare fields with types and defaults, and GPTConfig(vocab_size=65) works.
\n",
"
vocab_size: int = 65: a field: its name, its type, its default. Pure settings, no behavior. The other fields (block_size, n_layer, n_head, n_embd) are the size dials you turn up for a bigger model.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "6b131acb",
"metadata": {},
"source": [
"## One attention head\n",
"\n",
"This is identical to notebook 05, now written as a trainable `nn.Module`. `key`, `query`, and `value` are the learned projections; the `tril` buffer enforces causality (no looking at the future); and the `1/sqrt(d)` scaling keeps softmax soft (notebook 05, step 4). Nothing here is new; it is the same head, wired up so that autograd can train it."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "dcd6f19f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.948949Z",
"iopub.status.busy": "2026-07-01T09:18:31.948880Z",
"iopub.status.idle": "2026-07-01T09:18:31.951203Z",
"shell.execute_reply": "2026-07-01T09:18:31.950869Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"class Head(nn.Module):\n",
" \"\"\"One head of causal self-attention (notebook 05, now with learned, trainable weights).\"\"\"\n",
"\n",
" def __init__(self, cfg: GPTConfig, head_size: int):\n",
" super().__init__()\n",
" self.key = nn.Linear(cfg.n_embd, head_size, bias=False)\n",
" self.query = nn.Linear(cfg.n_embd, head_size, bias=False)\n",
" self.value = nn.Linear(cfg.n_embd, head_size, bias=False)\n",
" # a constant lower-triangular mask; registered as a buffer so it moves with .to(device)\n",
" self.register_buffer(\"tril\", torch.tril(torch.ones(cfg.block_size, cfg.block_size)))\n",
" self.dropout = nn.Dropout(cfg.dropout)\n",
"\n",
" def forward(self, x):\n",
" B, T, C = x.shape\n",
" k = self.key(x) # (B, T, head_size)\n",
" q = self.query(x) # (B, T, head_size)\n",
" # scaled dot-product affinities, then mask out the future\n",
" wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5 # (B, T, T)\n",
" wei = wei.masked_fill(self.tril[:T, :T] == 0, float(\"-inf\"))\n",
" wei = F.softmax(wei, dim=-1)\n",
" wei = self.dropout(wei)\n",
" v = self.value(x) # (B, T, head_size)\n",
" return wei @ v # (B, T, head_size)\n",
"\n"
]
}
],
"source": [
"print(inspect.getsource(Head))"
]
},
{
"cell_type": "markdown",
"id": "b35341c2",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
This prints notebook 07's Head, with two production touches:
\n",
"
self.key/query/value = nn.Linear(...): the three learned projections, as layers.
\n",
"
self.register_buffer(\"tril\", ...): the causal mask, carried with the model but not trained.
\n",
"
self.dropout = nn.Dropout(cfg.dropout): during training, randomly zero a fraction of the attention weights (regularization, explained in notebook 09). With dropout=0.0 it does nothing.
\n",
"
wei = q @ k.transpose(-2,-1) * k.shape[-1]**-0.5 ... -- scale, mask, softmax, blend: line-for-line what you built in notebooks 05 and 07.
\n",
"
The \"\"\"...\"\"\" right under the class line is a docstring -- documentation tools can read it; Python ignores it at runtime.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "8d9c815a",
"metadata": {},
"source": [
"## Multi-head attention\n",
"\n",
"Run several heads in parallel and join their outputs together, so the model can attend to different kinds of relationship at once, then use a projection (`proj`) to mix them back to the embedding width (notebook 06). The split `head_size = n_embd // n_head` is what keeps the heads affordable: four heads each do a quarter-width job, so the total cost matches that of a single full-width head."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ff5838ee",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.952166Z",
"iopub.status.busy": "2026-07-01T09:18:31.952109Z",
"iopub.status.idle": "2026-07-01T09:18:31.954367Z",
"shell.execute_reply": "2026-07-01T09:18:31.953980Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"class MultiHeadAttention(nn.Module):\n",
" \"\"\"Several attention heads in parallel, concatenated and projected (notebook 06).\"\"\"\n",
"\n",
" def __init__(self, cfg: GPTConfig):\n",
" super().__init__()\n",
" head_size = cfg.n_embd // cfg.n_head\n",
" self.heads = nn.ModuleList([Head(cfg, head_size) for _ in range(cfg.n_head)])\n",
" self.proj = nn.Linear(cfg.n_embd, cfg.n_embd)\n",
" self.dropout = nn.Dropout(cfg.dropout)\n",
"\n",
" def forward(self, x):\n",
" out = torch.cat([h(x) for h in self.heads], dim=-1) # (B, T, n_embd)\n",
" return self.dropout(self.proj(out))\n",
"\n"
]
}
],
"source": [
"print(inspect.getsource(MultiHeadAttention))"
]
},
{
"cell_type": "markdown",
"id": "e8267c2d",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
head_size = cfg.n_embd // cfg.n_head: split the width across heads (128 / 4 = 32 each).
\n",
"
self.heads = nn.ModuleList([Head(cfg, head_size) for _ in range(cfg.n_head)]): a list of heads that PyTorch knows about (a plain Python list would hide their parameters from the optimizer).
\n",
"
torch.cat([h(x) for h in self.heads], dim=-1): run every head, then glue their outputs side by side along the last axis: 4 heads × 32 = 128. This is the \"concatenate\" from notebook 06.
\n",
"
self.proj then self.dropout: a final projection that mixes the heads, then dropout.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "c96b0c8e",
"metadata": {},
"source": [
"## Feed-forward\n",
"\n",
"After the tokens have traded information through attention, each one is processed on its own by a small MLP: expand to four times the width, apply GELU, then project back (notebook 06). This is the \"now think about what you just heard\" step, and most of the model's numbers live here."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "b7d1c60a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.955325Z",
"iopub.status.busy": "2026-07-01T09:18:31.955270Z",
"iopub.status.idle": "2026-07-01T09:18:31.957398Z",
"shell.execute_reply": "2026-07-01T09:18:31.957111Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"class FeedForward(nn.Module):\n",
" \"\"\"Per-token MLP: expand 4x, nonlinearity, project back. Where most parameters live.\"\"\"\n",
"\n",
" def __init__(self, cfg: GPTConfig):\n",
" super().__init__()\n",
" self.net = nn.Sequential(\n",
" nn.Linear(cfg.n_embd, 4 * cfg.n_embd),\n",
" nn.GELU(),\n",
" nn.Linear(4 * cfg.n_embd, cfg.n_embd),\n",
" nn.Dropout(cfg.dropout),\n",
" )\n",
"\n",
" def forward(self, x):\n",
" return self.net(x)\n",
"\n"
]
}
],
"source": [
"print(inspect.getsource(FeedForward))"
]
},
{
"cell_type": "markdown",
"id": "cb899ace",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
self.net = nn.Sequential(...): a pipeline: each layer feeds the next, in order. Calling self.net(x) runs the whole chain.
nn.Dropout(cfg.dropout): the same regularization knob at the end.
\n",
"
def forward(self, x): return self.net(x): just run the pipeline.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "9d0dd305",
"metadata": {},
"source": [
"## The block\n",
"\n",
"One block is attention (communicate) followed by the feed-forward network (think), each wrapped in pre-norm and a residual connection. Stacking these blocks is the entire depth of the network. The residual, written `x = x + sublayer(norm(x))` (keep the input, add an edit on top), is what lets the gradients flow cleanly through a deep stack (notebook 06)."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "64127c66",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.958404Z",
"iopub.status.busy": "2026-07-01T09:18:31.958349Z",
"iopub.status.idle": "2026-07-01T09:18:31.960649Z",
"shell.execute_reply": "2026-07-01T09:18:31.960342Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"class Block(nn.Module):\n",
" \"\"\"Transformer block: communicate (attention), then compute (MLP). Pre-norm + residuals.\"\"\"\n",
"\n",
" def __init__(self, cfg: GPTConfig):\n",
" super().__init__()\n",
" self.ln1 = nn.LayerNorm(cfg.n_embd)\n",
" self.sa = MultiHeadAttention(cfg)\n",
" self.ln2 = nn.LayerNorm(cfg.n_embd)\n",
" self.ffwd = FeedForward(cfg)\n",
"\n",
" def forward(self, x):\n",
" x = x + self.sa(self.ln1(x)) # residual connection around attention\n",
" x = x + self.ffwd(self.ln2(x)) # residual connection around the feed-forward\n",
" return x\n",
"\n"
]
}
],
"source": [
"print(inspect.getsource(Block))"
]
},
{
"cell_type": "markdown",
"id": "2b6b50a5",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
self.ln1 = nn.LayerNorm(cfg.n_embd) (and ln2) -- the LayerNorm you wrote by hand in notebook 06, now built in, with learned dials.
\n",
"
self.sa = MultiHeadAttention(cfg) / self.ffwd = FeedForward(cfg): the two sub-layers.
\n",
"
x = x + self.sa(self.ln1(x)): normalize → attend → add back (residual).
\n",
"
x = x + self.ffwd(self.ln2(x)): normalize → think → add back. Two lines, the whole block.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "c15a3998",
"metadata": {},
"source": [
"## The full GPT\n",
"\n",
"This is the whole model, and every line should look familiar by now:\n",
"\n",
"- `token_embedding`: a lookup table that turns each token id into a learnable vector (its \"meaning\"). Nothing assigns these meanings by hand; the vectors start out random and are nudged by training until tokens used in similar ways end up with similar vectors. Meaning emerges from usage.\n",
"- `position_embedding`: a second table that adds in *where* the token sits (notebook 06's position information, learned here rather than fixed as sinusoids).\n",
"- `blocks`: the stack of Transformer blocks that do the real work.\n",
"- `ln_f` and then `lm_head`: a final LayerNorm clean-up, followed by a linear layer that turns each position's vector into **logits over the vocabulary**, that is, a score for every possible next token.\n",
"\n",
"If you pass in `targets`, the model also returns the cross-entropy loss (notebook 01). And `generate` is the autoregressive sampling loop from notebook 03, with two new dials we will explore in notebook 10 (`temperature` and `top_k`): predict, take the last position's logits, optionally filter them, apply softmax, draw one token, append it, and repeat, always cropping the context to the last `block_size` tokens so the model never looks back further than it was trained to."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ab6bbd71",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.961617Z",
"iopub.status.busy": "2026-07-01T09:18:31.961566Z",
"iopub.status.idle": "2026-07-01T09:18:31.963985Z",
"shell.execute_reply": "2026-07-01T09:18:31.963682Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"class GPT(nn.Module):\n",
" \"\"\"The full model: embeddings -> stacked blocks -> final norm -> next-token logits.\"\"\"\n",
"\n",
" def __init__(self, cfg: GPTConfig):\n",
" super().__init__()\n",
" self.cfg = cfg\n",
" self.token_embedding = nn.Embedding(cfg.vocab_size, cfg.n_embd)\n",
" self.position_embedding = nn.Embedding(cfg.block_size, cfg.n_embd)\n",
" self.blocks = nn.Sequential(*[Block(cfg) for _ in range(cfg.n_layer)])\n",
" self.ln_f = nn.LayerNorm(cfg.n_embd)\n",
" self.lm_head = nn.Linear(cfg.n_embd, cfg.vocab_size)\n",
" self.apply(self._init_weights)\n",
"\n",
" def _init_weights(self, module):\n",
" if isinstance(module, nn.Linear):\n",
" nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
" if module.bias is not None:\n",
" nn.init.zeros_(module.bias)\n",
" elif isinstance(module, nn.Embedding):\n",
" nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
"\n",
" def forward(self, idx, targets=None):\n",
" B, T = idx.shape\n",
" tok = self.token_embedding(idx) # (B, T, n_embd)\n",
" pos = self.position_embedding(torch.arange(T, device=idx.device)) # (T, n_embd)\n",
" x = tok + pos # add position info\n",
" x = self.blocks(x) # (B, T, n_embd)\n",
" x = self.ln_f(x)\n",
" logits = self.lm_head(x) # (B, T, vocab_size)\n",
"\n",
" loss = None\n",
" if targets is not None:\n",
" B, T, V = logits.shape\n",
" loss = F.cross_entropy(logits.view(B * T, V), targets.view(B * T))\n",
" return logits, loss\n",
"\n",
" @torch.no_grad()\n",
" def generate(self, idx, max_new_tokens, temperature=1.0, top_k=None):\n",
" \"\"\"Autoregressively sample max_new_tokens, one token at a time.\"\"\"\n",
" for _ in range(max_new_tokens):\n",
" idx_cond = idx[:, -self.cfg.block_size:] # never look back further than block_size\n",
" logits, _ = self(idx_cond)\n",
" logits = logits[:, -1, :] / temperature # focus on the last position\n",
" if top_k is not None:\n",
" v, _ = torch.topk(logits, min(top_k, logits.size(-1)))\n",
" logits[logits < v[:, [-1]]] = -float(\"inf\")\n",
" probs = F.softmax(logits, dim=-1)\n",
" idx_next = torch.multinomial(probs, num_samples=1)\n",
" idx = torch.cat((idx, idx_next), dim=1)\n",
" return idx\n",
"\n",
" def num_params(self):\n",
" return sum(p.numel() for p in self.parameters())\n",
"\n"
]
}
],
"source": [
"print(inspect.getsource(GPT))"
]
},
{
"cell_type": "markdown",
"id": "f0929694",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
self.token_embedding = nn.Embedding(cfg.vocab_size, cfg.n_embd): the meaning table: 65 rows (one per character), 128 numbers each.
\n",
"
self.position_embedding = nn.Embedding(cfg.block_size, cfg.n_embd): the same idea for positions 0..127.
\n",
"
self.blocks = nn.Sequential(*[Block(cfg) for _ in range(cfg.n_layer)]): build n_layer Blocks; the * unpacks the list into separate arguments.
\n",
"
self.ln_f / self.lm_head: a final LayerNorm, then a Linear that turns each token's 128 numbers into 65 next-token scores.
\n",
"
self.apply(self._init_weights): run a small setup on every sub-module: start weights small-random (std 0.02), biases at zero. Good starting points matter.
\n",
"
x = tok + pos: meaning plus position, added together (broadcasting fills the batch axis).
\n",
"
@torch.no_grad() above generate: a decorator meaning \"don't record gradients\": generation is pure inference.
\n",
"
Inside generate: crop to the last block_size tokens, keep only the last position's logits, divide by temperature, optionally keep just the top-k (torch.topk finds them; everything below gets -inf), softmax, then torch.multinomial(probs, 1) rolls the weighted die and torch.cat appends it.
\n",
"
num_params(): add up p.numel() (element count) over every weight grid: the parameter total.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "d22ac31b",
"metadata": {},
"source": [
"## Instantiate it\n",
"\n",
"Build a real, if small, GPT sized for our character data, and count its numbers. It comes out at around **825,000**, which sounds like a lot until you learn that GPT-3 has 175 billion, roughly **200,000 times more**. The striking part is that the design you are looking at is, line for line, what the giant models use. They are the same thing, only far wider, far deeper, and trained on far more text."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "e3fe17ab",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:31.964851Z",
"iopub.status.busy": "2026-07-01T09:18:31.964800Z",
"iopub.status.idle": "2026-07-01T09:18:32.003902Z",
"shell.execute_reply": "2026-07-01T09:18:32.003529Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"vocab=65 block=128 layers=4 heads=4 width=128\n",
"total parameters: 824,897\n"
]
}
],
"source": [
"from pathlib import Path\n",
"text = Path(\"data/input.txt\").read_text()\n",
"vocab_size = len(sorted(set(text)))\n",
"\n",
"cfg = GPTConfig(vocab_size=vocab_size, block_size=128, n_layer=4, n_head=4, n_embd=128)\n",
"model = GPT(cfg).to(device)\n",
"print(f\"vocab={cfg.vocab_size} block={cfg.block_size} layers={cfg.n_layer} heads={cfg.n_head} width={cfg.n_embd}\")\n",
"print(f\"total parameters: {model.num_params():,}\")"
]
},
{
"cell_type": "markdown",
"id": "a7ea997a",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
cfg = GPTConfig(vocab_size=vocab_size, block_size=128, n_layer=4, n_head=4, n_embd=128): fill in the size dials.
\n",
"
model = GPT(cfg).to(device): build the model and ship every weight to the GPU.
\n",
"
f\"{model.num_params():,}\": the :, format inserts thousands separators: 824,897.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "m8c1a",
"metadata": {},
"source": [
"### Where the 824,897 parameters live\n",
"\n",
"That single number is easier to trust once you see it split up. Group every parameter tensor by the part of the model it belongs to, and one thing stands out: the stacked Transformer blocks hold about 96% of the weights, while the embeddings and the output head are a thin shell around them."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "m8c1b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:32.004865Z",
"iopub.status.busy": "2026-07-01T09:18:32.004814Z",
"iopub.status.idle": "2026-07-01T09:18:32.182075Z",
"shell.execute_reply": "2026-07-01T09:18:32.181733Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" token embedding 8,320 ( 1.0%)\n",
" position embedding 16,384 ( 2.0%)\n",
" 4 transformer blocks 791,552 (96.0%)\n",
" final LayerNorm 256 ( 0.0%)\n",
" output head 8,385 ( 1.0%)\n",
" TOTAL 824,897\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAEiCAYAAABp8WNbAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAASNRJREFUeJzt3QW4FGXYN/D70N0pXdLS0lhIiwgoICVgId0pKCJIKCEi+QKCKCANEiog0t2hAkqHdOd81/9+32e+OXv2JAu758z/d117zu7s7Oyzk/c8GWRZliVERERE5Bqx/J0AIiIiInq6GAASERERuQwDQCIiIiKXYQBIRERE5DIMAImIiIhchgEgERERkcswACQiIiJyGQaARERERC7DAJCIiIjIZRgAkqtkz55datWqJdHZO++8I0mSJPF3MshPXnzxRX0Y//zzjwQFBcnUqVP9kp4tW7ZIvHjx5N9//430cbZmzRpNO/4/CZ988oku/7///hN/biO3fHdklClTRrp37+7vZLgaA0CiAHTr1i29eD2pCyOFb8OGDboNrly5wtUVhj59+kijRo0kW7ZsXE8u8fPPP+ux8Th69Ogh33zzjZw9e9Zn6aLIYQBIFKAB4KeffsoA0M8BILZBoAeACLxu374tTZs2ferfvWvXLvn111/lww8/fOrfTd6tXLlSH086AMSx8Thef/11SZYsmYwdO9Zn6aLIYQBI5GMPHjyQe/fucb1GwKNHj+TOnTuuC+59CUWcCRIkkNixY8vTNmXKFMmaNasW51FgQHE8HoEuVqxYUr9+ffnuu+/Esix/J8eVGABStLNnzx696C1atMietn37dp1WvHjxYPNWr15dSpcuHWIZ69atk+eff14vnDlz5tSTkCfk/HTs2FGyZMki8ePHl9y5c8uQIUM0aPGsfzV8+HAZOXKk5MqVS+c9cOCAvn/o0CE9yaVKlUq/q2TJksHS7Q2WmTZtWn2Ou2wsHw/PIpdTp05JnTp1tD4g5u/atas8fPgw2DxIK9JVsGBB/f706dPLBx98IJcvX45wXcOjR49K1apVJXHixPLMM8/IgAEDQpyw8fvLlSsnqVOnloQJE0qJEiXkp59+CrFM/I62bdvK999/r2nCulq+fHmUljFnzhwpUKCAzlu2bFnZu3evvj9+/HjdVvi9qAuF9elp8+bNUq1aNUmePLkkSpRIXnjhBVm/fr39PtZ1t27d9HmOHDnsbeBc1owZMzSN+H5s34YNG8qJEyeCfQ++v1ChQrp/VqpUSb+rd+/e+t62bdt0vaZJk0aXge9p2bKlRJZnHUCsR7x21skzevXqpcGBc/uHty7CsmDBAnn55Zf1+7xBTlTRokV1W2BbzZs3L0LLxbY16xbrp0mTJrq/e8Lx9dZbb+n+j3nz5s2rRdJhwXrB/oHtcu7cOZ32119/Sb169SRDhgya1syZM+v2vHr1arhpnTBhgh73+H6cU/744w+v850/f15atWqlxyC+o0iRIjJt2rRg8zjPJygexbkJ26RKlSq6b+G4++yzzzR9+D7kol26dCnMOoCmnuXs2bPl888/18/i+1955RX5+++/g30WaX/zzTc1qMexiXNfp06dNIfZeV5A2sAcF87tH5lzzquvvqrbAznJ5AcWUTTz8OFDK0WKFFaXLl3saSNGjLBixYqlj6tXr9rzJUuWzOratas9X7Zs2ay8efNa6dOnt3r37m2NGTPGKl68uBUUFGTt27fPnu/mzZvWc889Z6VOnVrnGzdunNWsWTOdr0OHDvZ8x44dQyRkFShQwMqZM6f1xRdfaFr+/fdfXV7y5Mn1vSFDhuh3VapUSZcxb968UH/fjRs3rG+//VaX+8Ybb1jTp0/Xx+7du/X95s2bWwkSJLAKFixotWzZUuetV6+ezj927Nhgy3r33XetOHHiWO+9957+hh49eliJEye2SpUqZd27dy/M9Wy+J0+ePFbTpk01/bVq1dLv+fjjj4PNmzlzZuujjz7Seb766ivr+eef1/mWLFkSbD5My58/v5U2bVrr008/tb755htr586dkV4Gtk2WLFl0feOB9Zw1a1b9LNb3l19+afXt29eKFy+e9dJLLwX7/G+//abTy5Ytq/Nhe2F5mLZ582adB+u6UaNG+l1432wDbBsYOHCgbscGDRroOsdvSZMmjZU9e3br8uXL9ne98MILVoYMGfT3tmvXzho/fry1YMEC69y5c1bKlCmtZ5991ho2bJg1ceJEq0+fPrpuwoNl4uG5D06ZMkVfY99D2oYOHRris9hHa9asGal1EZqTJ0/q944ePTrEezjO8NtwnPbs2VO3Z+HChfX4XLlypT3f6tWrdRn4b+B3YBr2UaQHn0+YMGGIdYtthOMbx2ivXr103Xbv3l2/x+jfv78u68KFC/r677//1v2kaNGi9rS7d+9aOXLksJ555hndrpMmTdLtie//559/wlwHmBfLL1eunK6Hjh076m/GenZuo1u3bum2jRs3rtWpUyedt2LFivrZkSNHhtiWSB/2Y6w3sx+XKVNGz0Xmu9q3b6/buUWLFmHuH2YdFytWzCpRooSu008++cRKlCiRHmNO2Edr1KhhDRo0SNdnq1atrNixY1v169e359mwYYP16quv6jLNcYFHVM45Zh/6+uuvw1zP9GQwAKRoCRcx58mrbt26+sDJatmyZTptx44denJZuHBhsAsTpq1du9aedv78eSt+/PjBAsrPPvtMT1p//vlnsO/FxQjfcfz48WAnbFyIsBynV155RS9Gd+7csac9evRIT+AIqsKCixOWiwuYt8AM7w0YMCDYdHOCN/744w+d7/vvvw823/Lly71OD+17cFFwph/rHhckcwE1FzgnnOgLFSpkvfzyy8GmY3kIAvbv3x/i+yKzDGwvrHsDFytMR7B17do1ezoCA0w38yL9WPdVq1bV587vRhCAC5uBwMz5WQNBAfaBzz//PNj0vXv36oXPOR0XYiwDF0Kn+fPn6/StW7dakRVeAAgI6Jz7AmzZskXn++677yK9Lrz59ddfdXmLFy8O8Z45zubOnWtPw41ZxowZdT8NLQDENk+XLp1u99u3b9vz4SYA8/Xr18+ehpuppEmTasDr5PwtzgDw4MGDGuQhELl06ZI9D25AMM+cOXOsyDBpRbCGINKYMGGCLs+5jRDkYdqMGTOCfR7bKUmSJPY+a7YlbhiuXLkSYj8uUqSIdf/+fXs6blJwLDrPMaEFgAhAnekcNWqUTsd+G9oxCIMHD9ZA07me27Rpo5/1FJVzDtLfunXrENPpyWMRMEVLFStWlB07dsjNmzftIt0aNWpocZMpgsF/FE1UqFAh2GdRFIXPGyg+QtERijqdRVCYJ2XKlNqFhHlUrlxZi1nXrl0bbJkoPjLFtoBimVWrVmnx1PXr1+3PX7x4UYv9UOTkrUgrMjwr3iO9nr8BxXooZnH+BhStoWh39erVEfoeFLd6Fr+ijiMq/xsojjJQ1IOiM7ONPKGIEdvAU2SWgeIrdDVimGJ+bIekSZOGmG7WC4qasO7ffvtt3RZmnWA/wjKxXZ1F/N6gGBPzYNs61yuKD/PkyRNivaIorUWLFsGmpUiRQv8vWbJE7t+/L77WoEEDLXY+cuSIPW3WrFmaFhQb+mJd4DOAY8QbVBd444037Neo8N+sWTPZuXNnqC0/USyOotKPPvpIiw+NmjVrSr58+WTp0qX6+sKFC5o+FJmjuNLJW3H0vn37dL/DPoP91plmHCOwYsWKSNXPNGnFceisc4ciUrNMZ6MJ7B9oLW3EjRtX2rdvLzdu3JDff/892PwohnUuw+zHKAqPEydOsOk4FiNyLsE+6EynOQc6zxnOYxD7AfYHVMvAfRe2W3iics4x51h6+v7/nkQUjeDkhcYWGzdu1HoqOBFj2v79+4MFgAg0UD/LyfOCYU5CzjoquDCirqEzqHPC9zmh/pYT6tbgpPnxxx/rI7RlZMqUSaICF0fPtHn7DQii0qVLF6HfEFpFbdRDcnr22Wf1v7M+HAKZgQMHalBx9+7dMC/GnusqKsvw3IbmYol9wdt0s16wTqB58+ah/mass9CCGrMMbFsEe97gwu6EbexZKR/BCIJV1PEcMWKE1tlCfU4EYwjSHhcCiM6dO2vQhzqHSC8uzqgTi0DM/I7HXRcQWgV+1LPz3HbOfQcBkSdTbxE3ZJ4QAOJGzxm0oB5fRLz22mtaFw1Bnmcfmtgfsa6++uorrZuK80jt2rU12PIM5Lyl1XM/wPb3PGYwL+bD8eSUP3/+YMt63P07LJ7LNNvV+dnjx49Lv379tJ6y5zIjUh8yKucc7D+h1SGlJ4sBIEVLaEyBIAi5ADix4YSDiwtO3uhWAAEEAkBnDoQRWmtJ54UMOR+4iw2to1JzIfN252w+D2iYgRy/0C6QURWRFp9IA9YLLmrehBbcRhbWMy6YaOSAdZ8xY0a9CKKF6MyZM0PM77muorKM0H5/eNvWbJdhw4ZpbrE34XWyjWXggrVs2TKv3+f5eW+/F59HA5dNmzbJ4sWLNTBBbtaXX36p0x63o2/kvuFYQMV/BIBYJi7uaMTk/B2Psy7QWCeiwYe/IdhGgwscC2iQ4AnrHTl3Cxcu1IYryJkbPHiwrjc0mnjaorp/R2WZ5rMo2cA5D6UX6KMPATcafiF3EesmvJzxqJ5z0NgODX3o6WMASNESclRMizsEgKY4A/8R/OEEhBZ+CCiiAq36UDSDIt+oMDkACGKisgxf3BHjN6C4q3z58l6DkIjACR25Lc6A988//9T/pgh27ty5GowjiHHmXiF4iyhfLCOi6wSQCxbedgltG2AZuGgi58jzRiCy0H0KHmidiUC3cePG8uOPP8q7774rvigGRlHq4cOHNScQrUmRE+b8HRFdF94gQIBjx455fd/kgjvXo+e+48l0Jo00o3WxE6aZ983xhaLdiECQi6JTrA9UEUBOq6fChQvro2/fvtoHJI6bcePGaa50WGlFrpczrSjSxzpBK1/nvChRwPHkzAVEK2bnsvwJreixfRAoo6je+OWXXyJ1bETmnIPgEkXYJieUni7WAaRoC8EeurBAvRITAOJOEicTk9PhrOsXGajfheJlBCTe7lhR/BwW3AWjWA9dkpw5cybE+6jDFBZcrM13RRV+A+7q0W2EJ6Q/osseM2aM/RwXdLxGYIt6YiZnARcEZxc0KOJDFyER5YtlRATqIuEihW42EOCHtV2Q+wGe66lu3bqaXhTfeua84LWpGxcW5Jp5ftbkwjmLvx831wvp/OGHH7T4F0Ozmd8U2XXhDYq2USSJunDenD59WubPn2+/vnbtmna3hN/prfjX5Ozj2EHg5VwPyG09ePCg1gU0OUm4ufuf//kfzdkMLzcM+xa6a0GXTCjydnbFhHR5Hs8IBBGohbUtkFakA2l19vuJ7ng89xnUT0a9RwTiBr7z66+/1lxWVAnwN5ND6Fx/eD5q1KgQ84Z2bET2nIN6qoB6hvT0MQeQoi0Ed8g5Qf9YzkAPFwYEXshliGrxDfqAw0UCF00Uf+BiiUrRuEtG0R2Ck/CKLdBXFhqg4GLy3nvvaa4FciURWJ48eVJ2794d6mdx94z6i7hgIJcJ9RhR3ymidZ4AFxUUd6EoC/Xq0JcYAjfkWCAgwIkdF8SwIFcO/fThookK57gQoyI+ihVNcQ4uyqg/hb7kkLOCej747SjiRq5HRPhiGRGBi/qkSZO0Lhz6KUPFeAQyyInAjQRyw1AkC9jmgH7l0Ccc1h1y0BA0IVcIfephP0DdPeQqIdcHAc/777+vRf9hQS4LirpRRQHLQ0OhiRMn6vcjWPAFBFIvvfSSrlcsHzmCUV0XoUGDEvxmb/W4sN+i37utW7dq/TsEa9j/w8rVxTrGzRvSgv0XjSbwGeyrOJ7RJ50xevRoPb7Q9yfWOXJksT2wf3rrVw6/F303YnshUEHDDOTcobEWGjah3iTSjEBl+vTpGhAhiA4rrdgPcIxhOVi/2Afw+zzrACJ9OCfhXIKgB78F5xH0t4g+85wNl/wFObrYF7HvYh/A9kfOvLcifnNsoKgcVVywrnCMRPacg9xFlOAUK1bsqf5W+j9PoaUx0ROBrhPQHQe6gnjw4IE9HV0tYNdG33Xeuqdw9oMWWtcJcP36de1+IXfu3NpVAfp5Qxcuw4cPt/uzMt02oMsQb44cOaL9B6J7EvQBlilTJu1L76effgr396G/LXTlge92dgmD7lnQRY0n0+WFJ3RLgeWgLzWsK3RNg/7STp8+Heb3m+/Bb6hSpYr2G4b+E/E96GPRafLkydqlCLpnyZcvn3ZJ4i09eI0uJLx5nGWEth1MFxieXXyg6w90G4Q+5PB92C/eeust7RfPCd0BYZuh6xrPLmHQxUmFChV0HeGBNCNdhw8ftufBPoX+Gj2hiyJ04YE+6fD96E4E+8W2bdssX3QDY6B/QbyH7e7sViUq68Ib09USuv/wdpytWLFC+xU029RzO3jrBxBmzZql3cXgc6lSpbIaN26sfcZ5Ql+b6CsTfe+hz0r08enso9KzH0DT1QnWH7pf2bRpk3X06FHtTzNXrly6DHwf+o5ENzcRgX4g0W0O0lqyZEntYsrb+QR9P6LPPpxHcEzjOPTcZpHdj02fic7uhELrBsbzs972mwMHDliVK1fWdYN0oi8/9LfoOR/Ot+geCt3VoIsYz2M0IuccnEPQLRD6OST/CMIfEwwSERnIrUAuhbfiQSIDVQHQ6AS5ZkQRheodyO1HV0Vo9EVPH+sAEhFRlA0aNEirKngbeo4oNCjqR9E7gz//YR1AIiKKMtMZMVFkoC40+RdzAImIiIhchnUAiYiIiFyGOYBERERELsMAkIiIiMhl2AjkKcIwQOgdH51+cvBrIiIi8iX07IeO39E1k3PYQW8YAD5FCP4wdBIRERHRk4IRssIbCYsB4FNkhvvBhsEwO0QUM2HoKwxn5nw9efJkHeO5X79+cv/+fR0+DENmmaHhMCTY0aNHdRgujA/r6Y8//pDGjRvr0FmAYdLatGmjzzFUIIYONEN6YYi3W7duSdOmTbV/PgzV1bJlSx2mi4hiLoxtjYymiAwvyFbAT3nDJE+eXK5evcoAkMglMJIKxn5FIIYg7ffff9fXGB8a489ivFSMMY2gD2P37tu3z2sAuGbNGhkzZoyOzuIJ41L/999/waYhAMQ4vBifFWkoWbKkLFmyRMdXJqKYKTJxBhuBEBE9QYsWLdLh0m7evKmBHYI/QA7gvHnz9HnhwoUlR44cPv1e5DYi+AN8b968eeXMmTM+/Q4iir78GgA+fPhQPv74Yz3xJUyYUHLlyiWfffaZVmI08BxFJhguBvNUrlxZi1OcLl26pEUjiHZTpEghrVq1CjF+6Z49e6RixYqSIEECzR4dOnRoiPTMmTNHi08wD07IP//8c7D3I5IWIiKn2bNnS4MGDSRt2rQaBCK3D+c+BIYo7o0M5AIWKVJEateuLX///Xewu/4SJUpI+fLlZeXKlSE+h2onOAcWL16cG4eI/pflR59//rmVOnVqa8mSJdaxY8esOXPmWEmSJLFGjRplz/PFF19YyZMntxYsWGDt3r3bql27tpUjRw7r9u3b9jzVqlWzihQpYm3atMn6448/rNy5c1uNGjWy37969aqVPn16q3Hjxta+ffusH374wUqYMKE1fvx4e57169dbsWPHtoYOHWodOHDA6tu3rxU3blxr7969kUpLWJAOrHL8J6KYD8d62rRp7XMEzk/lypWznn/+eatLly7W66+/Hmz+bNmyWdevXw91Wea9efPmWSVLlrTfO3nypP4/fPiwLuPcuXP2e3fu3LHKli2r51ciitmuRiLO8GsAWLNmTatly5bBptWtW1cDNXj06JGVIUMGa9iwYfb7V65cseLHj69BHCBYw4/dunWrPc+yZcusoKAg69SpU/p67NixVsqUKa27d+/a8/To0cPKmzev/fqtt97S9DiVLl3a+uCDDyKclvAwACRyl++++84+n3maPn261bVr1wgHgJ7SpUtnPXjwIMT0+vXrW6tXr7bPWzi3ffbZZ1FKPxFFL5GJM/xaBFyuXDn57bff5M8//9TXu3fvlnXr1kn16tX19bFjx+Ts2bNa1GqgciMGHzcDSeM/in1RwdnA/Oj/ZvPmzfY8lSpVknjx4tnzVK1aVQ4fPiyXL1+253F+j5nHfE9E0uLp7t27WjTjfBCR+4p/jfPnz+t/VFH5+uuvtbpKWObPny+9evXS5+fOnbOn4zyJImW07sU5DOcaM8/27dslT548+hqfRV3Avn37PpHfR0TRl18DwJ49e0rDhg213l3cuHGlWLFi0rFjR63PBwi4IH369ME+h9fmPfxPly5dsPfjxImj3SI45/G2DOd3hDaP8/3w0uIJXTwgSDQP9gFI5B5ohbdlyxa9kXSeE/Lnzy+lSpWStm3b6rkPxo8fr312nTx5UhtrdO7cWaejWxjTkg/BZMGCBaVo0aJ67pw+fbpOP3jwoN4Ao24gWhQPGjRIMmXKpMsaMmSIpgGfwWPFihV+WRdEFHj82g8gTmjff/+9zJw5U09su3bt0gAQPVg3b95cojvcfZsTubN/HiKK+XDT58y1gxEjRujDE/rn89ZHHxpuDB8+XJ+3a9dOH95KUtCwxBMCSmeDOiKigAkAu3XrZucCAlreoq8s3CUjAMyQIYNOx0kULW8NvMbdLGAeU6xiPHjwQFsGm8/jv+eJ2LwObx7n++GlxVP8+PH1QUQUFeggmogoxgWA6KjUc6w61GnBmLmA7mEQeKGeoAmykIuGun2tW7fW12XLlpUrV65ovRd0gwCrVq3SZaB+npmnT58+2vs+iprhl19+0aKWlClT2vPge5ADaWAeTI9oWogoemnS5KC/kxAjzJiR399JIKLoVAfwtddek88//1yWLl0q//zzj1Z4/uqrr+SNN97Q94OCgjQgGzhwoPaZhWKOZs2aaRFxnTp1dB7Up6lWrZr2qo+6LuvXr9e6NchVxHzw9ttvawMQVLjev3+/zJo1S0aNGhWseLZDhw6yfPly+fLLL+XQoUPyySefyLZt23RZEU0LERERUXTg1xxAtIJDR9AfffSRFuMimEI9GHS2bHTv3l07T33//fc1p69ChQoaqKGzZgP1CBGoobd95CjWq1dPRo8eHawuDjpHxbiZyCXEsEn4DizTWY8GdRHRWq53797aim7BggVSqFChSKWFiIiIKNBxLOCniGMBEwUWFgH7BouAiQIDxwImIiIiosCsA0hERERETx8DQCIiIiKXYQBIRERE5DIMAImIiIhchgEgERERkcswACQiIiJyGQaARERERC7DAJCIiIjIZRgAEhEREbkMA0AiIiIil2EASEREROQyDACJiIiIXIYBIBEREZHLMAAkIiIichkGgEREREQuwwCQiIiIyGUYABIRERG5DANAIiIiIpdhAEhERETkMgwAiYiIiFyGASARERGRyzAAJCIiInIZBoBERERELsMAkIiIiMhlGAASERERuQwDQCIiIiKXYQBIRERE5DIMAImIiIhchgEgERERkcswACQiIiJyGQaARERERC7DAJCIiIjIZRgAEhEREbkMA0AiIiIil2EASEREROQyDACJiIiIXIYBIBEREZHLMAAkIiIichkGgEREREQuwwCQiIiIyGX8HgCeOnVKmjRpIqlTp5aECRNK4cKFZdu2bfb7lmVJv379JGPGjPp+5cqV5a+//gq2jEuXLknjxo0lWbJkkiJFCmnVqpXcuHEj2Dx79uyRihUrSoIECSRLliwydOjQEGmZM2eO5MuXT+dBOn7++edg70ckLURERESBzq8B4OXLl6V8+fISN25cWbZsmRw4cEC+/PJLSZkypT0PArXRo0fLuHHjZPPmzZI4cWKpWrWq3Llzx54Hwd/+/fvll19+kSVLlsjatWvl/ffft9+/du2aVKlSRbJlyybbt2+XYcOGySeffCITJkyw59mwYYM0atRIg8edO3dKnTp19LFv375IpYWIiIgo0AVZyNbyk549e8r69evljz/+8Po+kvbMM89Ily5dpGvXrjrt6tWrkj59epk6dao0bNhQDh48KAUKFJCtW7dKyZIldZ7ly5dLjRo15OTJk/r5b7/9Vvr06SNnz56VePHi2d+9YMECOXTokL5u0KCB3Lx5UwNIo0yZMlK0aFEN+CKSlvAgEE2ePLl+DrmVRORfTZoc5CbwgRkz8nM9EgWAyMQZfs0BXLRokQZtb775pqRLl06KFSsmEydOtN8/duyYBm0oajXww0qXLi0bN27U1/iPYl8T/AHmjxUrlubSmXkqVapkB3+AnLvDhw9rLqSZx/k9Zh7zPRFJi6e7d+/qxnA+iIiIiPzNrwHg0aNHNXcuT548smLFCmndurW0b99epk2bpu8j4ALksjnhtXkP/xE8OsWJE0dSpUoVbB5vy3B+R2jzON8PLy2eBg8erEGieaDuIREREZGrA8BHjx5J8eLFZdCgQZr7h3p77733nha5xgS9evXSbFjzOHHihL+TREREROTfABCtaVF/zyl//vxy/PhxfZ4hQwb9f+7cuWDz4LV5D//Pnz8f7P0HDx5oy2DnPN6W4fyO0OZxvh9eWjzFjx9fy+CdDyIiIiJXB4BoAYx6eE5//vmnttaFHDlyaHD122+/2e+jHh3q9pUtW1Zf4/+VK1e0da+xatUqzV1E/TwzD1oG379/354HLYbz5s1rtzjGPM7vMfOY74lIWoiIiIiiA78GgJ06dZJNmzZpEfDff/8tM2fO1K5Z2rRpo+8HBQVJx44dZeDAgdpgZO/evdKsWTNtjYsuWkyOYbVq1bToeMuWLdqquG3bttoqF/PB22+/rQ1A0MULuouZNWuWjBo1Sjp37mynpUOHDtp6GN3QoGUwuolBf4RYVkTTQkRERBQdxPHnl5cqVUrmz5+vdeUGDBiguWwjR47Ufv2M7t27a/csqB+InL4KFSpooIbOmo3vv/9eA7VXXnlFW//Wq1dP++sz0ABj5cqVGliWKFFC0qRJox06O/sKLFeunAagffv2ld69e2vDFHQTU6hQoUilhYiIiCjQ+bUfQLdhP4BEgYX9APoG+wEkCgzRph9AIiIiInr6GAASERERuQwDQCIiIiKXYQBIRERE5DI+CwDRKpaIiIiIYmgAOGTIEO1Lz3jrrbckderUkilTJtm9e7cv00dEREREgRAAYqzeLFmy2KNl4LFs2TKpXr26dOvWzddpJCIiIiJ/dwR99uxZOwBcsmSJ5gBWqVJFsmfPbg+/RkREREQxKAcQ4+eeOHFCn2MkjMqVK+tz9Cn98OFD36aQiIiIiPyfA1i3bl0dXxfDpV28eFGLfmHnzp2SO3du36aQiIiIiPwfAI4YMULH7T1+/LgMHTpUkiRJotPPnDkjH330kW9TSERERET+DQDv378vH3zwgXz88ccaBDp16tTJl2kjIiIiokCoAxg3blyZO3fuk0gLEREREQVqI5A6derIggULfJ8aIiIiIgrMOoBo/DFgwABZv369lChRQhInThzs/fbt2/sqfUREREQUCAHg5MmTJUWKFLJ9+3Z9OAUFBTEAJCIiIoppAeCxY8d8nxIiIiIiCtw6gMa9e/fk8OHD8uDBA9+liIiIiIgCLwC8deuWtGrVShIlSiQFCxbU/gChXbt28sUXX/g6jURERETk7wCwV69esnv3blmzZo0kSJDAno4h4WbNmuXL9BERERFRINQBRBcwCPTKlCmjjT4M5AYeOXLEl+kjIiIiokDIAbxw4YKkS5cuxPSbN28GCwiJiIiIKIYEgCVLlpSlS5far03QN2nSJClbtqzvUkdEREREgVEEPGjQIKlevbocOHBAWwCPGjVKn2/YsEF+//1336eSiIiIiPybA1ihQgXZtWuXBn+FCxeWlStXapHwxo0bdWQQIiIiIophOYCQK1cumThxom9TQ0RERESBmQMYO3ZsOX/+fIjpFy9e1PeIiIiIKIYFgJZleZ1+9+5diRcv3uOmiYiIiIgCpQh49OjRdqtftPhNkiSJ/d7Dhw9l7dq1ki9fPt+nkoiIiIj8EwCOGDHCzgEcN25csOJe5Pxlz55dpxMRERFRDAkAjx07pv9feuklmTdvnqRMmfJJpYuIiIiIAqkO4OrVqzX4u3fvnhw+fFi7gyEiIiKiGBwA3r59W1q1aiWJEiXS8X+PHz+u09u1aydffPGFr9NIRERERP4OAHv27Cm7d++WNWvWSIIECezplStXllmzZvkyfUREREQUCB1BL1iwQAO9MmXK2OMAA3IDjxw54sv0EREREVEg5ABeuHBBh37zdPPmzWABIRERERHFkACwZMmSsnTpUvu1CfrQN2DZsmV9lzoiIiIiCowi4EGDBkn16tXlwIED2gJ41KhR+nzDhg3y+++/+z6VREREROTfHMAKFSrIrl27NPgrXLiwrFy5UouEN27cKCVKlPBd6oiIiIgoMHIAIVeuXDJx4kTfpoaIiIiIAjcAhPPnz+vj0aNHwaY/99xzj5suIiIiIgqkIuDt27dLoUKFJGPGjBrsFS1a1H4UK1YsSglBB9JoTNKxY0d72p07d6RNmzaSOnVqSZIkidSrV0/OnTsX7HPohLpmzZraKTWKobt16xZiZBL0V1i8eHGJHz++5M6dW6ZOnRri+7/55hsdyxj9GpYuXVq2bNkS7P2IpIWIiIgoxgaALVu2lGeffVYbfRw9elTHCDYPvI6srVu3yvjx40PkHHbq1EkWL14sc+bM0cYlp0+flrp169rvP3z4UIM/DEmHtEybNk2Du379+tnzIE2YB+MXo94iAsx3331XVqxYYc+DPg07d+4s/fv3lx07dkiRIkWkatWqmrsZ0bQQERERRRdBlmVZkf1Q0qRJZefOnZqb9rhu3LihuXNjx46VgQMHai7iyJEj5erVq5I2bVqZOXOm1K9fX+c9dOiQ5M+fXxuboBPqZcuWSa1atTQYS58+vc4zbtw46dGjh/ZVGC9ePH2OLmv27dtnf2fDhg3lypUrsnz5cn2NHL9SpUrJmDFj9DWKtLNkyaJD22HUk4ikJSKuXbsmyZMn1+UlS5bssdcdET2eJk0OchX6wIwZ+bkeiQJAZOKMKOUAvvLKKzoUnC+gWBU5dBhGzrOY+f79+8Gm58uXT7JmzapBF+A/WiGb4A+Qc4cVsH//fnsez2VjHrMM5B7iu5zzxIoVS1+beSKSFiIiIqIY3QgEHT43b95cc9VQFzBu3LjB3q9du3aElvPjjz9qkSuKgD2dPXtWc/BSpEgRbDqCPbxn5nEGf+Z9815Y8yBIvH37tly+fFmLkr3Ng1y+iKbFm7t37+rDwHcSERERRcsAELle69ev1yJYT2jIgYAqPCdOnJAOHTrIL7/8og0vYqLBgwfLp59+6u9kEBERET1+ETDqxjVp0kTOnDmj9eWcj4gEf6ZYFY0sUP8vTpw4+kDjitGjR+tz5K6heBZ19ZzQ8jZDhgz6HP89W+Ka1+HNg7LxhAkTSpo0aSR27Nhe53EuI7y0eNOrVy8thzcPBL1ERERE0TIAvHjxoraK9Sw2jWw9wr1792rLXPPAGMONGze2n6No+bfffrM/c/jwYe32xYw3jP9YhrO1LnIUEdwVKFDAnse5DDOPWQaKdjF6iXMeBLJ4bebB++GlxRt0O4O0OB9ERERE0bIIGN2frF69WkcDiSq0JEb9QafEiRNrP3tmeqtWrbR7llSpUmnwhJxHBFym1W2VKlU00GvatKkMHTpU6+P17dtXG5Yg+IIPP/xQW/d2795du69ZtWqVzJ49W1sGG/gO1GlE0Pn8889rK+SbN29KixYt9H20qAkvLUREREQxOgBEH4Ao3ly3bp22wvVsBNK+fXufJG7EiBHaIhedLqMxBVrvorsYA0W3S5YskdatW2swhgASgdyAAQPseXLkyKHBHnIsR40aJZkzZ9ZGLFiW0aBBA+02Bv0HIohEVzToIsaZwxleWoiIiIhidD+ACKpCXWBQUJQ6g3YD9gNIFFjYD6BvsB9AougXZ0QpBxCjaxARERGRixqBEBEREVH0FaUcQDh58qQsWrRIW8KiixSnr776yhdpIyIiIqJACQDRHQpG+8iZM6eOloFWu//884+gOiH69SMiIiKiGFYEjBbAXbt21T74MIrH3LlztZPjF154Qd58803fp5KIiIiI/BsAHjx4UJo1a6bPMWoHxtRNkiSJdr8yZMgQ36WOiIiIiAIjAER/e6beX8aMGeXIkSP2e//995/vUkdEREREgVEHEKNfoBPo/PnzS40aNaRLly5aHDxv3jyOjEFEREQUEwNAtPK9ceOGPv/000/1+axZsyRPnjxsAUxEREQU0wLAhw8fahcwzz33nF0cPG7cuCeRNiIiIiIKhDqAGH+3SpUqcvny5SeRHiIiIiIKxEYg6PeP4/0SERERuSgAHDhwoPYDuGTJEjlz5owOPux8EBEREVEMawSClr+A0UCCgoLs6RgJBK9RT5CIiIiIYlAAuHr1at+nhIiIiIgCNwDEkG9ERERE5KIA0Lh165YcP37cHhXEMF3EEBEREVEMCQAvXLggLVq0kGXLlnl9n3UAiYiIiGJYK+COHTvKlStXZPPmzZIwYUJZvny5TJs2TUcCWbRoke9TSURERET+zQFctWqVLFy4UEqWLCmxYsWSbNmyyauvvirJkiWTwYMHS82aNX2XQiIiIiLyfw7gzZs3JV26dPo8ZcqUWiQMhQsXlh07dvg2hURERETk/wAwb968cvjwYX1epEgRGT9+vJw6dUrHBM6YMaNvU0hERERE/i8C7tChg44AAv3795dq1arJjBkzJF68eFoXkIiIiIhiWADYpEkT+3nx4sXl33//lUOHDknWrFklTZo0vkwfEREREQVCETBMnjxZChUqJAkSJNB6gM2aNZMFCxb4NnVEREREFBg5gP369ZOvvvpK2rVrJ2XLltVpGzdulE6dOmnH0AMGDPB1OomIiIjInwHgt99+KxMnTpRGjRrZ02rXrq0jgCAoZABIREREFMOKgO/fv699AHoqUaKEPHjwwBfpIiIiIqJACgCbNm2quYCeJkyYII0bN/ZFuoiIiIgokIqATSOQlStXSpkyZfQ1hoVD/T80BuncubM9H+oKEhEREVE0DwD37dun3b/AkSNH9D+6f8ED7xlBQUG+SicRERER+TMAXL16ta++n4iIiIiiSz+ARERERBQ9MQAkIiIichkGgEREREQuwwCQiIiIyGUYABIRERG5DANAIiIiIpdhAEhERETkMgwAiYiIiFyGASARERGRy/g1ABw8eLCUKlVKkiZNKunSpZM6derI4cOHg81z584dadOmjaROnVqSJEki9erVk3PnzgWbB2MQ16xZUxIlSqTL6datmzx48CDYPGvWrNHh6+LHjy+5c+eWqVOnhkjPN998I9mzZ5cECRJI6dKlZcuWLZFOCxEREVGg82sA+Pvvv2tAtWnTJvnll1/k/v37UqVKFbl586Y9T6dOnWTx4sUyZ84cnf/06dNSt25d+/2HDx9q8Hfv3j3ZsGGDTJs2TYO7fv362fMcO3ZM53nppZdk165d0rFjR3n33XdlxYoV9jyzZs2Szp07S//+/WXHjh1SpEgRqVq1qpw/fz7CaSEiIiKKDoIsy7IkQFy4cEFz8BBcVapUSa5evSpp06aVmTNnSv369XWeQ4cOSf78+WXjxo1SpkwZWbZsmdSqVUuDsfTp0+s848aNkx49eujy4sWLp8+XLl0q+/bts7+rYcOGcuXKFVm+fLm+Ro4fciPHjBmjrx89eiRZsmSRdu3aSc+ePSOUlvBcu3ZNkidPrstKlizZE1mHRBRxTZoc5OrygRkz8nM9EgWAyMQZAVUHEAmGVKlS6f/t27drrmDlypXtefLlyydZs2bVoAvwv3DhwnbwB8i5w0rYv3+/PY9zGWYeswzkHuK7nPPEihVLX5t5IpIWIiIiouggjgQI5LihaLZ8+fJSqFAhnXb27FnNwUuRIkWweRHs4T0zjzP4M++b98KaB0Hi7du35fLly1qU7G0e5PJFNC2e7t69qw8D30dERETkbwGTA4i6gCii/fHHHyWmQCMXZMWaB4qUiYiIiPwtIALAtm3bypIlS2T16tWSOXNme3qGDBm0eBZ19ZzQ8hbvmXk8W+Ka1+HNg/LxhAkTSpo0aSR27Nhe53EuI7y0eOrVq5cWa5vHiRMnIr1uiIiIiGJUAIj2Jwj+5s+fL6tWrZIcOXIEe79EiRISN25c+e233+xp6CYG3b6ULVtWX+P/3r17g7XWRYtiBHcFChSw53Euw8xjloGiXXyXcx4USeO1mSciafGELmeQDueDiIiIyNV1AFHsi1a1Cxcu1L4ATV06FJciZw7/W7Vqpd2zoGEIAii0ykXAZVrdotsYBHpNmzaVoUOH6jL69u2ry0YABh9++KG27u3evbu0bNlSg83Zs2dry2AD39G8eXMpWbKkPP/88zJy5EjtjqZFixZ2msJLCxEREVF04NcA8Ntvv9X/L774YrDpU6ZMkXfeeUefjxgxQlvkotNlNKhA692xY8fa86LoFsXHrVu31mAsceLEGsgNGDDAngc5iwj20I/fqFGjtJh50qRJuiyjQYMG2m0M+g9EEFm0aFHtIsbZMCS8tBARERFFBwHVD2BMx34AiQIL+wH0DfYDSBQYom0/gERERET05DEAJCIiInIZBoBERERELsMAkIiIiMhlGAASERERuQwDQCIiIiKXYQBIRERE5DIMAImIiIhchgEgERERkcswACQiIiJyGQaARERERC7DAJCIiIjIZRgAEhEREbkMA0AiIiIil2EASEREROQyDACJiIiIXIYBIBEREZHLMAAkIiIichkGgEREREQuwwCQiIiIyGUYABIRERG5DANAIiIiIpdhAEhERETkMgwAiYiIiFyGASARERGRyzAAJCIiInIZBoBERERELsMAkIiIiMhlGAASERERuQwDQCIiIiKXYQBIRERE5DIMAIliiDfeeENSpkwp9evXt6ddvHhRXn/9dcmXL58UKFBAjhw5EuJzXbp0keeee04fb775pty6dUunnzhxQl544QUpWrSolChRQrZu3Rrsc0uWLJGgoCDZt2/fU/h1RETkSwwAiWKIDh06yHfffRdiWoMGDeTQoUOybds2yZAhQ4jP9e/fX/bs2aOPrFmzyvjx43X60KFD5e2335Zdu3bJ559/Ln379rU/c+fOHRkxYoQ8//zzT+GXERGRrzEAJIohXnzxRUmaNKn9+urVqxr0IYiDRIkSSeLEiUN8LlmyZPrfsiwN7JCrB/h//fp1e1kZM2a0P4PgsHXr1pIwYcIn/ruIyL1wo1mwYEEtwWjfvr2ep5waNWokRYoUkUKFCuk56dGjRzp99+7dUrp0aS3BKF++vBw9elSnT506VdKlS6fT8Zg1a5a4FQNA8tsB2rVrV8mbN68ULlxYWrZsKQ8ePNDp+PyHH34ouXPnlpIlS3ottqTwHTt2TNKkSSONGzeWYsWKSadOnex17Anb7ZlnnpH9+/fLBx98oNN69+4t06ZNk8yZM+tnP/30U53+zz//yKZNm4IVNRMR+dqFCxdkzJgxsn37dtm7d6/+x7nHCSUWCPbw/n///ScLFy7U6SixGDBggJZgNG3aVIYMGWJ/plmzZjodD5SQuBUDQPLbAVq1alUNOFD0ePfuXbv4cunSpTrf33//LZ988on06NGDWykKEOxt2bJFunXrptsF22rKlCle5x09erScOnVKA8Uff/xRp82cOVMD9pMnT8rEiROlVatWduD+xRdfcJsQ0VM5j6Fk4v79+/pA7p23EoyHDx/qdSQiJRj0vxgAkt8O0FdffVXixImjr5HThwAEFi1apHdsULNmTdmwYUOIXEUKX6ZMmSRHjhxazBErVixtDII73tBgHuTWzp07V19PnjxZG4WY7WA+u2PHDqldu7Zkz55dg30E8gcOHOAmoRgDDaBQpQKlGmgcNWfOHJ3+zjvvSM6cOe3iQ2fpxPDhw7VEA59BqQg9vrRp0+oNJ+omo4SicuXKkitXrhDzoTQiffr0kiRJEj03mWoqnTt31hIM3PjiufHDDz/odkX1mHPnzrl2UzEAJL8doM4AErlNVapU0denT5/W4AUQHKJlK1qzUuTgjhfBOIqCYc2aNZI/f3593qtXL5k/f74+/+uvv+zPIPhGi2HIkiWL/Pbbb/ocgR5eA+rSoBgYjzJlysiKFSv0okcUU+DGdOTIkXpjs3LlSunYsaPcvHnTzi03xYfmXPfrr7/K6tWrtUU8PoNqF/T4Ll++rL0N4FyDDAJkBqxduzbEfD/99JOcOXNGMwrMOWvs2LEybtw4LcFo166dHQC+9tpreg7bs2ePNmJr27atazcVA0Dy2wFqoIgSgQQq7FLUIfhGjt3PP/+sd70bN27UnIh69eppPctr167Je++9p/PiQmVaBKP+H97HHTFOjP369bNzNEaNGqX1N3ECnTBhAjcPuebmCTl8gOMEdWkvXboU6vyo5oKbqrhx4+prz1IQihoE1qgLnipVKm1whpIIzypGRrx48bQrLFPFCFVZatSooc/feustvTZB6tSpJX78+Poc50PP7q3cJI6/E0Ax5wAFc4BWqlQpzAMUxb/mLu3gwYMaRBrISUQwiWJhBIwIMnHQUvjbwhsU2XpCcXzZsmX1+bJly7x+Do12EESGBbmKRDEZ6s/ieDE54Cjx6NOnjwYXAwcOlNixY2suOo4/5DKlSJFC60U/++yz/k56tId1jsANVYwQXON88/7772sjDuTcoc4ySoyyZcum2wjXEdM1Fa5JuBYhcwGZDiieh7Nnz9o3vwsWLNAGjG7FHEDy2QGKAxAHKA40HKBogIA6gf/++6/Oaw5QU8SIxh6TJk2S2bNna5GLUatWLZk+fbo9DwIVU2+QfAO5hEQUNuT64Vxmcr8HDx6sN6ybN2/W3HIUMZpqLCgixjkPHau3aNGCq9YHELwh0EaghxIKFLmjChEaFCKjANeXhg0bagkGSipQ3xw9SJhcWTRiw3RkNAwbNkyno2gfN7dFihTRXg4QrLtVkMXa9ZHyzTff6I6EuwjsQF9//XWEO8NFEVzy5Mm1RZJpGBET4G543rx52ojglVde0WLD4sWLy+LFi/UuDNNu3LihuXmoWI0DEAEfcg5xAKOOH6D4EstCNzE4iHHXhrtpZOXnyZNHorPhww/6OwkxQteu/1uH0VeaNOF28YUZM3y7XQIBGqyhpALFhKZRmhNuTtE4BP3KoSEUeiwwueqoG41W9+R7CLQRYCPjgB4vzmARcCSgw0hk8eOuD/XVEMjgwD98+LCr63xglAg8nAcoAjbUQ4PQihHRzYs3CCRZ34yI/AU3q2jx+/LLLwcL/lCPGfUDcZOKBlOm+BC5Uij9QACIXEAUSdKTgc7sGfz5BnMAIwFBX6lSpewsY5wEUASKCvI9e/Z0bQ4ghY85gL7BHMDAFNNyANetW6f1mFHsaKBaCoZWRB+lOPejeBIlQmhQgNxCFBWjX1MEKMgkQLFlTMPzWGCex5yYA/gE3Lt3TysDo6WXM6cKLS/Dqyj/NPEADfwDlIgCW4UKFewRi5xWrVrldX4EgW4eUoyiJxYBRxDu+tCIAX3ZOeH1oUOHvH4Gd4V4GMj5MxH6k3Lnzo0ntmw38fU24nYJzO1y/z6PF194kuc0Chw8jwX+8WKWHZHmHQwAnyC0GDPjpzqZ7gQocH38sb9TQN5wuwQm1scnCqzzGIbBQ5WzsDAAjCB0BIr+njyHjcFr06eQJxQXO4efQZECuhVAn3Zu7dYEdycIgDHUEutBBg5ul8DE7RKYuF0CE7eLaM4fgj90kxMeBoARhE6MS5QooV2T1KlTxw7o8Dq0oWRQL8T0OG6gWxP63/GBGQAGHm6XwMTtEpi4XQKT27dL8nBy/gwGgJGA3LzmzZvrCBXo+w/dwJg+iYiIiIiiCwaAkdCgQQPt3BNjpaIjaIwVuXz58hANQ4iIiIgCGQPASEJxb2hFvhQ+FIn3798/RNE4+Re3S2DidglM3C6BidslctgRNBEREZHLxPJ3AoiIiIjo6WIASEREROQyDADpsVy5ckUmTJgQ5jxTp06Vrl27ck0HsBo1asjt27dDbM9t27ZJt27d/Jo2N7t165Zky5aNx89ThP29YMGC2lfrkSNHorSMF198Ufbt2+fztFFIQ4cOfazVsmDBAvnzzz9duWoZANITDwAp8P3888+SMGHCENsTXR4NGzbMr2lzs88//1zKlCnj72S4Cm5Y9+7dqx3q5sqVy9/JoXAwAIw6BoD0WPr06SMHDhzQLnEw7F3Hjh2lUKFC+vrXX38NMf8PP/wgVatW1ZyN6dOnS6lSpaRIkSL2iCn//POPvkZ/i/nz59eudyIypqGbYZ0999xz8tZbb+k6w7p78OCBrFy5UrcDtgfWL9YjxrNu0qSJFChQQAoXLixTpkzRZWTPnl1u3LgRbHsOGDBA1qxZI/Xr17fHw37ttdf0u5DDge+Fd955Rzp06KCBSp48eeT333/36/qIKf766y8dZ7x69er+ToprvPHGG3L58mUpXry49vVqcvEwEhRKMXDMvPLKK9r/K4wbN84+h7399tty//59P/+CmGHIkCF63sL6/v7774OdhwDPMQ3nK9y04nz14YcfhnoudJ7jYMmSJXre2rx5syxatEjat2+vyzh//ry4CQNAeuwcCgQTu3bt0mKTv//+W/bs2aPZ6u+++67cuXPHnnfGjBkybdo0fe/ff/+VhQsXysaNG2X37t0aXCxdulTnO3jwoPTo0UMDEQy1t27dOm6lcOBC1b17d113uAhNnDhR3nvvPV3X2B6HDx+W+fPn63Y6duyYrlvkctStWzfU7Yn+Lp0++eQTqVixoi6vdevWetI0MMThpk2bZPz48Ro40uNDwIHxxOnpwTGC0Zqw/ydKlMiefvHiRalWrZoeM5kyZZJ58+bpdAQaW7du1XMYhgSdzUGRHxvWJ9Yjqp/gZhLnodOnT3udF+crs70QjHs7F+K6E5rSpUtL7dq1ZfTo0bqMdOnSiZswACSfQaCGu+BYsWLp3dazzz6rgQcsW7ZMgxKcYFHUiCH0EDCgiBF3XniO4BHy5s2rQQjq4BQrVszOaaLQ5c6dW9clNGzYUObOnavrEdsB26Nx48byxx9/SM6cOfVk2qZNG80hjOiQQWb7IvfQXPi2bNliv2eGR8Rwidxejw83Rzh+8CD/S5IkiVSuXDnEPo7Ar0KFCppT9dNPP+mNFT2e9evXS7169SRBggSSKlUqzXFNnDhxlM+FzEAIHTuCpqcCFzIUaeHEiax5jKOMHCp0Cu2E952dRMeOHVuLLSlsCJadz52vnVKmTKm5GKjzN2LECA0Chw8fHqXV6/wOs824vXwDN0Q//vijzJkzR4utkJOBsU09c2Xp6QjtnNSqVSstucA5bcyYMbz5eUKQO4drhnH37t1InwvjxIljLyOsz7sJcwDpsSRNmlSuX7+uz3EnjIsW6pqhiBcBH3KhAJWpUZfjzTff1PdwVzdr1iwtWgHUvThz5gy3RhRhXe/YsUOfY70ipw4t27CucdJD3ctKlSppUTteIwcPRbo4sYa2PT1h+86cOVOfI7cDdaToyUDR74kTJzSgQICOmyUGf4EHdQExFOi9e/f0GKPHh/MMitgRpKE+5qpVq7S+HnJXUZ8P1YI2bNhgz+950+l5LsTyAK3pcb7D9QlVYyJyzovpmANIjyV16tRaYRpFIAjuUMSI57jbQpEvsvENVJRGPY3XX39dc55QgReBIAIS3GGj9V1ksvrp/0OFaVScRv08VEpHUXzGjBl1XeOkWaVKFS2mxfs4mWKdYxuNHDkyzO2JoNFAwIjPfvfdd1o0g+1F5GY4JlDciLpjqK5Cjw/rE+ceFLUj9w6NCxG8oasqVA1CpoJzXaOhB85XOFf17NnT67kQPv74Y62XjjqDqPt3/f+CPhQT4wYLn8F1yU31ADkUHFE0h1witIpDpWkiIrfiuTByWARMRERE5DLMASQiIiJyGeYAEhEREbkMA0B6bGgYgJ7Vo8rZQ7u38TTRbx36sSNul+gwbCG6bkG3IBhVIhDqRJk+0SJ63IUnqp97WjBqx9OCc5RpZeocYYKezj4cVRyf/n8xAKSA98wzz2gXMhRYott2CSsA9GVfk5MnT9ZhDtHpeUT48rvZZ+bTY9Y1ulpCJ+uPswwif2AASDbPsXnNuLzIfcMYrxj+C/0noQk9mtqjvyUDHQuj+xDkfJixYHFy69Kli71MEyxgHGD09I4m/bhbNmP9omuSDz74QPLly6fD89y+fTvEHSDu3NCH3auvvqo9vn/55Zd2GtDMH10EvPzyyzp+6uPkSgYSbhff8DbOMfYVdC9Rvnx5uXbtmr7GfuwcyxrzYX9DNzro0NyMW+1tXGUMTWVGTMF3hDV+Mo4n9KX4xRdf6Hs4VtD1BeZFP2Y1a9bUfRwdDBvoqgLHE+YxHXh7/g5P6KPO29ioYY29auD3YBq+Dx2HO2F9Id2LFy/WXHp8N45zzIsuOAIJzltY1+g+BOsVNwNIP84XpnPgI0eO6LoFtKh/4YUXdHtg+2GoQ5P7ia5GsBz0TwfYboMGDQrxnb7e9m6GjtA9x4cPbRthcAFsR+zPnTp1speB6wGOX1xLohqwxzgWkWVZBw4csOrVq2fdv39f10fTpk2tMWPGWHHjxrUOHTpkPXjwwMqXL5/VtWtXfX/cuHFW+/bt9Xnz5s2t2rVrW48ePbL2799v5cmTR5+PHz/eGjZsmM5z69Yt67nnnrP+++8/ndauXTudvnTpUkR/1vXr1605c+bYy9mzZ48VO3Zsa+/evdaxY8esEiVK6PxTpkyx8ubNq/NfvnzZSpcunXX37l1r8+bNVqlSpfT5uXPnrJQpU1qLFy+O9tuW28V3nPsRrF692kqWLJl16tQpfX3v3j3r2rVr+vzMmTO6v5r5UqVKZZ09e1b3r9y5c1v//vuvtW3bNqtcuXL28q5cuaL/X3jhBd1voU2bNtaQIUP0+Y8//mi99tpr9jHz5ptv6r5uPtOvXz99PnDgQP0O7N94ZMiQQaevWLHCatu2rX4Gx+NLL72k3+P5Ozx/c1BQkLV161Z93ahRIz2GIFu2bHocbdmyxSpevLh1+/Zt6+LFi1bOnDl1WTg2X375ZevOnTs6P94znzt58qRVsWJF+xgbPny41bt3b32Oc8jNmzctf0mdOnWIaZcuXbLX9ahRo3Qdm/PcypUr9TnW/+jRo3U/qFSpkv17J0+ebJ/38NtxXjTMtsa5Z+fOnbo+sG19ve3dDPswrkO4tmCdYX2tXbs21G1kpmHeunXrWuvWrdN9G9vu+PHjun3Lly9vdenSxXI7dgRNyjk2r8mlw50V7pDNaB64+zLjYSJXALl+Bu7K0GknckMwbuapU6e0U03U4TODcV+9elWOHj2qOSQYrBuQa4HhyQDTzXJMroM3yI3Bd5hiSNMzPOpcxYsXTzvyfOmll2LEluV2ebKQa4V9CJCrgP0S+yFGF8A41sg9g3LlyumID4CcBRT74b8ZVxkdbqOzbU9YljlOkAvXoUMH+z303egctgq53oB9H8chOqw1IxVgRAQcTxh2zOReoCNbjPaCTrmdvyO8sVEXLVoUrJ6ac+xVPNA5+9atWzUnv0WLFvYwaPgeA8ftwIEDpVatWvoaOS7NmjXTzsXxu/AbAsnx48e1c2GcK1CygFIMwO9DkT3Oaxg1AtsL2x1j/CJXFZBjWrBgQXtZWI6nXr16aS4g1sGT2PbmHOlWZnx4QO4rcr1D20Y4Zw4bNkzu3LmjI0xVq1ZNBxhA7l+WLFns7XH8+HFxOwaApLyNzYsiCxQ/GrFixbIvBnjurL/ibfxFLHP8+PGaTe8ptLFqQ5se3ricphg5puF2ebISJUpkP0fRJ4b22rlzpwYyaExgAkBv+1xUxlV27t/O73Z+h/M4M6/xfdgXcHyiKMwJF0PPZUVlnOjIQEC8bNkyLXoDjMKAQBLFbI0aNdJgyAQ1gaB9+/ZaBQBBOtJoRrFB8Wu7du30t+AGF9scQT2CjNWrV3tdlrd1jeoBGKoPAfmT2PZu53n8JU+e3Os2QtDXsWNHLR7GSEhdu3a1x/31xX4f07AOIKnHHZt39uzZGoQdPHhQcyaQG4GT7dixY+0TGHID8Ryt5vBdsHz5cr3DBUw3y9m/f3+k6hHhgrRw4UKtK3LhwgW9KMYE3C6+E96Yn6gThlw+BH8IEsyxEJrwxlX29fjJOJ4mTZqkufPmBg256uEJbWzUsMZeRTqRK4Z6jeYCaupYwdChQ/Vii3q3gBzRDBkyaB3epk2bBlwdQGzbTJky6bkFQxkaCAqQo4chwkxgjTrIGId5+/bt+hq//9ChQ2EuH8vp1q1bsHqSHDv7yUFOtbdthH0S2wJDWuLYMGP+YpsiOD958qTmFqKlPjEHkP4Pss+9jc0bUbjbQtEFLk4YAxgHIXIUjx07pndqWCbmwZ32Rx99ZFeeR1FM1qxZdRl169aVX375Re/EkeWPIuiIwnKQdvwOZPOjMnqyZMmi/fbldvGdsMY5BjR2QpEm3sfF2+yXoUE1h7DGVfb1+MkoykIjljJlyuh3ophw7ty54X4utLFRwxp7FccqHrjAYp3FjRtXi0tNMSbmw3GOokz8buSGotgN8yFdP/zwg/gLgtjMmTPbr5Eu5JwitxLbACUSCFgNFIt//fXX2vACUI0EgTJ+K24YcNOKQBdBRFiwXpELaHDs7CcnrG2EQB7XFmRC4FgxAePo0aP1GoHrQmjVi9yGI4FQjIG+yVA3EBcA5GBs3LjxqfYJRtwuFP0gKEd9Ms9WzkQxHesAUozRqlUru+J+7969GfwFCG4XClTvv/++rF27VhsOELkNcwCJiIiIXIaNQIiIiIhchgEgERERkcswACQiIiJyGQaARERERC7DAJCIiIjIZRgAEhEREbkMA0AiIiIil2EASEREROQyDACJiIiIxF3+HyiicOwvZx/oAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from collections import OrderedDict\n",
"groups = OrderedDict()\n",
"for name, p in model.named_parameters():\n",
" top = name.split(\".\")[0]\n",
" label = {\"token_embedding\": \"token embedding\", \"position_embedding\": \"position embedding\",\n",
" \"blocks\": \"4 transformer blocks\", \"ln_f\": \"final LayerNorm\",\n",
" \"lm_head\": \"output head\"}.get(top, top)\n",
" groups[label] = groups.get(label, 0) + p.numel()\n",
"\n",
"total = sum(groups.values())\n",
"for label, n in groups.items():\n",
" print(f\" {label:22s} {n:8,d} ({100*n/total:4.1f}%)\")\n",
"print(f\" {'TOTAL':22s} {total:8,d}\")\n",
"\n",
"import matplotlib.pyplot as plt\n",
"plt.figure(figsize=(6.5, 3))\n",
"bars = plt.bar([l.replace(\" \", \"\\n\", 1) for l in groups], list(groups.values()),\n",
" color=[\"#5b5bd6\" if \"blocks\" in l else \"#8b8bd6\" for l in groups])\n",
"for b, n in zip(bars, groups.values()):\n",
" plt.text(b.get_x() + b.get_width() / 2, n, f\"{n:,}\", ha=\"center\", va=\"bottom\", fontsize=7)\n",
"plt.ylabel(\"parameters\"); plt.title(\"where the parameters live (blocks dominate)\")\n",
"plt.xticks(fontsize=7); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "m8c1c",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
The loop reads each parameter tensor's name (like blocks.0.sa...) and adds its size to the matching component.
\n",
"
The token and position embeddings and the output head are small -- a few thousand numbers each; the four blocks together hold 791,552, about 96%.
\n",
"
Inside each block most of that is the feed-forward network's two big matrices (the 4× expand and contract from notebook 06).
\n",
"
The bar chart makes the imbalance plain: real capacity lives in the stacked blocks, which is why \"make it bigger\" almost always means more or wider blocks.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "a3439a27",
"metadata": {},
"source": [
"## Forward pass and an untrained sample\n",
"\n",
"A forward pass returns logits with shape `(batch, time, vocab)` and, when given targets, a loss near `ln(vocab) = 4.17`. This is because a freshly built model has random weights and is essentially guessing all 65 characters as equally likely (notebook 01's random baseline). Generating from it produces pure gibberish. That is our starting line; notebook 09 trains it into something that writes Shakespeare."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "6c9f196c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:32.183220Z",
"iopub.status.busy": "2026-07-01T09:18:32.183138Z",
"iopub.status.idle": "2026-07-01T09:18:40.526725Z",
"shell.execute_reply": "2026-07-01T09:18:40.526309Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"logits: (4, 128, 65) | loss: 4.197 | ln(vocab) = 4.174\n",
"\n",
"--- untrained sample (gibberish) ---\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"XwO\n",
"AOdTZYN3cESOPkTIlTqNGXy:.CeiZoFkci-:jP?vV?S?.MpsHhkYKrAMr\n",
"hFulHerHEe.-o-XwKbQRT!dj!SCB$l n'm?'c3hhCZTV?koIuovIR\n",
"OTCGD;PFh!VSBBj3UpFUapTSDDaZsQFFQiXNQT&:Nlp-$iMGbe\n",
"SS\n",
"McDkoSkjpvvpeQSxmn-FnFQajfL:S,\n"
]
}
],
"source": [
"import math\n",
"B, T = 4, cfg.block_size\n",
"x = torch.randint(0, vocab_size, (B, T), device=device)\n",
"y = torch.randint(0, vocab_size, (B, T), device=device)\n",
"logits, loss = model(x, y)\n",
"print(\"logits:\", tuple(logits.shape), \"| loss: %.3f | ln(vocab) = %.3f\" % (loss.item(), math.log(vocab_size)))\n",
"\n",
"stoi = {c:i for i,c in enumerate(sorted(set(text)))}; itos = {i:c for c,i in stoi.items()}\n",
"decode = lambda l: \"\".join(itos[i] for i in l)\n",
"ctx = torch.zeros((1,1), dtype=torch.long, device=device)\n",
"print(\"\\n--- untrained sample (gibberish) ---\")\n",
"print(decode(model.generate(ctx, 200)[0].tolist()))"
]
},
{
"cell_type": "markdown",
"id": "1ff898b4",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
x = torch.randint(0, vocab_size, (B, T), device=device): random token ids shaped like a real batch, just to exercise the forward pass.
\n",
"
logits, loss = model(x, y): run the model; with targets it also returns the loss.
\n",
"
math.log(vocab_size): ln(65) = 4.174, the know-nothing baseline; the untrained loss lands right next to it.
\n",
"
ctx = torch.zeros((1,1), dtype=torch.long, device=device): the generation prompt: one sequence holding one token (id 0, the newline) -- a neutral start.
\n",
"
decode(model.generate(ctx, 200)[0].tolist()): generate 200 tokens, take the first (only) sequence, turn the tensor into a plain list, decode to text. Untrained = gibberish.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "m8c2a",
"metadata": {},
"source": [
"### Follow the shapes through one forward pass\n",
"\n",
"The diagram at the top of this page traced the path from token ids to logits. Here it is with real tensors, so you can watch each stage keep the batch and time axes and change only the last one: ids become 128-wide vectors, the blocks refine them without reshaping, and the head finally opens each position out to one score per vocabulary character."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "m8c2b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T09:18:40.527925Z",
"iopub.status.busy": "2026-07-01T09:18:40.527858Z",
"iopub.status.idle": "2026-07-01T09:18:40.608533Z",
"shell.execute_reply": "2026-07-01T09:18:40.608158Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"idx (token ids) : (1, 20) (batch, time)\n",
"token + position embed : (1, 20, 128) (batch, time, width 128)\n",
"after the 4 blocks : (1, 20, 128) (same shape -- blocks preserve it)\n",
"after final LayerNorm : (1, 20, 128) (same shape)\n",
"logits (lm_head) : (1, 20, 65) (a score per vocab char, per position)\n"
]
}
],
"source": [
"# Run the model's stages by hand and print the shape after each (mirrors GPT.forward).\n",
"idx = torch.randint(0, cfg.vocab_size, (1, 20), device=device)\n",
"tok = model.token_embedding(idx)\n",
"pos = model.position_embedding(torch.arange(20, device=device))\n",
"h = tok + pos\n",
"hb = model.blocks(h)\n",
"hn = model.ln_f(hb)\n",
"logits = model.lm_head(hn)\n",
"print(f\"idx (token ids) : {tuple(idx.shape)} (batch, time)\")\n",
"print(f\"token + position embed : {tuple(h.shape)} (batch, time, width 128)\")\n",
"print(f\"after the 4 blocks : {tuple(hb.shape)} (same shape -- blocks preserve it)\")\n",
"print(f\"after final LayerNorm : {tuple(hn.shape)} (same shape)\")\n",
"print(f\"logits (lm_head) : {tuple(logits.shape)} (a score per vocab char, per position)\")"
]
},
{
"cell_type": "markdown",
"id": "m8c2c",
"metadata": {},
"source": [
"Line by line: what each line does\n",
"
\n",
"
idx: a batch of token ids, shape (batch, time).
\n",
"
token_embedding + position_embedding: each id becomes a 128-wide vector and the position vector is added -- now (batch, time, 128).
\n",
"
model.blocks(h): the four Transformer blocks run and keep the shape exactly, just refining the numbers.
\n",
"
ln_f then lm_head: a final normalize, then a linear layer opens each position out to one score per vocabulary character -- (batch, time, 65).
\n",
"
Every stage keeps the batch and time axes; only the last axis changes. That regularity is exactly what lets the blocks stack.
\n",
"
\n",
""
]
},
{
"cell_type": "markdown",
"id": "f60cf0d1",
"metadata": {},
"source": [
"## Recap\n",
"\n",
"The GPT is assembled and lives in `gpt.py`: embeddings, then blocks, then a final norm, then logits, with about a million numbers in total, and both the forward pass and `generate` already working (just untrained so far). Every line traces back to something you built by hand in Part 1; nothing here is new, it has only been wired together.\n",
"\n",
"Next, notebook 09 trains it on Shakespeare and we watch the loss fall and the text sharpen."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (.venv)",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}