{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2fa091c4",
   "metadata": {},
   "source": [
    "# 10 - Sampling, and what lies beyond\n",
    "\n",
    "This final notebook covers two things.\n",
    "\n",
    "1. **Sampling.** The trained model gives a probability for every possible next token. *How* you pick from those probabilities (greedy, temperature, top-k, or top-p, all explained below) changes the character of the output considerably, and we will see each in turn.\n",
    "2. **What lies beyond.** An honest map of what separates your roughly one-million-number Shakespeare model from a frontier model, meaning one of the large commercial systems. The short version is that the difference is mostly scale, data, and alignment rather than new mysteries; you have already built every core idea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "colab-setup-10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Colab setup -- fetch the files this notebook needs.\n",
    "# (Does nothing when run locally in the course folder.)\n",
    "import os, urllib.request\n",
    "BASE = (\"https://raw.githubusercontent.com/waze\"\n",
    "        \"emlabs/llm-book-code/main/\")\n",
    "for f in ['gpt.py']:\n",
    "    if not os.path.exists(f):\n",
    "        d = os.path.dirname(f)\n",
    "        if d: os.makedirs(d, exist_ok=True)\n",
    "        urllib.request.urlretrieve(BASE + f, f)\n",
    "        print(\"downloaded\", f)\n",
    "if not os.path.exists(\"checkpoints/gpt_shakespeare.pt\"):\n",
    "    print(\"NOTE: this notebook loads the model trained in\\n\"\n",
    "          \"notebook 09. Run notebook 09 first (in the same\\n\"\n",
    "          \"Colab session), then re-run this cell.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "colab-setup-md-10",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n<ul>\n<li><code>import os, urllib.request</code>: two standard-library toolboxes: checking files on disk, and downloading from the web.</li>\n<li><code>BASE = (...)</code>: the web address of this course's folder on GitHub, split over two lines (Python glues adjacent strings together).</li>\n<li><code>for f in [...]</code>: loop over the file names this notebook needs.</li>\n<li><code>if not os.path.exists(f)</code>: only download what is missing -- running locally, everything already exists, so nothing happens.</li>\n<li><code>os.makedirs(d, exist_ok=True)</code>: create the folder for the file if it has one (<code>exist_ok</code> means don't complain if it's already there).</li>\n<li><code>urllib.request.urlretrieve(...)</code>: download the file and save it under the same name here.</li>\n</ul>\n</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "364a3946",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:04.162539Z",
     "iopub.status.busy": "2026-07-01T09:30:04.162384Z",
     "iopub.status.idle": "2026-07-01T09:30:04.655647Z",
     "shell.execute_reply": "2026-07-01T09:30:04.655255Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loaded trained GPT (824,897 params) on mps\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch.nn import functional as F\n",
    "from gpt import GPT, GPTConfig\n",
    "device = \"mps\" if torch.backends.mps.is_available() else (\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# load the model we trained in notebook 09\n",
    "ckpt = torch.load(\"checkpoints/gpt_shakespeare.pt\", map_location=device, weights_only=False)\n",
    "cfg = ckpt[\"config\"]\n",
    "model = GPT(cfg).to(device); model.load_state_dict(ckpt[\"model\"]); model.eval()\n",
    "stoi, itos = ckpt[\"stoi\"], ckpt[\"itos\"]\n",
    "decode = lambda l: \"\".join(itos[i] for i in l)\n",
    "print(f\"loaded trained GPT ({model.num_params():,} params) on {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3171a8ca",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>ckpt = torch.load(..., map_location=device, weights_only=False)</code>: read the checkpoint file back. <code>map_location</code> says where the tensors should live; the flag allows the bundled config object to load too.</li>\n",
    "<li><code>cfg = ckpt[\"config\"]</code>: pull out the pieces we saved in notebook 09 (config, weights, tokenizer tables).</li>\n",
    "<li><code>model = GPT(cfg).to(device); model.load_state_dict(ckpt[\"model\"])</code>: build a fresh model skeleton and pour the saved weights into it.</li>\n",
    "<li><code>model.eval()</code>: inference mode (dropout off).</li>\n",
    "<li><code>decode = lambda l: \"\".join(itos[i] for i in l)</code>: rebuild the decoder from the saved table.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7758e8a6",
   "metadata": {},
   "source": [
    "## How sampling works\n",
    "\n",
    "At each step the model hands you a **probability distribution** over the whole vocabulary, for example \"next character: 31% `t`, 12% `h`, and so on.\" Sampling is simply how you pick one character from that list, and the rule you choose changes the whole personality of the output:\n",
    "\n",
    "- **Greedy**: always take the single most likely token (using `argmax`, which just means \"the position of the largest value\"). It is completely predictable, but it gets stuck in loops and reads as dull; you will see it repeat \"the world of the world\" below.\n",
    "- **Temperature**: a creativity dial. You divide the logits by a number `T` before applying softmax. A `T` below 1 widens the gaps so the top choices dominate (safer, more repetitive); a `T` above 1 narrows the gaps so long-shot choices get a look in (more varied, more typos); and `T` equal to 1 leaves the distribution untouched.\n",
    "- **Top-k**: before sampling, discard everything except the `k` most likely tokens, so the model cannot pick something bizarre from the long tail of unlikely options.\n",
    "- **Top-p** (also called nucleus sampling): the same idea but adaptive. Keep just enough of the top tokens for their probabilities to add up to `p` (say 0.9), however many that turns out to be.\n",
    "\n",
    "Below is a single generate function that supports all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f9eb30e6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:04.656830Z",
     "iopub.status.busy": "2026-07-01T09:30:04.656736Z",
     "iopub.status.idle": "2026-07-01T09:30:04.659420Z",
     "shell.execute_reply": "2026-07-01T09:30:04.659089Z"
    }
   },
   "outputs": [],
   "source": [
    "@torch.no_grad()\n",
    "def generate(model, prompt=\"\", max_new_tokens=300, temperature=1.0, top_k=None, top_p=None, greedy=False, seed=0):\n",
    "    torch.manual_seed(seed)\n",
    "    idx = torch.tensor([[stoi[c] for c in prompt]] if prompt else [[0]], dtype=torch.long, device=device)\n",
    "    for _ in range(max_new_tokens):\n",
    "        logits, _ = model(idx[:, -cfg.block_size:])\n",
    "        logits = logits[:, -1, :]\n",
    "        if greedy:\n",
    "            idx_next = logits.argmax(dim=-1, keepdim=True)\n",
    "        else:\n",
    "            logits = logits / temperature\n",
    "            if top_k is not None:\n",
    "                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))\n",
    "                logits[logits < v[:, [-1]]] = -float(\"inf\")\n",
    "            if top_p is not None:\n",
    "                s_logits, s_idx = torch.sort(logits, descending=True)\n",
    "                probs = F.softmax(s_logits, dim=-1)\n",
    "                cum = torch.cumsum(probs, dim=-1)\n",
    "                remove = cum - probs > top_p                 # keep until cumulative prob passes p\n",
    "                s_logits[remove] = -float(\"inf\")\n",
    "                logits = torch.full_like(logits, -float(\"inf\")).scatter(1, s_idx, s_logits)\n",
    "            probs = F.softmax(logits, dim=-1)\n",
    "            idx_next = torch.multinomial(probs, num_samples=1)\n",
    "        idx = torch.cat([idx, idx_next], dim=1)\n",
    "    return decode(idx[0].tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e289adb3",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>def generate(model, prompt=\"\", ..., greedy=False, seed=0):</code>: one function with every sampling strategy as an optional setting.</li>\n",
    "<li><code>torch.manual_seed(seed)</code>: same seed &rarr; same dice rolls &rarr; comparable outputs across strategies.</li>\n",
    "<li><code>idx = torch.tensor([[stoi[c] for c in prompt]] ...)</code>: encode the prompt to ids (note the double brackets: a batch of one sequence); with no prompt, start from token 0.</li>\n",
    "<li><code>logits, _ = model(idx[:, -cfg.block_size:])</code>: keep only the last 128 tokens of context (the model can't see further back) and predict.</li>\n",
    "<li><code>logits = logits[:, -1, :]</code>: take the predictions at the <i>last</i> position only -- that's where the next token gets decided.</li>\n",
    "<li><code>idx_next = logits.argmax(dim=-1, keepdim=True)</code>: greedy: just take the single highest scorer, no dice.</li>\n",
    "<li><code>logits = logits / temperature</code>: the creativity dial: dividing by T &lt; 1 stretches the score gaps (sharper, safer), T &gt; 1 squashes them (flatter, wilder).</li>\n",
    "<li>Top-k block: <code>torch.topk</code> finds the k highest scores; <code>logits[logits &lt; v[:, [-1]]] = -inf</code> erases everything below the k-th (and -inf becomes probability 0 after softmax).</li>\n",
    "<li>Top-p block: <code>torch.sort</code> descending, <code>torch.cumsum</code> builds the running total of probability, <code>cum - probs &gt; top_p</code> marks tokens past the cutoff -- \"keep just enough of the top tokens to cover 90%.\" <code>scatter</code> writes the kept scores back to their original positions.</li>\n",
    "<li><code>probs = F.softmax(logits, dim=-1); idx_next = torch.multinomial(probs, num_samples=1)</code>: turn scores into probabilities and roll the weighted die; <code>torch.cat</code> appends the pick and the loop continues.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3f316d2",
   "metadata": {},
   "source": [
    "### Greedy versus temperature\n",
    "\n",
    "Greedy is predictable and quickly repetitive. A low temperature is safe but a little dull; a high temperature is creative but error-prone. Watch how the texture of the text changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cf305517",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:04.660448Z",
     "iopub.status.busy": "2026-07-01T09:30:04.660388Z",
     "iopub.status.idle": "2026-07-01T09:30:06.254696Z",
     "shell.execute_reply": "2026-07-01T09:30:06.254245Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== GREEDY (argmax) -- note the repetition ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO:\n",
      "The world the seat of the world of the world,\n",
      "And the seal of the world of the world the world,\n",
      "And the seat of the world of the world,\n",
      "And the world of the world of the world,\n",
      "And the world of the w\n"
     ]
    }
   ],
   "source": [
    "print(\"=== GREEDY (argmax) -- note the repetition ===\")\n",
    "print(generate(model, prompt=\"ROMEO:\", max_new_tokens=200, greedy=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70fa8cb5",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>generate(model, prompt=\"ROMEO:\", max_new_tokens=200, greedy=True)</code>: always take the top choice. Watch it fall into a loop (\"the world of the world...\") -- pure determinism finds a rut and stays there.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e2bfff4b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:06.256054Z",
     "iopub.status.busy": "2026-07-01T09:30:06.255966Z",
     "iopub.status.idle": "2026-07-01T09:30:09.545620Z",
     "shell.execute_reply": "2026-07-01T09:30:09.545169Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== temperature = 0.5 ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO:\n",
      "We shall can the be stealter, and the world to sure,\n",
      "But thou would he in the world confinitence\n",
      "Of the moon of the mother wind the doing of this\n",
      "To be some of her life.\n",
      "\n",
      "PAULINA:\n",
      "\n",
      "=== temperature = 1.0 ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO:\n",
      "Wwas their common twices the next of hollowing.\n",
      "\n",
      "KING LEDWARD IV:\n",
      "Capuliten, my wife, let them: our mind,\n",
      "I  well apatient in my teteritanges,\n",
      "And more sight'd and let him wherefo\n",
      "\n",
      "=== temperature = 1.5 ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO:\n",
      "Bwas their coundry wit.\n",
      "\n",
      "MOPSA.\n",
      "\n",
      "3BELLADY:\n",
      "No gentleep, Lucious dejaUTIO:\n",
      "This many confircy ambounted Rome; well,\n",
      "Fathermenn, we at till you, wruth alas;\n",
      "If'd alf the pleasces.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for T in [0.5, 1.0, 1.5]:\n",
    "    print(f\"=== temperature = {T} ===\")\n",
    "    print(generate(model, prompt=\"ROMEO:\", max_new_tokens=180, temperature=T, seed=1))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5ea48b5",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>for T in [0.5, 1.0, 1.5]:</code>: same prompt, same seed, three temperatures, so the <i>only</i> difference you see is the dial.</li>\n",
    "<li><code>generate(..., temperature=T, seed=1)</code>: low T is safe and a touch repetitive; T=1 is the raw distribution; high T picks long-shot characters and the spelling falls apart.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m10c1a",
   "metadata": {},
   "source": [
    "### The real distribution behind the temperature knob\n",
    "\n",
    "You just watched temperature change the generated text. Here is what it is actually reshaping: the trained model's own probability for the next character after a prompt. Low temperature sharpens the bet onto the top few characters; high temperature flattens it toward an even guess. This is the real version of the synthetic slider at the top of the page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "m10c1b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:09.546803Z",
     "iopub.status.busy": "2026-07-01T09:30:09.546733Z",
     "iopub.status.idle": "2026-07-01T09:30:09.772436Z",
     "shell.execute_reply": "2026-07-01T09:30:09.771992Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after 'To be, or not to ' the model's top next characters (T=1.0):\n",
      "    't'  0.162\n",
      "    'm'  0.113\n",
      "    'h'  0.094\n",
      "    's'  0.080\n",
      "    'b'  0.075\n",
      "    'y'  0.070\n",
      "    'a'  0.045\n",
      "    'p'  0.037\n",
      "    'd'  0.034\n",
      "    'c'  0.031\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAEiCAYAAADauUtBAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAOu9JREFUeJzt3Qm8jHX///HPsS/HUoSU7NllF+nWIlpUSt2iIkmpLKlsyZYK2ZO4cVNI1F3U3SJFkiVCWiSVyL4WQvb5P97f333Nf86Yc5xlzjlz5ryej8dw5pprrut7LXPNZz7f5Yrx+Xw+AwAAACJIlvQuAAAAABCMIBUAAAARhyAVAAAAEYcgFQAAABGHIBUAAAARhyAVAAAAEYcgFQAAABGHIBUAAAARhyAVAAAAEYcgFUCaWLx4scXExNh//vOfTLXHX3vtNbfdq1evtmjbpi1btlhmcs0111jVqlUtEul4DBw4MF2O0QMPPGClSpXyP9c6te4RI0ZYWtB2a32IPgSpCKvly5e7C8bBgwfZs0FmzZplY8aMYb9EqVdffdUFBkh/O3fudNehdevWpXdRMpRjx465/aYflJEmksuG1EOQirAHqYMGDSJIDYEgNboRpEZWkKrrUGYOUu+//377+++/rWTJkkkKBLXfkhoITp482TZu3GipKaGyPfvss25bEX0IUoEUXDSjuRyRsn1IvqNHj7L7MpjTp0/byZMnU7ycrFmzWq5cuVK1Gtw7v7Jnz245c+a09JItWza3rYg+BKkIG1XF9OjRw/1dunRpd3EMbhM1c+ZMq127tuXOndsuvPBCu+eee2zbtm0h231999131rhxY8uTJ4+VK1fO35bxiy++sPr167tlVKhQwT777LNzyqH1/vTTT/bPf/7T8ufPb4UKFbJu3brZ8ePHzyl3Usq0Zs0a+8c//uHK9Mwzz7jX3nvvPbvlllusePHi7kJdtmxZGzx4sJ05cybO+z/88EP7/fff/fvFa8MVX9sxrw1nYOYgoXKcOHHCBgwY4PaVylGiRAnr2bOnm34+4Vjup59+ao0aNbKCBQtabGysOzbeMgKdPXvWXnjhBbv00kvdF8v1119vv/76a5x5vvzyS7v77rvtsssu86+ze/fu52RL1BZO6/rtt9+sWbNmljdvXnccnnvuOfP5fOesV80tqlSp4tZbtGhRe+SRR+zPP/+MM5/ajmpZhQsXdueEzuUHH3wwwf2nY7l+/Xp3bnrHV/s0kPbXk08+aRdddJEr5x133GH79u07Z1kff/yxXX311W6efPnyuXNLyz4f7zxSGR577DErUqSI28dJWa4+c9qnZcqUcfuoWLFibtsPHDhgyeEdnx07dliLFi3c39r+p59+Os7nI7HHR+dhlixZbOHChXHe+/DDD1uOHDns22+/dZ+XunXruunt27f3H4/ENMXQPtI1R/tH1w0tRzUgwX788Ue79tpr3efkkksusZdeeinO6woy+/fv764rBQoUcPtc+/7zzz+PM19g201tu64dOt+1/PjoPNJnQftR5bztttts+/bt58wX6rqS0Lmt+bRMUcbS229eO1fvWG7atMluvvlmt+577703ZJvUQKNHj3bZXK1P+/aHH36I87o+J8GfleBlnq9sodqkKtjXddjbp1qWrkfB1y1Nb968uS1dutTq1avnzj2d/9OnT4/3GCDtZEvDdSHK3Xnnnfbzzz/bm2++6S5MuhCKd3FRYNKvXz8XOD700EPuC3rcuHEuKPrmm29ccOPRF5MuHAoYFaxMmDDB/f3GG2/YE088YZ06dbI2bdrY8OHD7a677nJBpS6agbQeXYCGDBliX331lb388stuuYEXn6SUSV/UN910kyvHfffd575EvS8DXbwVgOj/RYsWuS+ow4cPu/JJ37597dChQ+7LRPtGNG9yhCqHvuD1ZaULrb6wK1WqZN9//71bl47JvHnzUnW5CnZ0vKpXr+4CRH0pKPBctmzZOesZOnSoCzQUqGif6AteX3YrV670z/P222+7TO6jjz7qfmCsWrXKHRftP70WSMHOjTfeaFdeeaVb1vz5810woy8plcWjgEfHSoFL165dbfPmzfbKK6+446xyKhu0d+9ea9q0qTtne/fu7Y6/viDffffdBPedAowuXbq4Y6pjLd754dHrF1xwgSublqn3dO7c2ebMmeOfZ8aMGdauXTsXSAwbNsztA537Cv5VzvgCgUAKUFV+nYNepiuxy9UPDQX82kcKUHVcJ02a5P7XZyg5WTkdH61XPywVjOlH5ciRI13woOOblOOjat3//ve/1qFDB3ce6jP/ySefuOpmBSRXXHGF7dmzxx13bb/OWQWH0rBhwwTLqXUrYFOQ3KdPH3fstW6dT7rWeHQN0fmm652uG/rx3KtXL6tWrZr7/Ig++1OmTLHWrVtbx44d7a+//rJ///vfbj/oXK5Ro0acdU+bNs39gFZ59dnRj+X46DqlH9Yqk7ZJ1xv94Dif853bmq5zQsdEP6C0faLPtEefKW2DzhsdSwXpCdG1Vtv++OOPu+0bO3asXXfdde7YBX8+EpKYsoXaT6+//rr7fnjqqafc9UXfBRs2bLC5c+fGmVfXKs2n80qfk6lTp7ogWT8ydD4gHfmAMBo+fLjSV77NmzfHmb5lyxZf1qxZfS+88EKc6d9//70vW7ZscaY3btzYLWPWrFn+aT/99JObliVLFt9XX33ln/7JJ5+46dOmTfNPGzBggJt22223xVnXY4895qZ/++23yS7TxIkTz9nmY8eOnTPtkUce8eXJk8d3/Phx/7RbbrnFV7JkyXPmVdlD7bPPP//cTdf/5yvHjBkz3L758ssv40zXfJp/2bJlvoSkdLmjR492z/ft2xfvOrztqVSpku/EiRP+6WPHjnXTtd8T2qdDhgzxxcTE+H7//Xf/tHbt2rn3dunSxT/t7Nmzbl/nyJHDXx6VX/O98cYbcZY5f/78ONPnzp3rnn/99de+pKpSpYrbj/Ed3yZNmriyebp37+7Ov4MHD7rnf/31l69gwYK+jh07xnn/7t27fQUKFDhnenzradSoke/06dP+6UlZbqj9/uabb7rlLlmy5LznbDDv+Dz33HNxptesWdNXu3Zt//PEHh/ReaJj+9BDD/n+/PNP3yWXXOKrU6eO79SpU/55dPyCrwsJ0THIly+fr379+r6///47zmuBx8z7nEyfPt0/TedysWLFfC1btvRP0/4PPMdFZS1atKjvwQcf9E/T/tPy8ufP79u7d+95y7lu3To3v65lgdq0aeOm69oX3zFKzLmtz0vwcoKPZe/evUO+Fnht87Yrd+7cvu3bt/unr1y50k3XuR+4T0N9boKXmVDZvGt+8H7SORLo6aefdtMXLVrkn6Z1BJ/fOhY5c+b0PfXUU/HsKaQVqvuRJvRrXVk5ZR7279/vfyhbU758+XOqwZSRUkbPo6pj/fJXJk8ZGY/3t7I/wfTrPTiTJR999FGyyqQMh7I8wVSN5VHWQMtQ9kbZKjU5CLdQ5VB2UfumYsWKcbZFWQsJ3pZwL9fLOKvpg/ZpQrQOVc16vExX4DEM3KfKBmqdyhqpCl/ZrWDKSHqU7dNzVbl6TUG0Hap2veGGG+JshzIlOteCt+ODDz6wU6dOWTgpSxaYidR2K8uoJiBeFlOjYij7FlhGtS3UeZ6YYyjK3Ok9nqQsN3C/K/Ol+ZShlrVr1yZ721XzEUjbHni8E3t8RM1SVOWrTKWyeppPGTO1S0wu7SN9dpVhDG7bGJw9VnlU0+DRuaxq4sDt0b71znF9Hv744w+XhaxTp07I/diyZUt/jVNCvGuXMs2BVLt0PuE6twOz3+ejJh5qDuHRftI5521HavGWr9qtQMqoippeBapcubL/OiQ6FvrOCfW9grRFdT/SxC+//OICDAV/oagqL5Da0gV/OehLTG0Tg6dJcLtCCV6XqhdVzey10UpqmXSxDQyuPKoKVTWkqt1UzRdI1dnhFqoc2hZVY8X3RaeqvtRcbqtWrVzQoCo2fdGrnamq5FSFpn0eSO1MA6kKPPgYbt261VXXvv/+++cc2+B9quWrDVmgyy+/3P0feKz1PrXTTGg71GZOAYOCIDVpUFs5fdGqajWlHUPOt90qo3g/AIKpjWRiqJ1hoKQsV8GUtn327NnnnDPJPZcV9AWfP9r2wOOa2OPjUdt3lVFV5y+++KILMhLjyJEj7hEYTKpsamcpiRkDNdS1Sduj9ryBFDirWYN+qAYGhcHHJ75poegHjc53XcsCKaA6n3Cc2/ohENjO+XxCXVv12XzrrbcsNXn7Se3oAykBoWDd+2EY32cz1DmK9EGQijShbIIu7OqYEJjl8QS3zww1T0LTgzvJhBL8xZLUMgVmmTzKUOniry96tYPTl4e+lJUtUTu182UVQ5XLE9yxJKFyaD1qEzdq1KiQ7wkO7sO9XL13yZIlLuOlLIXa8amtpQKjBQsWxNm/5zuG2m5l1BQwaR8qi6uOJ+p8o3ZiidmnobZDAZDaNIfiBVHezQbU/lJtH9XeUe0UFWxoWnLbESdmu73tUvtRfZkGS2ymMPg4JmW5qlXQMHIKAtVuUtur96sNZnL2e0LbnZzj41GGywu+1b4xsdSOUkGaRx16kjrYfWKuQWozqnNVQaD2pbZN71ObSC8gPt9nL9zCcW4rmA3+0RmOcoW6fsd3/UvqshMjJd8rSF0EqQir+C4KCt70gVfGwMtypTZ9iQVmKNQ4Xl+GXieRcJRJPYnV4UhNB9TZyqNOH4ndN15GLfgGCMG/9hOibVHPZmUwwznkTFKWqy8vzaeHglpluNSJSIFrkyZNEr1OBR3qlKVMVNu2beNUyYaiY6qgJfAY6v0SeKxV9X/VVVclKiBQFbce6lin3t3q2KXMnTLF8UnpfveyYwpokrK/wrVcZY3Ua15BnLLYHi8YTE1JOT463goA9cNQ1dw6z5Sx9zrTJHQsdD6p04/HW5e3j9TzPDj7lhwKBpXd13UhsCzqNJcSCqq1/Qp0A7OnSRmjNKFzO9zDVYU6d/TZDOwAqOtfqGr14OtfUsrm7SetX82VPOpUp+tsUsaORfqiTSrCShmvUAGXvkD0a1VfgMG/TvU8uUPcJGT8+PFxnqt3uHg9cMNRJu8XeOD71RZSA7uH2jehqky9L0hlIgOzCOpVnVjKgCnTqF7OwTRsU3LHy0zscpX1DOb1YE7MEFjn26f6Wz2D46Ne4IHz6rmaayhg9rZD+1Q9wIOpraB3vipQCz4XErsdOr4pudOa2lcq8FLQFarNYKjhqsK53FD7XdLiLmmJPT6iH0DK9urzofnVVlntJNU29XzXIQWOCtS9h4JiUa93jRSgTGfwMHXJyaaF2pfqXb5ixQpLCe/apZFKknqMEnNue731w3XHQI3+oeuHR80ztB+87fCuf2oSEXh+64dx8MggSSmbhsgKtV+8GqHEjIaAyEAmFWGljg6iDJo6PilQuPXWW92F6Pnnn3dDu6h6TdVg+lJQxlHDgahTiYYkCictW8MnqapSXw7esC0apkbCUSZ9QSoToGFL1JlBv/ZVrRrqi037RlXgasyv8RdVvaZ9oyFOlNlQORTsafgZZTb05ZyUu8uonZc6qChzqS9ffenr4q/pqtpTp42kSuxy1dRBQbYu/spSqA2hAnW1XwvMXCWGqvd1bLTv9QWnAOudd96Jt32YmleoeYGOgTplqPmGmhxoTESvmlhNMjTEkYIQ3YVIQYnOTWVa1GlHAbCyccreqtwa5kZlUGcaBegqg/fFFx8dXw2To3NK2ThlLuNrBxqK1qH3a5/XqlXLfX5UfrXP1fZo3wcG4+FeruZTbYCG8VIwqzbKaqoRqlYg3BJ7fNQ+WkPGKZOqz443dJSCLQ295bV11LFT28OJEye6z7SCVp0b8bX91Larnaayifps6jqhz7WCJXWA1HmRFBqOTVlUnUf6TGgfqixqOxvYJjaptJ3qAKdzVD94df1R9jt4nOFQEnNuK7OsMuo6pZoJXYvUTjcxbXVD0edAn3/9iFAgrKBRQ8ppnGWPmhwoeNSPKQ0BpWuH9pWui4Ft/JNSNl3jdT3QDxmvSZYCZO0DXec1xi0yiDQbRwCZxuDBg92wMBq6KHiYmnfeeccNkZM3b173qFixou/xxx/3bdy40T+PhiPRcD7BNFSIhhYKpnVoGcHDkfz444++u+66yw0tc8EFF/g6d+58zvAyKS2TaBimK6+80g23Urx4cV/Pnj39Q2MFDh915MgRN1SMhgPSa4HDq2zatMkNUaRhTzRMzTPPPOP79NNPQw5BFV85Tp486Rs2bJh7XcvRNmuYn0GDBvkOHToU8j3hWu7ChQt9t99+u9t+DQ+k/1u3bu37+eefzxmC6u23346zfG+4msDhgnTstD9iY2N9hQsXdsMkaeiw4Pk0TI2OmfZf06ZN3bBf2n86B86cOXPOtkyaNMmVXcdK50W1atXc8dq5c6d7fe3ata7cl112mdvWIkWK+Jo3b+5bvXq173w0pJPOTy1X5fSG1fGGAgoe+ifUEGPe9GbNmrnhoXLlyuUrW7as74EHHjhvGeJbT1KWq+GC7rjjDneOar67777b7ZvzDW8UH+/4nG/IoMQcHw3rVLduXd+ll17qH7YreBizOXPm+Ke99957vsqVK7vh5BI7HNX777/va9iwoVu/hoWqV6+eG4LrfJ+T4OGSNGzViy++6KbpPNKQWx988EG8QzVp6L7E0jWsa9euvkKFCrl9e+utt/q2bdt23mOU2HN7+fLl7hjocxy4zPiOZajtD9yukSNH+kqUKOHWefXVV/uHAAw0c+ZMX5kyZdw6a9So4a6fwctMqGyhzicNSaZrVOnSpX3Zs2d3ZejTp0+cYQET+l6Jb2gspK0Y/ZPegTIQTrr7iKrwVX3k3VAA0UkZNbX/S0l2CgAQmWiTCgAAgIhDkAoAAICIQ5AKAACAiEObVAAAAEQcMqkAAACIOASpAAAAiDiZbjB/3Spt586dboDncN8CDgAAAPHTyKe6mUTx4sXd7bQTkumCVAWoJUqUSO9iAAAAZFrbtm1zdyVMSKYLUpVB9XaObgcHAACAtKHb3SpZ6MVjCcl0QapXxa8AlSAVAAAg7SWmySUdpwAAABBxCFIBAAAQcQhSAQAAEHEyXZtUAACQeZ05c8ZOnTqV3sWIajly5Djv8FKJQZAKAAAyxficu3fvtoMHD6Z3UaJelixZrHTp0i5YTQmCVAAAEPW8ALVIkSKWJ08ebuiTyjdN2rVrl1122WUp2s8EqQAAIOqr+L0AtVChQuldnKh30UUXuUD19OnTlj179mQvhyA1DZRosSHZ7902r1JYywIAQGbjtUFVBhWpz6vm14+DlASp9O4HAACZQkqqnpH2+5kgFQAAABGHIBUAAAARhzapAAAg00pJv5HU7msSc55q8wEDBtjAgQOTXIa3337b+vXrZ1u2bLHy5cvbsGHD7Oabb453/sWLF9u11157znT14C9WrJilFoJUAACACLRr1y7/33PmzLH+/fvbxo0b/dNiY2OTvMzly5db69atbciQIda8eXObNWuWtWjRwtauXWtVq1ZN8L1ad/78+f3PNVpCaiJIBQAAiEDFArKUBQoUcJnVlGYux44dazfeeKP16NHDPR88eLB9+umn9sorr9jEiRMTfK+C0oIFC1paoU0qAABABhYbG5vgo1OnTv55V6xYYU2aNInz/mbNmrnp51OjRg27+OKL7YYbbrBly5ZZaiOTCgAAkIGtW7cuwdcDq+h1562iRYvGeV3PNT0+CkyVZa1Tp46dOHHCpkyZYtdcc42tXLnSatWqZamFIBUAACADK1euXKouv0KFCu7hadiwoW3atMlGjx5tM2bMSLX1Ut0PAACQSar7ixUrZnv27Inzfj1PalvXevXq2a+//mqpiUwqAABAJqnub9CggS1cuNCeeOIJ/zR1nNL0pK5TzQBSE0EqAABAJqnu79atmzVu3NhGjhxpt9xyi82ePdtWr15tkyZN8s/Tp08f27Fjh02fPt09HzNmjJUuXdqqVKlix48fd21SFy1aZAsWLLDURJAKAACQSTRs2NCNjfrss8/aM8884wbznzdvXpwxUjU+69atW/3PT548aU899ZQLXPPkyWPVq1e3zz77LOQA/+EU4/P5fJaJHD582I01dujQoTjp70i9m0VS7kwBAADOpezf5s2bXTYwV65c7KJ03N9JicPoOAUAAICIQ5AKAACAiEOQCgAAgIhDkAoAAICIQ5AKAACAiEOQCgAAgIhDkAoAAICIQ5AKAACAiEOQCgAAgIgTEUHq+PHjrVSpUu6uBPXr17dVq1bFO++7775rderUsYIFC1revHmtRo0aNmPGjDQtLwAAAFJXNktnc+bMsSeffNImTpzoAtQxY8ZYs2bNbOPGjVakSJFz5r/wwgutb9++VrFiRcuRI4d98MEH1r59ezev3gcAAJBY992X/FuXJ8fMmYm/3XlMTEyCrw8YMMAGDhyYpPWvX7/e+vfvb2vWrLHff//dRo8ebU888cR53/fdd9/Z448/bl9//bVddNFF1qVLF+vZs6dFdSZ11KhR1rFjRxdoVq5c2QWrefLksalTp4ac/5prrrE77rjDKlWqZGXLlrVu3bpZ9erVbenSpWledgAAgNSya9cu/0NJPN3rPnDa008/neRlHjt2zMqUKWNDhw61YsWKJeo9hw8ftqZNm1rJkiVdcDt8+HAXHE+aNMmiNpN68uRJt7F9+vTxT8uSJYs1adLEVqxYcd73+3w+W7Rokcu6Dhs2LJVLCwAAkHaKBQSRBQoUcJnVxAaW8albt657SO/evRP1njfeeMPFbEogqha7SpUqtm7dOpdofPjhhy0qM6n79++3M2fOWNGiReNM1/Pdu3fH+75Dhw5ZbGys21G33HKLjRs3zm644YaQ8544ccL9Agh8AAAARIvY2NgEH506dUrR8pU4/Mc//uHiLo/XNPPPP/+0qG2Tmhz58uVzEfyRI0ds4cKFrk2rUtdqChBsyJAhNmjQoHQpJwAAQGpbt25dgq+rmUBKKHFYunTpONO8BKNeu+CCCyzqgtTChQtb1qxZbc+ePXGm63lC6Ww1CShXrpz7W737N2zY4ILRUEGqmhIoiPUok1qiRImwbgcAAEB6Kfe/mCjapGt1v9LGtWvXdtlQz9mzZ93zBg0aJHo5eo+q9UPJmTOn+wUR+AAAAIgWsalc3a/EYaiEovda1Fb3K8vZrl07N/ZpvXr1XO+1o0ePut7+0rZtW7vkkktcplT0v+ZVz34Fph999JEbJ3XChAnpvCUAAADRV93foEEDN/znqVOnLHv27G7ap59+ahUqVEi1qv6ICFJbtWpl+/btc2N2qV2Dqu/nz5/vb+uwdetWV73vUQD72GOP2fbt2y137txuvNSZM2e65QAAAGQ25ZJQ3a9e+j/++KP/7x07drggVxlXbzmvvPKKzZ0711/T3aZNG9e/p0OHDtarVy/74YcfbOzYsW6M1dSU7kGqdO7c2T1CWbx4cZznzz//vHsAAAAgaXbu3Gk1a9b0Px8xYoR7NG7c2B9zafSlTZs2xRn+asGCBW4wfzXTVJ8iJRdTc/gpifFpsNFMRB2ntLM1jFVatU8t0SL5d7PYNi/xd6YAAADnOn78uG3evNn1UNct2JF++zspcVi633EKAAAACEaQCgAAgIhDkAoAAICIQ5AKAACAiEOQCgAAgIhDkAoAAICIQ5AKAACAiEOQCgAAgIhDkAoAAICIQ5AKAACAiJMtvQsAAACQXpYuPZCm62vUqFCi542JiUnw9QEDBtjAgQOTtP7169db//79bc2aNfb777/b6NGj7YknnkjwPVu2bHG3OA22YsUKu/LKKy21EKQCAABEoF27dvn/njNnjgsuN27c6J8WGxub5GUeO3bMypQpY3fffbd17949Se/97LPPrEqVKv7nhQolPuBODoJUAACACFSsWDH/3wUKFHCZ1cBpyVG3bl33kN69eyfpvQpKU7r+pKBNKgAAQAYWGxub4KNTp05hWc9tt91mRYoUsUaNGtn7779vqY1MKgAAQAa2bt26BF/Pnz9/ipavQHfkyJF21VVXWZYsWeydd96xFi1a2Lx581zgmloIUgEAADKwcuXKperyCxcubE8++aT/uZoL7Ny504YPH56qQSrV/QAAABlYbBpV9weqX7++/frrr5aayKQCAABkYOtSubo/vnVefPHFlpoIUgEAADJJdf/Jkyftxx9/9P+9Y8cOF3Aq4+ot55VXXrG5c+fawoUL3fPXX3/dcuTIYTVr1nTP3333XZs6dapNmTLFUhNBKgAAQCaxc+dOf7ApI0aMcI/GjRvb4sWL3bT9+/fbpk2b4rxv8ODBbvD/bNmyWcWKFd24rXfddVeqljXG5/P5LBM5fPiwG2vs0KFDqZL+DqVEiw3Jfu+2eZXCWhYAADKb48eP2+bNm91dk3LlypXexcnU+/twEuIwOk4BAAAg4iQrSD169Gj4SwIAAACkJEgtWrSoPfjgg7Z06dLkvB0AAAAIf5A6c+ZM++OPP+y6666zyy+/3IYOHeoa4gIAAADpFqR6t8LSsAUaIHbWrFlWsmRJa968uRuW4PTp02EpHAAAADKnFHWcuuiii9xtsr777jsbNWqUffbZZ244guLFi1v//v3t2LFj4SspAABACpw9e5b9lwbCNXBUisZJ3bNnjxvg9bXXXnNjZylA7dChg23fvt2GDRtmX331lS1YsCAsBQUAAEgODUSfJUsW1zRRCTY9j4mJYWemUoC6b98+t3+zZ8+e9kGqqvSnTZtmn3zyiVWuXNkee+wxu++++6xgwYL+eRo2bGiVKjHGJwAASF8KUDVm565du+hDkwYUoF566aWWNWvWtA9S27dvb/fcc48tW7bM6tatG3IeVfn37ds3RYUDAAAIB2VPL7vsMtdv5syZM+zUVKQMakoD1GQHqfolkidPngTnyZ07tw0YMCC55QIAAAgrrwo6pdXQiOCOU/ny5bO9e/eeM/3AgQNhiZwBAACQuWUJZ6+tEydOuHQ6AAAAkGbV/S+//LI/XT5lyhSLjY31v6b2HUuWLLGKFSumqEAAAABAkoLU0aNH+zOpEydOjFO1rwxqqVKl3HQAAAAgzYLUzZs3u/+vvfZaNwzVBRdckKKVAwAAAGHr3f/5558n520AAABAeINU3f508ODBljdvXvd3QnSLVAAAACDVg9RvvvnGTp065f87PtxmDAAAAGkWpAZW8VPdDwAAgIgbJxUAAACIiEzqnXfemeiFquc/AAAAkOpBaoECBZK9EgAAACBVgtRp06YlacEAAABAhm6TOn78eHe3qly5cln9+vVt1apV8c47efJku/rqq92NBPRo0qRJgvMDAAAgijOptWrVsoULF7rAsGbNmgkONbV27dpEF2DOnDlu3FXdTlUB6pgxY6xZs2a2ceNGK1KkyDnzL1682Fq3bm0NGzZ0Qe2wYcOsadOmtn79ervkkksSvV4AAABEQZB6++23W86cOd3fLVq0CFsBNPB/x44drX379u65gtUPP/zQpk6dar179z5n/jfeeCPO8ylTptg777zjAui2bduGrVwAAADIAEHqgAEDQv6dEidPnrQ1a9ZYnz59/NOyZMniqvBXrFiRqGUcO3bM3WTgwgsvDPn6iRMn3MNz+PDhMJQcAAAAERGkhrJ69WrbsGGD+7ty5cpWu3btJL1///79dubMGStatGic6Xr+008/JWoZvXr1suLFi7vANpQhQ4bYoEGDklQuAAAAZMAgdfv27a5d6LJly6xgwYJu2sGDB1070dmzZ9ull15qaWHo0KFufWqnqvapoShLqzavgZnUEiVKpEn5AAAAkIa9+x966CFXxa4s6h9//OEe+vvs2bPutcQqXLiwZc2a1fbs2RNnup4XK1YswfeOGDHCBakLFiyw6tWrxzuf2tHmz58/zgMAAABRGKR+8cUXNmHCBKtQoYJ/mv4eN26cLVmyJNHLyZEjh2sioE5PHgW6et6gQYN43/fSSy/Z4MGDbf78+VanTp3kbAIAAACirbpf1eXKpAZT+1K1D00KVcW3a9fOBZv16tVzQ1AdPXrU39tfPfY1tJTaloqGnOrfv7/NmjXLja26e/duNz02NtY9AAAAkEkzqcOHD7cuXbq4jlMe/d2tWzdXDZ8UrVq1cu9R4FmjRg1bt26dy5B6nam2bt1qu3bt8s+vDK5GBbjrrrvs4osv9j+Sul4AAABErhifz+dLzIwaxD9wAH9lO0+fPm3Zsv1fMtb7O2/evK6NaqRSx6kCBQrYoUOH0qx9aokW/zcCQnJsm1cprGUBAADICHFYoqv7VQ0PAAAApIVEB6lqNwoAAABE/GD+cvz4cddGNBDDPAEAACDNO06pPWrnzp2tSJEirg2q2qsGPgAAAIA0D1J79uxpixYtcj3tNVj+lClT3K1HNfzU9OnTU1QgAAAAIFnV/f/9739dMHrNNde48UyvvvpqK1eunJUsWdLeeOMNu/fee9mzAAAASNtMqoaYKlOmjL/9qTfkVKNGjZJ0xykAAAAgbEGqAtTNmze7vytWrGhvvfWWP8NasGDB5CwSAAAASFmQqir+b7/91v3du3dvGz9+vOXKlcu6d+9uPXr0SM4iAQAAgJS1SVUw6mnSpIlt2LDB1q5d69qlVq9ePTmLRAbHXbUAAEBEjZMqpUqVcg8AAAAg3ar7ZeHChda8eXMrW7ase+jvzz77LCyFAgAAQOaWrCD11VdftRtvvNHy5ctn3bp1cw/18r/55ptd+1QAAAAgzav7X3zxRRs9erS765Sna9eudtVVV7nXHn/88RQVCgAAAJlbsoLUgwcPukxqsKZNm1qvXr3CUS78z333bUjRvpg5sxL7EgAAZI7q/ttuu83mzp17zvT33nvPtU0FAAAA0iST+vLLL/v/rly5sr3wwgu2ePFia9CggZv21Vdf2bJly+ypp55KUYEAAACAGJ/P50vMbihdunSi9lZMTIz99ttvEbtnDx8+bAUKFLBDhw65zl6RPoZo41jLENX9jJMKAADCGYclOpPq3QYVAAAAiNhxUj1KxCYyGQsAAACkbpA6ffp0q1atmuXOnds9dDvUGTNmJHdxAAAAQMqGoBo1apT169fPjZOqsVFl6dKl1qlTJ9u/f7917949OYsFAAAAkh+kjhs3ziZMmGBt27aNMyxVlSpVbODAgQSpAAAASPvq/l27dlnDhg3Pma5peg0AAABI8yC1XLly9tZbb50zfc6cOVa+fPkUFQgAAABIVnX/oEGDrFWrVrZkyRJ/m1QN5L9w4cKQwSsAAACQ6pnUli1b2qpVq6xw4cI2b94899DfmnbHHXckZ5EAAABA8jOpp06dskceecT17p85c2ZS3w4AAACEP5OaPXt2e+edd5L6NgAAACB1q/tbtGjhqvgBAACAiOk4pR78zz33nOssVbt2bcubN2+c17t27Rqu8gEAACATSlaQ+u9//9sKFixoa9ascY9AMTExBKkAAABI+yB18+bN/r99Pp8/OAUAAADSrU2ql02tWrWq5cqVyz3095QpU8JSKAAAAGRuycqk9u/f30aNGmVdunSxBg0auGkrVqyw7t2729atW117VQAAACBNg9QJEybY5MmTrXXr1v5pt912m1WvXt0FrgSpAAAASPPqfg3oX6dOnXOmq6f/6dOnU1QgAAAAIFlB6v333++yqcEmTZpk9957L3sVAAAAaV/d73WcWrBggV155ZXu+cqVK1171LZt29qTTz7pn09tV4GMrkSLDcl+77Z5lcJaFgAAMoNkBak//PCD1apVy/29adMm93/hwoXdQ695GJYKAAAAaRakfv7558laGQAAAJCq1f1AuNx3X/Kr0mfOpCodAIBolOzB/AEAAIDUQpAKAACAiJPuQer48eOtVKlS7taq9evXt1WrVsU77/r1661ly5ZufnXKGjNmTJqWFQAAAJkgSJ0zZ44brmrAgAG2du1au+KKK6xZs2a2d+/ekPMfO3bMypQpY0OHDrVixYqleXkBAACQCYJUjaHasWNHa9++vVWuXNkmTpxoefLksalTp4acv27dujZ8+HC75557LGfOnGleXgAAAER57/6TJ0/amjVrrE+fPv5pWbJksSZNmtiKFSvCtp4TJ064h+fw4cOWmSxdeiDZ723UqFBYywIAABDxmdT9+/fbmTNnrGjRonGm6/nu3bvDtp4hQ4ZYgQIF/I8SJUqEbdkAAACI0o5TqU2Z2kOHDvkf27ZtS+8iAQAAIFKr+3UL1axZs9qePXviTNfzcHaKUttV2q8CAABkLOmWSc2RI4fVrl3bFi5c6J929uxZ97xBgwbpVSwAAABk9tuiavipdu3aWZ06daxevXpu3NOjR4+63v7Stm1bu+SSS1y7Uq+z1Y8//uj/e8eOHbZu3TqLjY21cuXKpeemAKly21fh1q8AgMwoXYPUVq1a2b59+6x///6us1SNGjVs/vz5/s5UW7dudT3+PTt37rSaNWv6n48YMcI9GjdubIsXL06XbQAAAECUBanSuXNn9wglOPDUnaZ8Pl8alQwAAADpJep79wMAACDjSfdMKoDIUKJF8tvObptXKaxlAQCATCoAAAAiDkEqAAAAIg7V/cjQli49kOz3NmpUKKxlycxSMswWQ2wBAEIhkwoAAICIQ5AKAACAiEN1P4BMISWjFzSOTdm6adIAAElHJhUAAAARhyAVAAAAEYfqfiDCMYIB0gIjNACINGRSAQAAEHHIpAJAlEjPzmEAEG5kUgEAABBxyKQCSFe0uQUAhEKQCgCpjEAcAJKOIBUAkCIE4QBSA21SAQAAEHEIUgEAABBxCFIBAAAQcQhSAQAAEHEIUgEAABBxCFIBAAAQcQhSAQAAEHEYJxUAEPXuu29Dit4/c2alsJUFQOIQpAIAcB7csABIe1T3AwAAIOIQpAIAACDiUN0PAEAmb3fbqVORZL+3UaNCyX4vkBCCVAAAEPVoV5zxEKQCAICoH6EhJdniaA/CIzUjTptUAAAARBwyqQAAAJm4XXGkIkgFAGQIJVok/wu8cWxYiwIgDVDdDwAAgIhDkAoAAICIQ5AKAACAiEOQCgAAgIhDkAoAAICIQ+9+AAAiBCMYAP8fQSoAAEgTBOFICqr7AQAAEHHIpAIAAIQJ2eLwIZMKAACAiBMRQer48eOtVKlSlitXLqtfv76tWrUqwfnffvttq1ixopu/WrVq9tFHH6VZWQEAAJAJgtQ5c+bYk08+aQMGDLC1a9faFVdcYc2aNbO9e/eGnH/58uXWunVr69Chg33zzTfWokUL9/jhhx/SvOwAAACI0iB11KhR1rFjR2vfvr1VrlzZJk6caHny5LGpU6eGnH/s2LF24403Wo8ePaxSpUo2ePBgq1Wrlr3yyitpXnYAAABEYZB68uRJW7NmjTVp0uT/FyhLFvd8xYoVId+j6YHzizKv8c0PAACAjCdde/fv37/fzpw5Y0WLFo0zXc9/+umnkO/ZvXt3yPk1PZQTJ064h+fQoUPu/8OHD1taOXvqSLLfe+pUytZ99GjuZL/38OHsEb+N0b59mWEbo337MsM2Rvv2ZYZtjPbtywzbeDQF25fUbUzZev4v/vL5fOef2ZeOduzYoRL6li9fHmd6jx49fPXq1Qv5nuzZs/tmzZoVZ9r48eN9RYoUCTn/gAED3Dp4sA84BzgHOAc4BzgHOAc4Bywi9sG2bdvOGyemaya1cOHCljVrVtuzZ0+c6XperFixkO/R9KTM36dPH9cxy3P27Fn7448/rFChQhYTE2MZmX6NlChRwrZt22b58+e3aBPt25cZtjHat0/YxoyPY5jxcQwzDmVQ//rrLytevPh5503XIDVHjhxWu3ZtW7hwoeuh7wWRet65c+eQ72nQoIF7/YknnvBP+/TTT930UHLmzOkegQoWLGjRRF/+0RoAZIbtywzbGO3bJ2xjxscxzPg4hhlDgQIFMsYdp5TlbNeundWpU8fq1atnY8aMsaNHj7re/tK2bVu75JJLbMiQIe55t27drHHjxjZy5Ei75ZZbbPbs2bZ69WqbNGlSOm8JAAAAwiXdg9RWrVrZvn37rH///q7zU40aNWz+/Pn+zlFbt251Pf49DRs2tFmzZtmzzz5rzzzzjJUvX97mzZtnVatWTcetAAAAQFQFqaKq/fiq9xcvXnzOtLvvvts9Mjs1Y9BNEIKbM0SLaN++zLCN0b59wjZmfBzDjI9jGJ1i1HsqvQsBAAAARNQdpwAAAIBgBKkAAACIOASpQDq45ppr4gyjhoyHYwhEJj6b0YMgNQPjgwgAAKIVQSoAAAAiDkFqBvXAAw/YF198YWPHjnW3d9Vjy5YtltEzw126dHHV4BdccIEbK3fy5Mn+mzvky5fPypUrZx9//LFFA91drWfPnnbhhRe62/oOHDjQos1//vMfq1atmuXOndvdirhJkybueEaL06dPu+HzdPcU3ea5X79+7pZ/0WD69OnumJ04cSLOdN0d8P7777dooXG5GzVq5O5EqO1t3ry5bdq0yaKBrqneEI/ReI6Krie66U9sbKxdfPHF7kY/0ejs2bP20ksvue9ADbd12WWX2QsvvGDRjiA1g1JwqlvBduzY0Xbt2uUeukd6Rvf666+7C+mqVatcwProo4+6MXF1E4e1a9da06ZN3RfksWPHLBq2NW/evLZy5Up38XnuuefcLX6jhc7J1q1b24MPPmgbNmxwYx7feeedUfUFqWOYLVs2d77qMzlq1CibMmWKRQN97s6cOWPvv/++f9revXvtww8/dMc0moIc3flQdy7ULbd185g77rjDBQXRIJrPUenRo4dL2Lz33nu2YMECd53Rd0W06dOnjw0dOtT9yPjxxx/dTY28mx5FNY2TioypcePGvm7duvmiaXsaNWrkf3769Glf3rx5fffff79/2q5duxTh+FasWOGLpm2VunXr+nr16uWLFmvWrHHHasuWLb5opGNYqVIl39mzZ/3TdPw0LVo8+uijvptuusn/fOTIkb4yZcrE2eZos2/fPnfefv/9976MLtrP0b/++suXI0cO31tvveWfduDAAV/u3Lmj6rvx8OHDvpw5c/omT57sy2zIpCKiVK9e3f931qxZXfWbqos93i9HZXSiaVtFVVXRsF2eK664wq6//np3/JSVU9ONP//8M72LFVZXXnmla2rjUe3GL7/84jKQ0UA1NcpO7dixwz1/7bXXXFOjwG3O6HS8lPEvU6aM5c+f30qVKuW/JXc0iOZzVM0yTp48afXr1/dPU/OpChUqWDTZsGGDa3aj62lmQ5CKiJI9e/Y4z3VxDZzmXWyjoSou1LZGw3YF/shQ8wW1Ia5cubKNGzfOfXls3rw5vYuGRKpZs6b7saH2qWvWrLH169e7IDWa3HrrrfbHH3+4H1FqeqOHKPgBIkHu3LktsyJIzcBy5MgRFb+GEb0UeF911VU2aNAg++abb9w5O3fuXIsWXkDj+eqrr6x8+fIuQI8WDz30kMugTps2zXV8i4a2754DBw7Yxo0b7dlnn3VZqkqVKkVdtj+az9GyZcu6H/uB26jj9/PPP1s0KV++vAtU1WY6s8mW3gVA8qlaSh9O9epXz0ZVc6jRPxAJdG7qoqrObkWKFHHP9+3b5wKBaKEqYXW6eeSRR1xnDWWLo613cZs2bezpp592mUZlVKOJRhFRk6JJkya55jY6nr1797ZoEs3nqL73OnTo4DpP6TjqOtO3b9+o+x7MlSuX9erVy40Gox/6+uGva6lqNrT90YwgNQPTF0e7du1cVerff//tqlG99lRAelP7viVLltiYMWPs8OHDVrJkSffleNNNN1m00NA3+uzVq1fPZaa6detmDz/8sEUTDV3UsmVL16tfw09FEwUzs2fPtq5du1rVqlVdc5SXX37ZDd0ULaL9HB0+fLgdOXLENdvQMIVPPfWUHTp0yKJNv3793CgN/fv3t507d7ofVZ06dbJoF6PeU+ldCABA5FJVeJUqVVwAh4xDwXaNGjXcD0UgIyKTCgAISe37NO6kHq+++ip7CUCaIkgFAMTbu1+B6rBhw6JuWB8AkY/qfgAAAESc6OoCBwAAgKhAkAoAAICIQ5AKAACAiEOQCgAAgIhDkAoAAICIQ5AKABEoJibG5s2bl97FAIB0Q5AKAGE0cOBAd5efaPbAAw9E3S1SAUQeglQAyCROnjxpkSTSygMgshCkAshU9zLv2rWr9ezZ0y688EIrVqyYy3wGOnjwoD300EN20UUXWf78+e26666zb7/91r22b98+954XX3zRP//y5cstR44ctnDhQnvttdds0KBBbn5V1+uhafGZOnWqValSxXLmzGkXX3yxde7cOc7r+/fvtzvuuMPy5Mlj5cuXt/fff9//2pkzZ6xDhw5WunRpy507t7sj1NixY0NmPF944QUrXry4/65RM2bMsDp16li+fPnc9rRp08b27t0b573r16+35s2bu32g+a6++mrbtGmT21+vv/66vffee/5t1G1TZdu2bfbPf/7TChYs6Pbv7bffblu2bDlveQAgFG6LCiBTUYD15JNP2sqVK23FihUucLrqqqvshhtucK/ffffdLuj7+OOPrUCBAvavf/3Lrr/+evv5559d4KrAUoFW06ZNXZB1//33u+BS8/z999/2ww8/2Pz58+2zzz5zy9MyQpkwYYIrx9ChQ+2mm26yQ4cO2bJly+LMo4D3pZdesuHDh9u4cePs3nvvtd9//90FgGfPnrVLL73U3n77bStUqJALlh9++GEX7CpQ9Ch4VqD56aef+qedOnXKBg8e7Mqv4FTl0H746KOP3Os7duywf/zjHy6oX7RokXu/ynb69Gl7+umnbcOGDXb48GGbNm2am1/l0TKbNWtmDRo0sC+//NKyZctmzz//vN1444323XffuUA+vvIAQEg+AMgkGjdu7GvUqFGcaXXr1vX16tXL/f3ll1/68ufP7zt+/HicecqWLev717/+5X/+2GOP+S6//HJfmzZtfNWqVYsz/4ABA3xXXHHFectSvHhxX9++feN9XZfnZ5991v/8yJEjbtrHH38c73sef/xxX8uWLf3P27Vr5ytatKjvxIkTCZbl66+/dsv+66+/3PM+ffr4Spcu7Tt58mTI+bXc22+/Pc60GTNm+CpUqOA7e/asf5rWmzt3bt8nn3ySpPIAgJBJBZCpVK9ePc5zZR69qm5V0x85csRlJgMpQ6qqbs+IESOsatWqLou5Zs0aV12fFFrfzp07XfY1sWXNmzevy0AGVsuPHz/eZXa3bt3qyqg2nsGdtqpVq+bPYnpUZlXba3v//PNPl5UVLady5cq2bt06V72fPXv2RG+TlvXrr7+6pgGBjh8/HmffhSoPAIRCkAogUwkOvNSm0gvSFKAqaPXaWAZSO0uPgi4FmXqf2lwq8EoKNSdIaVlnz57tqt5HjhzpqtgVHKpZgJoxBFJwG+jo0aOuWl6PN954wzVhUHCq515HpsSWL5D2Xe3atd0yg2kd8ZUHAOJDkAoA/1OrVi3bvXu3a09ZqlSpkPtFgdx9991nrVq1cm061cnq+++/tyJFirjXlSVUp6aEKKDU8tU+89prr03W/lcb0YYNG9pjjz3mnxaYsYzPTz/9ZAcOHHBtYUuUKOGmrV69+pwMrtruqp1pqGxqqG3UvpszZ47bD8r4AkBK0bsfAP6nSZMmLiupjlELFixwWVJ1SOrbt68/kNPf6uT08ssvW69evezyyy+3Bx980L8PFXxu3rzZVZmrd/6JEydC7l9VtysLquX88ssvtnbtWtc5KrHU219l+uSTT1ynrn79+tnXX3993vdddtllLsjUun777Tc3YoA6UQVSRzB1jLrnnnvcOlQ+jQiwceNG/zaqM5SeaxsVzKpTV+HChV2PfnWc0j5QRlqjKWzfvp1zDECSEaQCQEB1unq4q2d7+/btXQCqQE096osWLeqCrjFjxriATdnCLFmyuL8VlKm3vrRs2dL1aFeGVNXcb775Zsj9265dO7esV1991Q1DpeGeFAwm1iOPPGJ33nmny+jWr1/fZUcDs6rxUZk0LJba06r9qTKqamMbSG1y1atfVfiNGzd21fiTJ0/2Z1U7duzossgaxkrLU1ZXw2QtWbLEBcEqV6VKldwQWWqTSmYVQHLEqPdUst4JAAAApBIyqQAAAIg4BKkAAACIOASpAAAAiDgEqQAAAIg4BKkAAACIOASpAAAAiDgEqQAAAIg4BKkAAACIOASpAAAAiDgEqQAAAIg4BKkAAACIOASpAAAAsEjz/wDvjFj+VX0Z2gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 700x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# The model's actual next-char probabilities after a prompt, at three temperatures.\n",
    "prompt = \"To be, or not to \"\n",
    "idx = torch.tensor([[stoi[c] for c in prompt]], device=device)\n",
    "with torch.no_grad():\n",
    "    logits, _ = model(idx[:, -cfg.block_size:])\n",
    "logits = logits[0, -1]                               # one score per possible next char\n",
    "\n",
    "topp, topi = F.softmax(logits, dim=-1).topk(10)      # the 10 likeliest, at T=1.0\n",
    "labels = [\"\\\\n\" if itos[i.item()] == \"\\n\" else itos[i.item()] for i in topi]\n",
    "print(f\"after {prompt!r} the model's top next characters (T=1.0):\")\n",
    "for c, p in zip(labels, topp):\n",
    "    print(f\"   {c!r:>4}  {p.item():.3f}\")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 3)); width = 0.27\n",
    "for k, (T, color) in enumerate(zip([0.5, 1.0, 1.5], [\"#1c4fd6\", \"#5b5bd6\", \"#c3c3ec\"])):\n",
    "    pr = F.softmax(logits / T, dim=-1)[topi]\n",
    "    ax.bar([j + k * width for j in range(10)], pr.tolist(), width, label=f\"T={T}\", color=color)\n",
    "ax.set_xticks([j + width for j in range(10)]); ax.set_xticklabels(labels)\n",
    "ax.set_xlabel(\"next character\"); ax.set_ylabel(\"probability\"); ax.legend()\n",
    "ax.set_title(\"temperature reshapes the real next-char distribution\")\n",
    "plt.tight_layout(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m10c1c",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>logits, _ = model(idx[:, -cfg.block_size:])</code>: run the trained model on the prompt; <code>logits[0, -1]</code> is its score for every possible next character.</li>\n",
    "<li><code>F.softmax(...).topk(10)</code>: the ten most likely next characters at temperature 1.0, printed with their probabilities.</li>\n",
    "<li>The loop over <code>T = 0.5, 1.0, 1.5</code> divides the logits by T before softmax, then plots each resulting distribution side by side.</li>\n",
    "<li>At <code>T = 0.5</code> the leading character shoots up past 0.35 (a sharp, safe bet); at <code>T = 1.5</code> it drops and the long tail rises (a flatter, riskier guess).</li>\n",
    "<li>Sampling then just draws one character from whichever curve you picked -- this is the real distribution the slider at the top of the page stood in for.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7be4a90",
   "metadata": {},
   "source": [
    "### Top-k and top-p\n",
    "\n",
    "Both methods trim away the unlikely tail, so the model makes fewer bizarre character choices, while still sampling rather than always taking the top option (so the output is less repetitive than greedy). These are what real chat models use in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "53e3dded",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:09.773664Z",
     "iopub.status.busy": "2026-07-01T09:30:09.773575Z",
     "iopub.status.idle": "2026-07-01T09:30:12.094514Z",
     "shell.execute_reply": "2026-07-01T09:30:12.094098Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== top-k = 10 ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KING:\n",
      "On last, thy short and to the wake, hastings for the world,\n",
      "Yet what in my happy safety! how not intence\n",
      "To his fell against affect him; to your great are\n",
      "All: this times holy and\n",
      "\n",
      "=== top-p = 0.9 (nucleus) ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KING:\n",
      "On last, thou hast and to the water of the mark,\n",
      "When the pleasure high men a guilty hast\n",
      "Hence must not loved leave against to as he\n",
      "me to requeen the arborn. A word, fair love i\n"
     ]
    }
   ],
   "source": [
    "print(\"=== top-k = 10 ===\")\n",
    "print(generate(model, prompt=\"KING:\", max_new_tokens=180, top_k=10, seed=2))\n",
    "print(\"\\n=== top-p = 0.9 (nucleus) ===\")\n",
    "print(generate(model, prompt=\"KING:\", max_new_tokens=180, top_p=0.9, seed=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50b9f4ac",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>generate(..., top_k=10, ...)</code>: sample only among the 10 most likely next characters: the wild tail is gone, but variety remains.</li>\n",
    "<li><code>generate(..., top_p=0.9, ...)</code>: the adaptive version: keep the smallest set of characters whose probabilities add up to 90% -- sometimes 3 candidates, sometimes 20, depending on how confident the model is.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1ed2d7b",
   "metadata": {},
   "source": [
    "## Beyond: from this to a frontier model\n",
    "\n",
    "Your model is a real GPT. A frontier model differs in degree and engineering, not in the fundamental ideas. Here is an honest map of the gap.\n",
    "\n",
    "### 1. Tokenization: subwords, not characters\n",
    "Real models use **BPE** (notebook 02): a vocabulary of roughly 50,000 to 200,000 subword tokens instead of 65 characters, so sequences are far shorter and each token carries more meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "27a62e95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:30:12.095707Z",
     "iopub.status.busy": "2026-07-01T09:30:12.095627Z",
     "iopub.status.idle": "2026-07-01T09:30:12.149995Z",
     "shell.execute_reply": "2026-07-01T09:30:12.149618Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "our char tokens : 44\n",
      "BPE tokens      : 10 -> ['The', ' quick', ' brown', ' fox', ' jumps', ' over', ' the', ' lazy', ' dog', '.']\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "enc = tiktoken.get_encoding(\"cl100k_base\")    # the tokenizer family used by GPT-3.5/4\n",
    "s = \"The quick brown fox jumps over the lazy dog.\"\n",
    "print(\"our char tokens :\", len(s))\n",
    "print(\"BPE tokens      :\", len(enc.encode(s)), \"->\", [enc.decode([t]) for t in enc.encode(s)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "871a6e83",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>enc = tiktoken.get_encoding(\"cl100k_base\")</code>: the tokenizer family GPT-3.5/4 actually use (~100k subword vocabulary).</li>\n",
    "<li><code>len(s)</code> vs <code>len(enc.encode(s))</code>: the same sentence is 44 characters but only 10 subword tokens: why real models see \"words,\" not letters.</li>\n",
    "<li><code>[enc.decode([t]) for t in enc.encode(s)]</code>: decode each token id on its own to reveal the chunk it stands for.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93e607c5",
   "metadata": {},
   "source": [
    "### 2. Scale (the big one)\n",
    "The same design, turned up enormously. Roughly:\n",
    "\n",
    "| | this notebook | GPT-2 | GPT-3 | frontier (2025-26) |\n",
    "|---|---|---|---|---|\n",
    "| parameters | ~0.8 M | 1.5 B | 175 B | ~10^12+ |\n",
    "| training tokens | ~10^6 | ~10^10 | 3x10^11 | ~10^13+ |\n",
    "| context | 128 | 1024 | 2048 | 10^5-10^6 |\n",
    "\n",
    "**Scaling laws** are the finding that the loss falls in a predictable way as you increase the model size, the amount of training data, and the computing power together. (One well-known result, named Chinchilla after the study that found it, is that for a given amount of computing power there is a best balance between model size and data, roughly 20 tokens of training text for every number in the model.) There is no new trick here; there is simply more of everything.\n",
    "\n",
    "### 3. Inference speed: the KV-cache\n",
    "Generating text in the naive way re-runs attention over the whole context for every new token, so the work grows with the square of the length (this is what \"order T-squared\" means: double the length and the work roughly quadruples). Since the keys and values of past tokens do not change, real systems **cache** them and compute only the new token's, which brings the work back down to roughly proportional to the length. This is the single most important speed-up when running a model. (Our `generate` recomputes everything each time, for clarity.)\n",
    "\n",
    "### 4. Modern architecture tweaks\n",
    "The 2017 Transformer you built, brought up to date in the style of models like Llama and GPT. Each of these is an optional refinement; the names are given mainly so you can look them up later.\n",
    "- **RoPE** (rotary position embeddings) replaces the learned table of positions with one based on relative distance, which copes better with longer text.\n",
    "- **RMSNorm** replaces LayerNorm with a simpler, cheaper version that has the same steadying effect.\n",
    "- **SwiGLU** replaces the plain feed-forward network with a slightly more capable \"gated\" version.\n",
    "- **Grouped-query attention** lets several heads share their keys and values, which makes running the model much cheaper.\n",
    "- **Flash-Attention** computes exactly the same attention without ever building the full table of scores in memory, saving both memory and time.\n",
    "\n",
    "### 5. Alignment: from text-predictor to assistant\n",
    "Your model has done only the first of three stages.\n",
    "1. **Pretraining**: predict the next token on a huge body of text (what you did). The result is a *base model* that continues text but does not reliably follow instructions.\n",
    "2. **Supervised fine-tuning** (SFT): continue training on curated instruction-and-response examples.\n",
    "3. **Preference tuning** (often called RLHF or DPO): nudge the model towards the answers people prefer.\n",
    "\n",
    "Stages 2 and 3 are what turn a base model into an assistant like ChatGPT or Claude, and they reuse the exact training loop from notebook 09, only with different data and a different goal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "360b1580",
   "metadata": {},
   "source": [
    "## You built an LLM\n",
    "\n",
    "From start to finish, by hand:\n",
    "\n",
    "- the **maths** (matrix multiply, softmax, cross-entropy, gradients) in notebook 01,\n",
    "- **tokenization** and a first model with **by-hand backpropagation** in notebooks 02 and 03,\n",
    "- **autograd** from scratch in notebook 04,\n",
    "- **attention** and the **Transformer block** in notebooks 05 and 06,\n",
    "- the same ideas in **PyTorch**, assembled into a **GPT**, in notebooks 07 and 08,\n",
    "- **training** it on a GPU until it wrote Shakespeare in notebook 09,\n",
    "- and **controlling** how it generates, with a clear map of the road to frontier models, in notebook 10.\n",
    "\n",
    "Everything a large language model does is built from these parts. The rest is scale, data, engineering, and alignment.\n",
    "\n",
    "### Where to go next\n",
    "- **nanoGPT**, by Andrej Karpathy, is a compact and scalable version of exactly this, with which you can train a GPT-2 for real.\n",
    "- Reproduce **GPT-2** using the BPE tokenizer and a larger configuration.\n",
    "- Add a modern piece yourself: swap in RoPE or RMSNorm, or implement a KV-cache.\n",
    "- Try **fine-tuning**: take a pretrained open-weight model and fine-tune it on your own data.\n",
    "\n",
    "Nicely done."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (.venv)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}