{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b9088e7f",
   "metadata": {},
   "source": [
    "# 05 - Self-attention, from the intuition up\n",
    "\n",
    "The bigram saw a single character. A Transformer lets every token look back over all the earlier tokens and pull in whatever is relevant. That mechanism is **self-attention**, and it is the one idea that makes language models work.\n",
    "\n",
    "Here is why it matters. When you read \"the trophy did not fit in the suitcase because *it* was too big,\" your eye glances back to work out what \"it\" refers to. The bigram cannot do that, because it sees only the character immediately before. Attention is exactly that glance backward, turned into arithmetic. We will build it in four small steps, each a small change to the one before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cb1003a7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:49.846249Z",
     "iopub.status.busy": "2026-07-01T09:02:49.846145Z",
     "iopub.status.idle": "2026-07-01T09:02:50.021917Z",
     "shell.execute_reply": "2026-07-01T09:02:50.021520Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: (4, 8, 32)   (batch, time, channels)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(1337)\n",
    "def softmax(x, axis=-1):\n",
    "    x = x - x.max(axis=axis, keepdims=True); e = np.exp(x); return e / e.sum(axis=axis, keepdims=True)\n",
    "\n",
    "# a toy batch: B sequences, T tokens each, C features per token\n",
    "B, T, C = 4, 8, 32\n",
    "x = np.random.randn(B, T, C)\n",
    "print(\"input:\", x.shape, \"  (batch, time, channels)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f39ca28",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>import numpy as np; np.random.seed(1337)</code>: the array toolbox, with randomness pinned for repeatable numbers.</li>\n",
    "<li><code>B, T, C = 4, 8, 32</code>: three sizes: a <b>batch</b> of 4 sequences, <b>T</b>=8 tokens each, <b>C</b>=32 numbers describing each token.</li>\n",
    "<li><code>x = np.random.randn(B, T, C)</code>: a 4&times;8&times;32 block of random numbers standing in for token embeddings. They're random because here we only care about the <i>mechanics</i> of mixing them.</li>\n",
    "<li><code>def softmax(x, axis=-1): ...</code>: the same probabilities helper as always, on one line.</li>\n",
    "<li><code>print(\"input:\", x.shape, ...)</code>: confirm the (batch, time, channels) shape we'll carry through the whole notebook.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3283ed",
   "metadata": {},
   "source": [
    "## Step 1: the simplest way to gather context, averaging the past\n",
    "\n",
    "(First, a note on shapes, because everything from here on uses them. `B` is how many sequences we handle at once, `T` is how many tokens each sequence has, and `C` is how many numbers describe each token. So `x` is a block of numbers with shape `(B, T, C)`.)\n",
    "\n",
    "What is the simplest way for token `t` to use its context? Simply average it with every token before it, giving a blurred summary of the story so far. One firm rule: a token may never look at the *future*. When the model is generating text, the future has not been written yet (predicting it is the model's whole job), so during training we forbid looking ahead as well. This \"only look backward\" rule is called **causality**, and we enforce it everywhere.\n",
    "\n",
    "The double loop below makes the averaging obvious: token 0 is just itself, token 2 is the average of tokens 0, 1, and 2, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "adddf24e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.022980Z",
     "iopub.status.busy": "2026-07-01T09:02:50.022892Z",
     "iopub.status.idle": "2026-07-01T09:02:50.024985Z",
     "shell.execute_reply": "2026-07-01T09:02:50.024689Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token 0 is itself:  True\n",
      "token 2 is mean of 0,1,2: True\n"
     ]
    }
   ],
   "source": [
    "xbow = np.zeros_like(x)                     # \"bag of words\" = running average\n",
    "for b in range(B):\n",
    "    for t in range(T):\n",
    "        xbow[b, t] = x[b, :t+1].mean(axis=0)   # mean of tokens 0..t\n",
    "print(\"token 0 is itself: \", np.allclose(xbow[0,0], x[0,0]))\n",
    "print(\"token 2 is mean of 0,1,2:\", np.allclose(xbow[0,2], x[0,:3].mean(0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc33cd07",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>xbow = np.zeros_like(x)</code>: an all-zeros block the same shape as <code>x</code>, to fill in. (\"bow\" = bag-of-words, i.e. a running average.)</li>\n",
    "<li><code>for b in range(B): for t in range(T):</code>: visit every sequence <code>b</code>, every position <code>t</code>.</li>\n",
    "<li><code>x[b, :t+1]</code>: slice along time: tokens 0 through t (the <code>+1</code> because slices exclude their end). Only the past and the current token -- never the future.</li>\n",
    "<li><code>.mean(axis=0)</code>: average those vectors across time, giving one blended 32-number summary for position t.</li>\n",
    "<li><code>np.allclose(...)</code>: a sanity check that token 0's summary is just itself, and token 2's is the mean of tokens 0,1,2.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "363ccc6f",
   "metadata": {},
   "source": [
    "## Step 2: the same thing as a matrix multiply\n",
    "\n",
    "That averaging loop is secretly a single **matrix multiply**, and seeing this is what unlocks attention. Stack the averaging weights into a `(T, T)` table called `wei`, where row `t` says how much token `t` draws from each earlier token. The whole gather then becomes `wei @ x`.\n",
    "\n",
    "Look at the printed table: row 2 is `[0.33, 0.33, 0.33, 0, 0, ...]`, an equal third from tokens 0, 1, and 2 and nothing from the future. That **lower-triangular** shape (zeros above the diagonal, so each row reaches only backward, never forward) is what enforces causality, at no extra cost.\n",
    "\n",
    "The key reframing is this: \"who attends to whom\" is just a `(T, T)` grid of weights. Right now every earlier token gets an equal share. The entire rest of attention is one upgrade: stop averaging equally, and let the data decide those weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fcf6a6a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.026127Z",
     "iopub.status.busy": "2026-07-01T09:02:50.026073Z",
     "iopub.status.idle": "2026-07-01T09:02:50.028238Z",
     "shell.execute_reply": "2026-07-01T09:02:50.027896Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matches the loop: True\n",
      "\n",
      "weight matrix (row t = how token t mixes the past):\n",
      "[[1.   0.   0.   0.   0.   0.   0.   0.  ]\n",
      " [0.5  0.5  0.   0.   0.   0.   0.   0.  ]\n",
      " [0.33 0.33 0.33 0.   0.   0.   0.   0.  ]\n",
      " [0.25 0.25 0.25 0.25 0.   0.   0.   0.  ]\n",
      " [0.2  0.2  0.2  0.2  0.2  0.   0.   0.  ]\n",
      " [0.17 0.17 0.17 0.17 0.17 0.17 0.   0.  ]\n",
      " [0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.  ]\n",
      " [0.12 0.12 0.12 0.12 0.12 0.12 0.12 0.12]]\n"
     ]
    }
   ],
   "source": [
    "tril = np.tril(np.ones((T, T)))             # 1s on/below the diagonal\n",
    "wei = tril / tril.sum(axis=1, keepdims=True)  # normalize each row to average\n",
    "xbow2 = wei @ x                              # (T,T) @ (B,T,C) -> (B,T,C) by broadcasting\n",
    "print(\"matches the loop:\", np.allclose(xbow, xbow2))\n",
    "print(\"\\nweight matrix (row t = how token t mixes the past):\")\n",
    "print(np.round(wei, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee99804",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>tril = np.tril(np.ones((T, T)))</code>: an 8&times;8 grid of 1s with everything <i>above</i> the diagonal zeroed (\"tril\" = lower triangle). Row t has 1s only in columns 0..t -- i.e. \"position t may look at 0..t.\"</li>\n",
    "<li><code>wei = tril / tril.sum(axis=1, keepdims=True)</code>: divide each row by how many 1s it has, turning it into equal shares that sum to 1.</li>\n",
    "<li><code>xbow2 = wei @ x</code>: the punchline: multiplying by that (T,T) weight grid computes every position's average-of-the-past in a single matrix multiply. NumPy broadcasts it across the batch.</li>\n",
    "<li><code>np.allclose(xbow, xbow2)</code>: proves the matmul gives exactly what the double loop gave.</li>\n",
    "<li><code>np.round(wei, 2)</code>: print the weight grid so you can <i>see</i> row t mixing tokens 0..t equally, future columns at 0.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m5c1a",
   "metadata": {},
   "source": [
    "### Look at the averaging matrix\n",
    "\n",
    "The cell above printed `wei` as numbers. Seen as a picture it is clearer: a lower-triangular staircase where row `t` spreads an equal share across tokens `0..t` and puts nothing on the future. This is the crude, fixed version of the weights that attention will soon learn instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "m5c1b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.029269Z",
     "iopub.status.busy": "2026-07-01T09:02:50.029217Z",
     "iopub.status.idle": "2026-07-01T09:02:50.081697Z",
     "shell.execute_reply": "2026-07-01T09:02:50.081319Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcEAAAGYCAYAAAA6OlnXAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAUDhJREFUeJzt3Qd8E+X/B/DvtdBFaaEtUApl740geykIIqI4kKUgS5mi/FREZcuSIajIFEEBwQX4B8SBgMreggzZe7RQSlkttPm/Po8mJm3aJmnSJrnP29dJc7lcnlwu973neb7PnWYwGAxCRESkQz45XQAiIqKcwiBIRES6xSBIRES6xSBIRES6xSBIRES6xSBIRES6xSBIRES6xSBIRES6xSBIRES6xSBIditRooS8+OKL3HJuStM0GTlyZJbXg3VgXbGxseJNvvjiC6lQoYLkzp1b8uXLZ/frT506pbbL5MmTXVI+0mkQXLJkiUybNi1Hy7Bu3Trp0aOHlCtXToKCgqRUqVLSq1cvuXjxYo6Wi4ic4/Dhw+oErnTp0jJ37lyZM2dOusuuWbPGKScTnmaJGxyLs1MucaMNf+DAAXn11VdzrAxDhgyRa9euSfv27aVs2bJy4sQJ+fjjj2XVqlWyd+9eiYyMzLGyuZMjR46Ij4/bnD8R2WzDhg2SkpIi06dPlzJlymS4LILgjBkzdBcIl7jBsViXQdAdTJ06VRo1amRxgH/00UeladOmKhi+99574ilu3bolefLkccm6/f39XbJe0h9cv//u3bsSGBiYLe935coV9a8jzaDknbLldD4hIUGdVaAvCQfQggULyiOPPCK7d+9Wzzdr1kxWr14tp0+fVm3tmLCsUWJioowYMUKdueH10dHR8uabb6r55vC6AQMGyOLFi6V8+fISEBAgtWrVkt9++82mcjZp0iRNDQfzwsLC5NChQ5m+/vfff1e1yGLFipnK+dprr8mdO3dMy6AfAeXEZ01t6NCh4ufnJ3FxcaZ527ZtU4E4NDRUNdEiIG/atMlq383Bgwelc+fOkj9/fhXM4c8//1TNP2jaxfZAbRZNvlevXrV6lly7dm21HJqLZs+ebVp3Rn2CCxYsUMugXIMHD5YCBQqoAPzUU09JTEyMxWtxFo51RkVFqc/z0EMPqXJb62c8fvy4mjKD2vvrr78uVatWleDgYAkJCZHWrVvLvn370nw+lPOrr76SsWPHStGiRdVnbd68uRw7dkxscf78ebX9ChUqpL7jypUry/z589Msd+7cOWnXrp3aDtjfsR/8+OOP6v1RjvS2pRF+E5iMkpKSZPjw4Wp/xr6A9TZu3FjWr18vjvroo49U+fE9YJ/Bd49aQGrXr19XZUTgwHt3795dbt++bbHMZ599Jg8//LD6rNgulSpVkpkzZ6ZZFz7v448/rrYF3g/BD/uZ8X1wnMDvBuvA733ixIlqn7HFJ598oj4PXov9q3///mqd5u+N4whgH82o7xSfF7VAMB6TUv8OAM2p+K3gPR988EHZsWOH1SbYZ599Vh1HsL/hc3///fd29T1+8MEHUrx4cbW9cAxATc2crb/zrB6LvVG21AT79Okj33zzjQpQ+HHgi/njjz9UYHnggQfknXfekfj4eHXgwJcNOJgBfgBPPPGEWv6ll16SihUryv79+9Vyf//9t6xYscLivTZu3CjLli2TV155RX3J+GEgiGzfvl2qVKlid9lv3ryppoiIiEyX/frrr9XBoW/fvhIeHq7eEwcafC48B88995wK4DgQv/HGGxavx7yWLVuqAxL8+uuv6mCOAx9+vAjQxoMNAm6dOnUsXm9sxh03bpw6w4aff/5ZNeviwIUfxl9//aV+uPh369atph/2nj171HYqXLiwjBo1SpKTk2X06NHqYGGrgQMHqrKjrPgBo18B3zm+D/NA//7770vbtm2lVatWKlDhX9QGUkNwAqwrI/h82A/w+UuWLCmXL19WB1YcLBBgcUA0N2HCBLUtETix36E8Xbp0USccGcF669WrZzrZwrb54YcfpGfPnnLjxg1T8xFOelD2M2fOqP0Q749kDHyfjsL6582bJ506dZLevXurg9mnn36qth32sxo1ati1PvSHoWw4OA8aNEhtfxxIsQ1wImUO+yy26/jx49XBEuXAwRMByggBDwEIv9VcuXLJ//3f/0m/fv3U7xfBKHVzOj7Hyy+/rD4LTljxu8H3hZMMzMeJ5ObNm9X+gj75zPqoEMyw37Zo0UL9/vAeKBOCEk7OkASDdXz++eeyfPly9RyOMdWqVbO6PpThwoUL6veD784anDDge8Cy2CewHz399NNqf8T7AX5nDRs2lCJFishbb72lTl7wO8cJ0rfffqtOFDODMuN9sB3xPaEpF8cAHAdxMmbP7zwrx2KvZcgGoaGhhv79+2e4TJs2bQzFixdPM/+LL74w+Pj4GH7//XeL+bNmzcJR3rBp0ybTPDzGtHPnTtO806dPGwICAgxPPfWUQ2UfM2aMWue6desyXfb27dtp5o0fP96gaZoqh1H9+vUNtWrVslhu+/bt6n0+//xz9TglJcVQtmxZQ6tWrdTf5u9RsmRJwyOPPGKaN2LECPXaTp062VSmL7/8Ui3/22+/mea1bdvWEBQUZDh//rxp3tGjRw25cuVSy5rD99StWzfT488++0wt06JFC4uyvvbaawZfX1/D9evX1eNLly6p9bVr185ifSNHjlSvN1+n8X2s7ROp3b1715CcnGwx7+TJkwZ/f3/D6NGjTfPWr1+v3qdixYqGxMRE0/zp06er+fv378/wfXr27GkoXLiwITY21mJ+x44d1T5u3NbTpk1T6/vqq69My9y6dctQpkwZNR/lMP+MqT83NG3aVE1G9+/ftygzxMXFGQoVKmTo0aOHxXy8B/aJjDz55JOGypUrZ7iMcb9KvX78lsLDwzPdz7DvlipVymIePi/WuXbt2jS/szx58hj+/vtvi/lvvfWW2ofOnDmTbjmvXLli8PPzM7Rs2dJiP/j444/Ve82fPz/NZ4qJiTFkBscsa4dI7FuYj21w7do10/yVK1eq+f/3f/9nmte8eXND1apV1T5qhN9IgwYN1O87I8b3CQwMNJw7d840f9u2bWo+fl/2/s6zciz2VtnSHIpmFJxh4szKXqhBofaHlGakahsnnAlB6uag+vXrq5qTEc4on3zySdX8gtqNPdCMirNLnAkb3y8j5v0a6JNDORs0aKBqZahpGXXo0EF27dpl0dSH2hJqrigrIBHn6NGj6qwcZ2vGz431opaBsqVuJsJZXkZlwlkk1oHaDBibQLBdfvnlF3V2al5rQnMUaqK2Qk3dvMkIzXVYt7HpF9m39+/fVzWE1DVIa1ADzKwWCNhuxmZsvB+2F85eUcMwfkZzOFtGs7N5OQFn0unBd4gzd9Rg8bf5vojaGM6eje+FhArUqFHLMkKTI7aPo3x9fU1lxveOJmBsSzStWfuMtvwmcbZvrfkutdT7FbYXtjFqp9b2M2wLbBfU7LBN8dgcapXYZql/51gvWhLMty1qdvhOM+rSwL6L5mLUxM27M1DLRNM4mvdcAb9jY6uNtf0I3xFq/zh+oCZn/EzYdvj8+H2j5psZ/C5RkzRCC1DdunXVfmbP7zyrx2JvlS1BEM0EaMNGWz++QDRdZHTAMYcdBVV6ND2ZTxjGYN7RbYTmwNSwLJpb0D+FH8ulS5csJmvBEe34aKpAEyqaf2yB5i+0y6PtHwdhlBMHAjA/EKDZDj9WYzMhDqo4CCDg4Edr/NzQrVu3NJ8d5UF/qLWDS2r4IaK5C80m+KHg9cbljK/HNkQTnrVsucwy6MzhhMOc8QBh7OM0BsPU68T2Mj+Y2AtBAU03+O4RENF0jc+J5r3U28iWclqDfQf9S2hiSv19IKia74v4nPiMqfuQEJSzYuHChar5Dn0+aG7He+MAb+0z2pIJjX0Uv0dsNzS1pe5rtmd74bUIWGjuw4EWZXv77bfVc7bsp9jf165dm2bbYp3WfufmjPtV6u2Lkwb0kVnrf3eGzLYL+pnx2x42bFiaz2Xsm8zoc2V2TDM/QbTld57VY7G3ypY+QZwJ4SwJbfE//fSTTJo0SfUnfPfdd5nWNHCAQ8IDMjetwZdpD/QzIBnD3MmTJy06f8+ePav65pAEgLOtvHnzZrpeBFJ0MGNnxAEGNVccEHCmh8BoXmtDbQvbA30DOFCgzR4B1LyPxbg8tlV6/T2p2+qtZdhh2+Mzo/8R68FrsG70/9macGBPbcUaY/+kq6APFAcaJAKMGTNGBVWcZKBmYO0zOlJO43qef/55dWJiTXr9Sxmxlmxh3J/My7lo0SK1H6FWgO8SfXJ4Hv10tiQPpYbWFfSbYfgPgg9queg/R/INWj/s2V54f7ROYJ/H7xS/SQQg/HZwcpL6O7C2n2IZ/H7QX26N8aTXnWS2XYyfG33PqWu+jpxkZsTW33lWjsXeKtuGSKB5CM1gmHD2g05YZOgZN3x6BwNkXiF5Aj+y9JYxZ6xBmUMCDZqjcHaEmgI6kc2Zj/9DUwUCIGpaaL5DuW2BTmq8D87Wu3btapqf+r3Mm1KwLXAgQo0Q5UNTm/nnBtQMjWfD9sIZKT4DDmo4uKW3jXBARe3CWoakrVmTtkB2m3Gd5rUBbPOMamGZQUc/TmyQKGIONTdbEppsgX0HJ0MITpl9H/icONvGwdB8n8V3nRpqD+YZjEaovaAWY/4Z8RgHK/N1GmsUjsBJGvZDTGghQVIHfpNIRsH+YCskweD3goxH89qRPZmr2N+RgObIvm7cr7B9zbcZPhNOcB39/dhyvMmIsSxIknG0DBkd04wn7rb+zrN6LHY1NHkjKKOrCMlQCNQ46csIMq2RkY7WQpx8vfvuu3ZfzcrlzaE4aKRuDsFBF7Uh8yEO+EFaa9bBmQtqU8hmSw1NeOgjM7dlyxaLNnDU6lauXKkCG87ccNDBDmk+GX/wWNdjjz2m3g9nsdaaITI7KzSvTeBvZHJZ88wzz6jXfPnll6opFGnj5uP60K+JAwPSo3FwSC310ANbywSpM+2wHLYDMizN+woQrJD96Cw4kUHmYOrUeYzBtMbWIRIof+rPiG1qS3+LrfAe+M5QY0qdnp76+8A+hO2IwGWE5nhrVyfBd4yWABywjVA7w36b+v3B/HOibwf7uyNSp86j5oZsQaz/3r17dq3LWtnwW0Yms63wO8dnQd99ajhJQP9nerDvovwffvihRRlwUoRytGnTRhxh/D1aO0mxBY5zGHKATGVrV52y5TcM+F2a78vIBsZ3bwxatv7Os3osdjUcf6tXr24ampIZnODgu8UJMHIo0PKDK3xZ24dytCaIDmGMx0KSAD4gqunoyEaH/JQpUywO+qgRIapjvA2WQ83ohRdeUM2G6JzHmSXSjfFlos8O843jjYzQh4emB/MhEpC6iccapMljB0OzGlKGzccGojwZnZWgKQgHNDR9YIdFDQ4HzPRqONj58OWh+QjbCGfj5tCch74/7OhIPUe/EzrHsW5sB6wfZ+AZwTIY54h+ABzY8Ho0gWDnSQ19A3gO2xcp5tjGCE7YntjBnAH9Fei3wPeOVHo01aCWj0CLGlvqM1Bbh0jgBALDObCNkIiEWjnGiprXCpwBQyuw7ZGUgKQLBA00f+OkC/s0/gY8h22HFgGc1eLMG2n2qO2nhh8tgiW2BQIBgj6aPo0tAeafEbVA9FPjh4/vcNasWaoM1k6SMoOTQrSA4PvG94J9HWXGum1p/k+9LgQh/F4xXADlwUkr9nFbLzmIZjzUJPE5cSaP4wEOivgusX2wD6RXq0ctHbVX/MaxHbFvoVaI3z6OJWjCdoQxwQ7HEhxTEGw6duxo1zpwQMeYXXTpYL/APomhNgj4SExKPZY1vSZTrAO/SwQrBDf0CRubjm39nWf1WOxqONbZ0ySL/R8tSsayo4kfwz3QBJ9e87NVrk4/RVr3G2+8Yahevbohb968Kg0af3/yyScWy928edPQuXNnQ758+VRar3mKblJSkmHixIkqpRtp7/nz51dDDEaNGmWIj483LYfXIf130aJFKv0Yy9asWdMiJT0jxvRta5MtKcMHDx5UwwSCg4MNERERht69exv27dunXo9hBKnNnTtXPYftcufOHavr3LNnj+Hpp59W6dj4PCjHc889ZzFkI6O0b6RWI6Ud2xXp0e3btzdcuHDBaho91onthXTz0qVLG+bNm2f43//+p4aY2DJEYseOHRbLGYckmG9/pPoPGzbMEBkZqVK/H374YcOhQ4fU5+vTp4/DQyRQTgxfwDobNmxo2LJlS5phBsbyfP3111ZT0a19R6ldvnxZ7WPR0dGG3Llzq8+BNPg5c+ZYLIchMU888YQadoJ9YdCgQWpYQOrtAVOmTDEUKVJEfb8oO4b4pC470urHjRuntodxv161apX6HlJvI1uGSMyePdvQpEkT036F7xu/U/PfU3r7lfH7xnYz+v777w3VqlVT+0qJEiXU7xVDE1Ivh7IiBd+ahIQEw9ChQ9VQEuyD2G4YSjB58mR1DMgMhkRUqFBBfS8YOtK3b181jMScPUMksK8OHDjQUKBAATXMyXi4NO4vkyZNSvMaa9v++PHjhq5du6p9BWXDd/34448bvvnmmwzf3/x9sI9gn8N31bhxY3Vcsfd3bu+xODQ0VL0e74v9wpEJw6NSzzMfLpIevO/y5cszXAbbAb8rc9jnQkJCMl2/xXsZvIgxCJLzYDwZDkquhAMVvrv33nvP4M2snRQQpSejYOtqd+7cMUiuoHQrBbZOqBCknpfZCZqtQRAVHZwYmlu9erV6rbVxk+nhtUPJoo/VPHMPHevoG00vG9IZ72Hed2F+mTAiyjlJ6KO+f1v8K3cX8f1vTK1dkpPk5l+fqf5t49Avd7z2MIMgmaC/wnj9QWQnIoEFfT3ppa07An0NuNYokkfQ14A2fCQHoV8J/VNE5EZ8/URzMAga03QQAM2DoLOgTxv9q+bwGO9lzwXZGQTJBEkFCEi4gADO1nD1HYzBsydLNjMYS4cMUXTi44ojxmQZT7pDB5FuaGrMhOOvdSEcn8yvmmMckob59tD+bX8lIiJScIKKi4X41+grmq9jzZeG5ERJ3DtTDbewpSaIrGLjuOSaNWuqzHlk0OPiFxh/igxgZMfjguKA7Fdkr+NqR8joxyXqkMmLqyjZkx3KmiAREVmHWqDDNUH7Xrdz506Lq3lhiAYgJwFdKBhugytrGWF4BAIeblOG8dgY/oFhZXYNj2BNkIiI0q0J1uyXtZrgnk9srgnmFI+uCeKaeLgyBwb35tSlfoiIchp6tTAYHld/SX1j8CzRfP6ZHH2tB/DoIIgAaO8FtImIvBWGI6BZ0BObQ3OKRwdB4+Wd/Cp1cziN19XObJic00UgIi+XcOOGlCkZbfcl7zKXhZpg9typT99B0NgEigDorkHQndvCici7sFtIZ0GQiIhcSGNzKBER6ZXGxBgiItIrzftrgp7Rc0lEROQC7BMkIiLr2BxKRES6pXl/cyhrgkREpNuaoGeUkoiIyAVYEyQiogyaQx2tCbI5lIiIPJmP9s/k6Gs9AGuCRERkHfsEiYiIvJeuEmOWTektFza+L0sm9bT6fO3KxWXXN+/IgZUjZOhLj5rmlywaIX8sflPN//CdjtlYYiIiNxgioTk4eQC3CIIzZsyQEiVKSEBAgNStW1e2b9/ukvf5eMkG6TXs83Sfnzb0Oek29DOp9tRoebRhZalcJkrNHzvoSRk7a41UeXKUROTLI60bV3FJ+YiI3LI5VHNw8gA5Xsply5bJ4MGDZcSIEbJ7926pXr26tGrVSq5cueL09/p911FJuJVo9bnCBULF19dHDhy9ICkpBvn6x13Susk/wa5e9VLyw+8H1N9frt4hj/07n4jIq2msCbrc1KlTpXfv3tK9e3epVKmSzJo1S4KCgmT+/PmSnRAEL1yJNz3G30UKhEp4vjxyLf6W2fzrElUwX7aWjYiIvLAmmJSUJLt27ZIWLVr8VyAfH/V4y5YtaZZPTEyUGzduWExEROQiGptDXSo2NlaSk5OlUKFCFvPx+NKlS2mWHz9+vISGhpqm6Ohop5XlYky8RBUMNT3G3xdi4uXq9VsSFprHbH4+tSwRkdfT2BzqVoYOHSrx8fGm6ezZs05bNwJbcopBqpSNEh8fTdq3qiVrfvunH3D7nydNyTAdH6sta37b77T3JSJyWxprgi4VEREhvr6+cvnyZYv5eBwZGZlmeX9/fwkJCbGY7LF61gBZPKmHtGpYWY6tHSN1q5WU5R/1Vf2BMHjCV7JwfHfZv2K4/LT5kPx17IKa/86HK+XdPo/JX9+PkLgbd+SH3//K0ucmIiL3kKNXjPHz85NatWrJunXrpF27dmpeSkqKejxgwACnv1+bPh+nmffUwJmmv7fvPyW1nh2bZpnjZ2KkYZf3nV4eIiK3pvFWSi6H4RHdunWT2rVrS506dWTatGly69YtlS1KREQ5yScL4/1yfASeZ1w7tEOHDhITEyPDhw9XyTA1atSQtWvXpkmWISKibKaxJpgt0PTpiuZPIiIit64JEhGRm9J4P0EiItIrLQt9gh5y7VDWBImISLd9gp4RqomIiFyANUEiIrKOzaFERKRbmvc3h7ImSERE1rEmSEREuqV5f02QiTFERKRbbA4lIiKrNE1TkzfXBBkEiYjIKgZBIiLSL+3fydHXegD2CRIRkW55RXPoC2/2Er+gYHFH//v+oLi7KU9UyukiEJEbYnMoERHplsbEGCIi0itNB0GQfYJERKRbXtEnSEREzqfpoCbIIEhERLodIsEgSEREuq0Jsk+QiIh0izVBIiLK4CYSjtYExSMwCBIRkVUa/nO4WdMzoiCDIBERWcU+QSIiIi/GmiAREVnHIRJERKRbmuN9ggYPGSLBmiARETm9T9DxhJrsxXGCRETkFmbMmCElSpSQgIAAqVu3rmzfvj3D5adNmybly5eXwMBAiY6Oltdee03u3r3rOTXB3377TSZNmiS7du2SixcvyvLly6Vdu3Yue7+4o3vkzLovxWAwSFT9NlKwRjOL5w8uGif3bsaLT67c6nHlbsPFJ7ef3LudIEe/+1iSEq5JUIGiUqZdX/HJ5ae78hGRvmjZWBNctmyZDB48WGbNmqUCIAJcq1at5MiRI1KwYME0yy9ZskTeeustmT9/vjRo0ED+/vtvefHFF9X7Tp061TOC4K1bt6R69erSo0cPefrpp136XoaUZDmzbolU7DJUfP2D5MD84ZK/XC3JHZTXYrmyTw+UoIJFLeZd2LJKwirUlsjaj8jpdV/Klb0b1d96Kh8R6ZCWfdcOReDq3bu3dO/eXT1GMFy9erUKcgh2qW3evFkaNmwonTt3Vo9Rg+zUqZNs27bNc5pDW7duLe+995489dRTLn+vmxdOSGBEUfHLGya+fgGSr3Q1iT95wKbXXj+6WyKqNFR/R1RpINeP7tFd+YhIvzVBzcHJVklJSapFsEWLFqZ5Pj4+6vGWLVusvga1P7zG2GR64sQJWbNmjTz22GN2fUbdJMYkJcSJX978pse58+aXewlxaZY7tnKmaD6aCiqF67ZW8+7fvSO5AoLU337B+dW69FY+IiJH3Lhxw+Kxv7+/mszFxsZKcnKyFCpUyGI+Hh8+fNjqelEDxOsaNWqkupDu378vffr0kbffftt7g2BiYqKa0tu4WVXmyT6qJnb/7m35++sPJCC8sOQvU0PchbuXj4i8i+aEPkEkrJgbMWKEjBw5Mstl27Bhg4wbN04++eQT1Yd47NgxGTRokIwZM0aGDRvmnUFw/PjxMmrUKIdei1qWeQ0Jtaw8UaVSLROm/kWtKqxiXbl14YQKMrkCAlXgwfykm3GqluZs7l4+ItIfzQlB8OzZsxISEmKan7oWCBEREeLr6yuXL1+2mI/HkZGRVtePQPfCCy9Ir1691OOqVauqPJOXXnpJ3nnnHdWc6nVDJIYOHSrx8fGmCRvXVsFRpeR2zDmVQZmcdFeuH/9TQktWtUhMQZYlpCTfl/gT+ySwwD8JKPnK1JDYA5vU37EHNruk9uXu5SMi/dGc0CeIAGg+WQuCfn5+UqtWLVm3bp1pXkpKinpcv359q2W7fft2mkCHQApoHvXKmqC1tmRbaT6+Urx5Jzm0ePw/QxDqtVGZl4eXTZZSj/VUGZmHl04SQ3IyIo7kK1tDwio8qF4bVb+tHP3uI7m040c1BCG6yTNO/mTuXz4iIlfC8Ihu3bpJ7dq1pU6dOmqIBGp2xmzRrl27SpEiRVSLILRt21ZllNasWdPUHIraIeYbg6HbB8GbN2+qghudPHlS9u7dK2FhYVKsWDGnv1/+cg+oyVyFDq+b/q7aY7TV1+XOEyKVXnjH6eXxtPIRkc5o2TdEokOHDhITEyPDhw+XS5cuSY0aNWTt2rWmZJkzZ85Y1PzeffddVdvEv+fPn5cCBQqoADh27Fj7immwp97oZOjYfOihh9LMx9nAggULMn09EmNCQ0Ol1xfbxC8o2EWl9H5TnqiU00UgoizAsbBQeKjqJjLvf8vK+kJDQyWyxyLx8fsn89xeKUm35dL8551WJlfJ0Zpgs2bN7Gq7JSKi7KPx2qFERETey6MSY4iIKPtoOqgJMggSEZF1vKkuERHplaaDmqBHDZYnIiJyJjaHEhGRbmuCDIJERGSVJlkIgg6Pss9eDIJERGQVa4JERKRfWvZdNi2nMDGGiIh0i82hRERkFZtDiYhItzRmhxIRkV5p2j+To6/1BOwTJCIi3fKKPsHGJUIlKDhvThfDY606cEHc2eNVonK6CEQ6rglqDr/WE3hFECQiIhfQshDMGASJiMiTaTpIjGGfIBER6RabQ4mISLfZoQyCRERklY+PpiZHGBx8XXZjECQiIt3WBNknSEREusWaIBER6TY7lEGQiIh02xzKIEhERLqtCbJPkIiIdIs1QSIi0m1NkEGQiIisYp8gERHpliZZqAl6yBW0dVUTPLZvh/z61XwxGFKk3qPPSPUmLS2eXzTxLUm8c1tSkpOlYp3G0qhtRzV/5ZxJEnvhrBhSUiS6bCVp2aWPaD4+uiyju5ePiMhjguD48ePlu+++k8OHD0tgYKA0aNBAJk6cKOXLl3f6e+GgvO6rT6Xz62PFPzBIFowZLOUeqCeBwSGmZdq/Mlw9h2UXTRwiZao/KJHFSsujL/RX8w0Gg6yYNVH+3rtNyj9QX3dldPfyEZFzaToYIpGjp+IbN26U/v37y9atW+Xnn3+We/fuScuWLeXWrVtOf68LJ/+WiKhikjd/uPgFBEqpqg/Iyb/2WCyDgzQkJ9+X5PvJpuq8cT5qMcn377usw9fdy+ju5SMi1yTGaA5OniBHa4Jr1661eLxgwQIpWLCg7Nq1S5o0aeLU97p5/ZrkzRdmepw3X7gkXL+WZrkvxr8pMedOS82HWkuhYqVM85fPnCCnD++XUpVrStnqdZxaNk8po7uXj4icS2NNMHvFx8erf8PC/jvQmktMTJQbN25YTM72wtD3pf+Uz+TK2ZMSc/60af5Tfd+SgVMWiEEMcurQPqe/rzeV0d3LR0Rk5DaZCSkpKfLqq69Kw4YNpUqVKun2IYaGhpqm6Ohom9cfnC/MotaScP2qmmeNf0CQFK9QTU4c2G0x3zdXbilXs74c3btNXMHdy+ju5SMi59J00BzqNkEQfYMHDhyQpUuXprvM0KFDVW3ROJ09e9bm9UeVLCex509LQtxVSbp7R07s362a5Yzu3r4ltxP+qYnev3dP9XWFRxZV/VfxV6+o+SkpyXLszx0SXrholj6rp5bR3ctHRK5pDtUcnDyBWwyRGDBggKxatUp+++03KVo0/YOjv7+/mhzh4+srDz/XQ5ZMfkclZ9R79GmV1fjVtFHS+sUBKptx+ScTJPn+PZXBWOHBRiqz8V5ioqycPUnuJd5V84uVryI1m7bOwqf13DK6e/mIyLk0HVwxRjPgqJRD8NYDBw6U5cuXy4YNG6Rs2bJ2vR59gmgWXfj7YQkKzuuyclLOerxKFL8CokyOhYXCQ1ULWUjIf0OWHHXj32NrreGrxTcgj0PrSL57S3aNbuO0MnllTRBNoEuWLJGVK1dK3rx55dKlS2o+Nj7GDRIRUQ7SstCs6RkVwZztE5w5c6Y6S2jWrJkULlzYNC1btiwni0VERKKPxJgcrQnmYEssERFlguMEiYiIvJhbZIcSEZH70XSQHcogSEREVrE51IpSpUrJ1atX08y/fv26eo6IiLyDpoPEGLuzQ0+dOiXJyclWr+t5/vx5Z5WLiIjIfZpDv//+e9PfP/74oxrLZ4SguG7dOilRooTzS0hERDlCY5/gf9q1a2faKN26dbPYULlz51YBcMqUKdn8FRERkatoOriVUi577vIAJUuWlB07dkhERIQry0VERDlMY00wrZMnT+bAV0FEROR8HCJBRERWsTmUiIh0S2NzKBER6ZWWhQQXD8mLYXMoERFZ56NpanKEo6/ziD5BZIoeO3ZMrly5YsoaNWrSpIlkt8J5AiRPHt5/0FvtPhkn7uyBkvlzughElF1BcOvWrdK5c2c5ffp0mlshof3Y2tVkiIjI82g6GCdo92XT+vTpI7Vr15YDBw7ItWvXJC4uzjThMREReQctm68dOmPGDHXhlYCAAKlbt65s3749w+Vxzer+/furm7H7+/tLuXLlZM2aNa6tCR49elS++eYbKVOmjL0vJSIiD+Kj/TM5+lp7LFu2TAYPHiyzZs1SAXDatGnSqlUrOXLkiBQsWDDN8klJSfLII4+o5xCTihQpoloo8+XL59ogiMKhP5BBkIiInGXq1KnSu3dv6d69u3qMYLh69WqZP3++vPXWW2mWx3y0Pm7evFlduhMcuX613UFw4MCB8r///U8uXbokVatWNb25UbVq1ewuBBERuSEtCzfHteNlqNXt2rVLhg4daprn4+MjLVq0kC1btqR7U4f69eur5tCVK1dKgQIFVL7KkCFDxNfX13VB8JlnnlH/9ujRwzQPGwlJMkyMISLyHpoTEmNu3LhhMR99d5jMxcbGqqTKQoUKWczH48OHD1td/4kTJ+TXX3+VLl26qH5AtFD269dP7t27JyNGjHBdEOS1Q4mI9EH79z9HXwvR0dEW8xGgRo4cmeWyYXge+gPnzJmjan61atVS97SdNGmSa4Ng8eLF7X0JERHp1NmzZyUkJMT0OHUtEHBXIgSyy5cvW8zH48jISKvrRUYouuPMmz4rVqyouurQvOrn5+eaIRLwxRdfSMOGDSUqKkpl4wAyedAuS0RE3pUd6uPgBAiA5pO1IIiAhZocbs5uXtPDY/T7WYMYhCZQ8wu2/P333yo42hoA1We0b5OIzJw5U6WxPvbYY2qMhnFwPNJSEQiJiMg7aNk4ThBxZe7cubJw4UI5dOiQ9O3bV27dumXKFu3atatF4gyeR3booEGDVPBDJum4ceNUoow97G4O/eijj1RBcaf5CRMmmOZjAP3rr79u7+qIiMhNadl4xZgOHTpITEyMDB8+XDVp1qhRQ9auXWtKljlz5ozKGDVCX+OPP/4or732mhqVgHGCCIjIDrWHQ4kxNWvWTDMfVVxEbSIiIkcMGDBATdZs2LAhzTw0leJSnllhd3NoyZIlZe/evWnmI2KjU5KIiLzrLhI+Dk6ewO6aINpt0eZ69+5dNTYQ13b78ssvZfz48TJv3jzXlJKIiLKdpoMLaNsdBHv16iWBgYHy7rvvyu3bt9UIfWSJTp8+XTp27Cju7MDurbJi0WwxGFKkedsO0uDhx0zPJSXelU8/GC1XYy6Jj+YjDVs8Lk0fbaeeW/PN57Jl/Q8SnPefNN9nXxwgpStU1V35PKGM7l4+Ik+i8c7yaWH0P0boY0IQvHnzpunipvZeUxSZpphOnTqlHleuXFl1irZu3VqcDVmsy7+YJQOHTZbAoDwy6e1+Uv3BRpLn34MetHiig5StVF0S796RSe/0l0o1HpQCkUXUc4880UGatPrngOkK7l4+Tyiju5ePiNyP3X2Cbdq0kcTERPV3UFCQKQDiSt/NmjWza11FixZVGaa4ZtzOnTvl4YcflieffFL++usvcbbTxw9L4aIlJF9YhPgHBErF6g/K4T93mp738w9QB0fA8wULF5Ub17Pv1lDuXj5PKKO7l4/IU5tDNQcnr2wODQ4OlqeeekpdvDRXrn9ejjEdCGDPPfecXetq27atxeOxY8eqmiGyfVArdKb4uKsSGhZueowD5fW4q1aXjbt6RS6cOSlFS/xXq12/5lvZvG6NlCxfWdp1eUkdRPVUPk8oo7uXj8jT+GQhwcVTEmPsrgl+9913Eh8fr5pDkRiDm+uiBtipUyfVL5iVpqylS5eqYRbpXSEANVA0x5pPznbvXpJ8Nn2sPNmlt+kg2KhFWxn2wQJ5c8JM8fPzlx++/cLp7+st5fOEMrp7+YjchZbFySuDIJJiMDIfzZ+o+TVv3lyN5Me9oByxf/9+VbvEOEPctX758uVSqVIlq8siAzU0NNQ0pb4wa0ZC84dL/LX/agXXr8WqeeYQ1Bd98r5UrlFHatZtYpofki+/+Pj4qqles0flzPEjDn1WTy6fJ5TR3ctHRB4aBFPXvjBqH3cB3rZtm7q10rBhwxyumZUvX16NO8S6cBmcbt26ycGDB60ui0vmoBZqnHBhVlsVL11BLp47qQ6MSIo4tG+HVKxW22KZ/1v6qfj5+0urp7ukaWYz2r9rs0QWdf5FxN29fJ5QRncvH5Gn0bLxsmk5RTPg1DgTCHrWPpDxpc68nyBuoli6dGmZPXt2pssi6KJG+NPuU5In+L8MwPTs37lZViyeo8ravO1z0rB5G5k18W3p1HuwpBgMMmJAZ4ksUlx8c/1zVfInOvVSyRWfz5gg508fV7cGKVC4qHTs/apN72cvdy+fJ5QxJ8r3QMn8Tv8cRPbAsbBQeKiqHJjfscFRN/49traf/bvkDgx2aB337tyUr19u7LQy5WgQ3Lhxo80rbNq0aZYKhASbYsWKyYIFC5weBIlcgUGQvDUIPjfnjywFwa9eauT2QTBXdgS29KB5E2MCEfQSEhJkyZIl6vpwuCgqERGR2w2RANxC6dNPP1VDIwDDGXr06KHOHOxx5coVlVRz8eJF9VpcCRwB8JFHHnGkWERE5GSaZ3TtZV8QxKD2Vq1aqSzROnXqqHnIDMUYv59++kkeeOABm9eFQEpERO5J42XT0sK9m5544gl1T0HjYPn79++ra4q++uqr8ttvv2X7F0VERM7nY3aHeEde67U1QfMAqFaSK5e8+eab6sa6RERE4q2D5ZHlgzv8poYxe3nz5nVWuYiIKIdpOhgnaHcQ7NChg/Ts2VMNlkfgw4TLnaE5FJdOIyIi76Dp4LJpdjeHTp48WUV4ZHWiLxBy586trvaCO0IQEZF38NHBBbTtDoJ+fn7qQtm4jufx48fVPFzhBbdVIiIi8iR2N4diPCAGtiPoVa1aVU34G3d/wHNEROQdNB3cT9DuILhw4UK5c+dOmvmY9/nnnzurXERElMM0HSTG5LLnWnK4zCgm1AQDAgJMz+Gi2WvWrDHdZZ6IiDyfloUanYfEQNuDYL58+UzRvVy5cmmex/xRo0Y5u3xEREQ5HwTXr1+vaoG4y8O3334rYWFhFskyxYsXl6ioKFeVk4iIspkPs0PT3kni5MmT6q4PntLeS0REjtHYHJoWanxEROT9NF5A2zME+uWSIH+H7gpFlGWHLyS4/VasEMVLGhJZw8hBRETpjqHzycJrPQGDIBER6bY51O5gjexQ3Fne2jhCPEdERN5B+/d+go5MHhID7a8JbtiwQZKSktLMv3v3rvz+++/OKhcREeUwH95U9z9//vmn6e+DBw/KpUuXLK4Ys3btWilSpEg2f0VERETZUBOsUaOGqX3YWrNnYGCgfPTRR1koChERuRNNB32CNgdBDJLHFWNKlSol27dvlwIFClhcMQbXDfX19XVVOYmIKJv5sDk07SD5lJQU7ohERDqg8Yox1h09elRdS/TKlStpguLw4cOz5cshIiLK9uzQuXPnSt++fSUiIkIiIyMt2n3xN4MgEZF38OEFtNN67733ZOzYsTJkyJAc+EqIiCi7+PCKMWnFxcVJ+/btuRcSEXk5TQd9gnZfMQYB8KeffnJNaYiIiNy5T7BMmTIybNgw2bp1q1StWlVy585t8fwrr7zizPIREVEO8RFN9Qs6+lqvDIJz5syR4OBg2bhxo5rMITGGQZCIyDtoOmgOtTsIYtC8K0yYMEGGDh0qgwYNkmnTprnkPfbt3CJfLZgpBkOKPNqukzR5pI3pucTEuzLz/RESe+Wi+Pj4SNOWT0jzNk+r5xJuxMusySMl7mqMFClWSl567V3J7eenu/J5QhlZPiLn8dHBYHmHb/mEi2gfOXJE7t+/n+VC7NixQ2bPni3VqlUTV8H1Tb/67BN5fdQUGT55rvy4cpncTIi3WKb1053kvY8+l7cnzJT1a1fI5Yvn1fwfvlsiteo1kXEzFkmBQoXl919W6658nlBGlo+IXB4Eb9++LT179pSgoCCpXLmynDlzRs0fOHCgqs3Z6+bNm9KlSxc1/jB//vziKiePHpKo6BKSP7yABAQGStWadeSvvTtNz/v7B0j5yjXU33g+Mipa4uOuqsf7dmyWek0fUX/j3307N+uufJ5QRpaPyBW3UtIcmjylOdTuIIgmy3379qlbKgUEBJjmt2jRQpYtW2Z3Afr37y9t2rRRr3el69euSr7wCNNj/H39aqzVZa/FXpFzp09I8VJl1ePbt29JUJ5g9Xf+sAiJS+d13lw+Tygjy0fkmj5BzcHJK/sEV6xYoYJdvXr1LK4Wg1rh8ePH7VrX0qVLZffu3ao51BaJiYlqMr+Rr7Pdu5cks6eMlme79hH/gEBxN+5ePk8oI8tHZBsf9gmmFRMTo+4YkdqtW7fsunXG2bNnVRLM4sWLLWqUGRk/fryEhoaapujoaJvfL19YuEWtBX9jnjncJePT6eOl6gN1pXaDpqb5QUF55Patm+rvuGt43X+1IWdx9/J5QhlZPiJyeXNo7dq1ZfXq/5IajIFv3rx5Ur9+fZvXs2vXLnUB7gceeEBy5cqlJgy5+PDDD9XfSHKw1hQbHx9vmhBIbVWybEU5f/akyk68e+eO7N+zXSrXfNBimW8XzRU//wB5vP0LFvOr1a4nWzf+rP7Gv9Vr2/45vaV8nlBGlo/IubQs/ucJNANO3e3wxx9/SOvWreX555+XBQsWyMsvv6zuNL9582YVxGrVqmXTehISEuT06dMW87p37y4VKlRQ1yWtUqVKputAcyhqhL8fOCfBeUMyXX7v9k3y9cJZkqLS+ztK05ZtZdp7b8mL/V6XlBSDvPnScxIVXVx8ff9pJX7mhZekSs06khB/XWZOGqFqMEWKlZSXXhsmfv7+4mzuXj5PKCPLZ12FqLxO39bkPnAsLBQeqioHISEhTllfaGiojPh+jwTkcWzfuXsrQUY9UdNpZXKbIAgnTpxQTZNIkEF2J2pzCFy4gkxWNGvWTN3B3tZxgvYGQSK9YhD0bq4KgqP+L2tBcERb9w+CdiXG3Lt3T9X8cNk0DGkgIiLSTZ8grhP67bffuqwwGHbhqqvFEBGRfTQ13s/xySsTY9q1a6eGSRARkT6GSPg4OHnlOMGyZcvK6NGjZdOmTSoJJk+ePBbP8wLaRETeQeMFtNP69NNPJV++fGqIAybLDca7SBARkeewqzkUiaTot8OQCNxNIvWErFEiIvIOPg5eN9Q42WvGjBlSokQJdQGVunXryvbt222++hgqYeius/sz2hsE0Rx67tw5u9+IiIg8i0829gnicpyDBw+WESNGqMtpVq9eXVq1aqUuqpKRU6dOyeuvvy6NGzd27DPatbCPjwqCV6/+c2cAIiLyYloWLp5tZxCcOnWq9O7dW100pVKlSjJr1ix1t6L58+en+xpcWQx3IRo1apSUKlUqe7JDcbukN954Qw4cOODQGxIRkX7cuHHDYjK/CYL5/WmRY2J+NyFUuvB4y5Yt6a4bSZq4ljVu75dt2aFdu3ZV9xREVdXPz08CAy3vEnDt2jWHC0NERO7DRzQ1OfpaSH2jAzR3jhw50mJebGysqtUVKlTIYj4eHz58ON1LeCJRc+/evZIVdgdBDmYnItIHzQlDJHCjA/PLpvk74ZrBuPb0Cy+8oK5cFhERkb1BsFu3bll6QyIi0s/9BENCQjK9digCma+vr1y+fNliPh5HRkamWR73rkVCTNu2bU3zUlJS1L+4C9GRI0ekdOnSrgmCZ86cyfD5YsWK2btKIiLSMT8/P3XxlXXr1pmGOSCo4fGAAQPSLI+7De3fv99i3rvvvqtqiNOnT7frXrN2B0GM4cjomnDW7gNIRESex8fB8X7G19oDwyPQ0oh71tapU0d1veFm7cgWNeajFClSRN3BCOMIU99uDxdxAVtuw5elILhnz540d5bAPKS3jh071t7VERGRm9Ky8bJpHTp0kJiYGBk+fLhcunRJ3VZv7dq1pmQZtEIiY9Qt7idoDe42P2nSJHVFmezC+wkS2Yb3E/Rurrqf4EfrDkhgsGP3E7xzM0EGNq/iXfcTzEj58uVlx44dkhP8c/moiYisO3nllltvmpIFLS/ET+S2QRBnCOZQkbx48aIa94GryRARkXfQeBeJtND5mDoxBoEQ2Ti4iCkREXkHH0cuK2b2Wq+sCf76668WQRAdlQUKFJAyZcqo8RlEROQdtCzcId5T7ixvd9Rq1qyZa0pCRESUzeyusWKMhrWremPexIkTnVUuIiLKYVoWJ68MgrNnz1aj9VOrXLmyuvUFERF5B59svqmuRzSHYhBj4cKF08xHvyCyRImIyHto4t3srgkiC3TTpk1p5mNeVFSUs8pFREQ5TMvCTXU9pCJof00Qd/599dVX1eXSHn74YTUPFzl988035X//+58rykhEROQeQRB3lb969ar069dP3Q0YcDHTIUOGyNChQ11RRiIiygEah0hY3yjIAh02bJgcOnRI3VkeV4pxxo0SiYjIffhwsHz6goOD5cEHH8zGr4OIiLKTpoOaoKdc2YaIiMjpeJ0zIiKyKiuD3j2jHpjDQRB3nhg1alSaWzIdPnzYJe+3Z8dmWTL/EzGkpMjjz3SWZi0ft3h+wawPZMemjRJeoKCMnjrHNH/MWwPk7p076u+4qzHSoOkj8nzvgbornyeUkeXz/m1I2UfTQXNojtcEcaWZX375xfTYVRfhTk6+L0s+nSFDx06ToKA8MmzwS1KrXmPJGxJqWqZBkxbStMVj8tknUyxeO2zCx6a/xwwZILXqNdJd+TyhjCyf929Dyl4+OkiMyfFyIuhFRkaapoiICJe8z/G/D0uRYiUkLLyABAQGSfUH6sqBPZY3AS5XqaoE503/DsjXrsZIzOWLUr5ydd2VzxPKyPJ5/zYk8rogePToUXWlmVKlSkmXLl3kzJkzLnmf69diJX9YAdPj/OERcu1arF3r2P7HBqndoIm6fZTeyucJZWT5vH8bUs40h2oOTp4gR/fSunXryoIFC2Tt2rUyc+ZMOXnypDRu3FgSEhKsLp+YmKjubG8+Zadtm9ZLvUb/XCXHHbl7+TyhjCyf929Dsp3Gu0i4VuvWraV9+/ZSrVo1adWqlaxZs0auX78uX331Vbq3cQoNDTVNuI6prfKFRUjctRjT47irOOMNt/n1sTGXJS42RspWrCKu4O7l84Qysnzevw0pe2k6uHaoW7VX5MuXT8qVKyfHjh2z+jwuyxYfH2+azp49a/O6S5erIOdOn1T9FXfv3JZ9u7dJ1Zp17GriqdOwmcuq+O5ePk8oI8vn/duQyKuD4M2bN+X48eNWb9UEuDRbSEiIxWQrX99c0rlHfxn/zqvyzqBe0vrJ51TG26RRb6qzXZg9bbyMerOfnDl1XF7p/qxs+2O9RRNPnUYPOeFTemb5PKGMLJ/3b0PKXj6iZWnyBJrBYDDk1Ju//vrr0rZtWylevLhcuHBBRowYIXv37pWDBw+q+xNmBn2CaBbdfvhChtlqROTeShbMk9NF8Gg4FhYKD1UtZPZUDjI7ti7bclSCgvOKI27fTJAO9cs6rUxeOU7w3Llz0qlTJ3VXCgS9Ro0aydatW20KgERE5Frav/85+lpPkKNBcOnSpTn59kREpHM5fsUYIiJyT1oWsjw9JTeKQZCIiNJt0nQ0wYXNoURE5NE0HdQE3WqIBBERUXZicygREem2JsggSEREVnGIBBER6ZaP9s/k6Gs9AfsEiYhIt9gcSkREVrE5lIiIdEtjYgwREen7prqaw6/1BOwTJCIi3WKfIBER6TY7lEGQiIisYmKMh8jlq6mJiDzT2au3xd1FhweJ3mg6SIxhnyAREekWm0OJiCiD7FDHeEhFkEGQiIisw70EfRxs13T0PoTZjTVBIiLSbU2QfYJERKRbrAkSEZFuq4IMgkREZBXHCRIRkX5pWRjv5yE1QfYJEhGRbrE5lIiI9NolyCBIRET6jYKsCRIRkVVMjCEiIt3SeAFtIiIi76Wr5tDd2zfLonkzxGBIkbbPdpGHWz1u8fz8T6bKtk0bJTyioIybPjfN6z8YN0xiLl+y+pxeysjyeff284Qyunv5vInm/V2COT9E4vz58/L8889LeHi4BAYGStWqVWXnzp1Of5/k5PvyxdyP5d1x02T8h5/Kqm+/lIQb8RbLNGz2iAwZ+b7V1/+5Z4f4+Lh2c7l7GVk+795+nlBGdy+f10ZBzcHJA+To3hAXFycNGzaU3Llzyw8//CAHDx6UKVOmSP78+Z3+XsePHJKixUtKWEQBCQgMkuq168qfu3dYLFO+UlXJGxKS5rX379+XlcsWyVMduzq9XJ5URpbPu7efJ5TR3cvnrYkxmoP/2WvGjBlSokQJCQgIkLp168r27dvTXXbu3LnSuHFjFS8wtWjRIsPl3TIITpw4UaKjo+Wzzz6TOnXqSMmSJaVly5ZSunRpp79X3LWrEhYeYXocFl5A4q7G2PTaNcuXSZPmj0pgoGvvLO3uZWT5vHv7eUIZ3b185Lhly5bJ4MGDZcSIEbJ7926pXr26tGrVSq5cuWJ1+Q0bNkinTp1k/fr1smXLFhVLED/QuugxQfD777+X2rVrS/v27aVgwYJSs2ZNFd3Tk5iYKDdu3LCYXO1abIxqQmnS4lFxV+5eRpbPu7efJ5TR3cvn7tmhmoOTPaZOnSq9e/eW7t27S6VKlWTWrFkSFBQk8+fPt7r84sWLpV+/flKjRg2pUKGCzJs3T1JSUmTdunWeEwRPnDghM2fOlLJly8qPP/4offv2lVdeeUUWLlxodfnx48dLaGioaULkt1X+sHC5djXW9Pja1RjJb3ZGmZ5TJ47J+TOn5JUeHWTkGwPk7KkTMnHEG+IK7l5Gls+7t58nlNHdy+dtNCd0CaauuKAyk1pSUpLs2rVLNWkaoe8Wj1HLs8Xt27fl3r17EhYW5jnZoYjaqAmOGzdOPUZN8MCBA+oMoFu3bmmWHzp0qKouG2GD2hoIS5evKOdOn1BnhEF58si+ndvk6Y5p3yO1B+rUl5mLVqi/Yy5flA/GDZchoyaJK7h7GVk+795+nlBGdy+f19Gynh6a+hiN5s6RI0dazIuNjZXk5GQpVKiQxXw8Pnz4sE1vN2TIEImKirIIpG4fBAsXLqyqveYqVqwo3377rdXl/f391eQIX99c0qVnfxkzdJAYDAZp+0wnyRsSqs4Ge78yRPUzzJw6Tv7cvV0SEm5I/67PyAu9B0i9xg859H7eWEaWz7u3nyeU0d3LR2mdPXtWQswSlRw9hmdkwoQJsnTpUtVPiKQae2gG7Ek5pHPnzmoD/f7776Z5r732mmzbtk02b96c6etRE0Sz6O6jFyU4b9psMCIiZ4kOd9+EGhwLC4WHSnx8vEXAycr6QkNDZdNf5x0+tt5MuCENKxexqUxoDkX/3zfffCPt2rUzzUeL4PXr12XlypXpvnby5Mny3nvvyS+//KJaFu2Vo32CCHhbt25VzaHHjh2TJUuWyJw5c6R///45WSwiIpLsS4zx8/OTWrVqWSS1GJNc6tevn+7r3n//fRkzZoysXbvWoQCY482hDz74oCxfvlz19Y0ePVoNkZg2bZp06dIlJ4tFRESSvVeMQb4Han4IZhgyh1hw69YtlS0KXbt2lSJFiqgESeMQu+HDh6vKE8YWXrp0Sc0PDg5Wk8dcNu3xxx9XExER6VeHDh0kJiZGBTYENAx9QA3PmCxz5swZi6v9YGQBmlGfffbZTBNv3DoIEhGRm9Ky9+KhAwYMUJM1SHoxd+rUKXEGBkEiIrKK9xMkIiLd0ng/QSIiIu/F5lAiItLt/QQZBImISLdRkEGQiIh0mxjDWywTEZFusSZIRES6zQ5lECQiIr12CTIIEhGRfqMg+wSJiEi32BxKRES6zQ5lECQiIuuykBjjITHQO4Kgj48mvj4essWJyCNdiLsj7iohwTVl07y/S5B9gkREpF9eURMkIiIX0Ly/KsggSEREVjExhoiIdEvTwRVjOE6QiIh0i82hRESk1y5BBkEiItJvFGRNkIiIdJsYwz5BIiLSLdYEiYgo/dZQR7NDxTMwCBIRkV67BBkEiYjIOo4TJCIi8mJsDiUiIt02iOZoECxRooScPn06zfx+/frJjBkznP5+u7Ztks/nzhCDwSBPtu8szR9ta/H8vBlTZdsfGyS8QEGZ8OE80/zpE0fJiaNHJFeuXFKrbgPp3L2P08vmCeXzhDKyfNyG3AedR+Nl01xrx44dcvHiRdP0888/q/nt27d3+nslJ9+Xz+d+LCMmTJf3P/5Uvv/mS0m4EW+xTKNmLWTo6ElpXtu0+aMybe5imfjxfDly8IAc2LtLd+XzhDKyfNyG3AddUw/UHJw8QY6OEyxQoIBERkaaplWrVknp0qWladOmTn+vY0cOSdFiJSUsooAEBAZJzdr1ZN/uHRbLVKhcTYJDQtK8tkbtuqJpmqrFlChdVq5djdFd+TyhjCwftyH3QdfUBDUHJ0/gNoPlk5KSZNGiRdKjRw91sHS2uKux6uBtFBYRIddi7TsQ37l9W3Zv3yKVqtXUXfk8oYwsH7ch90Hy2MSYFStWyPXr1+XFF19Md5nExEQ1Gd24cSObSieqD2zG1HHSsk07iShQSNyNu5fPE8rI8nEb5jR32wc1XjYt+3z66afSunVriYqKSneZ8ePHS2hoqGmKjo62ef35wy1rLddiYyUsPMLm1y+eP1OCg4Ol7TMdxRXcvXyeUEaWj9uQ+6CTad7fKegWzaHIEP3ll1+kV69eGS43dOhQiY+PN01nz561+T3KlK8oZ0+fVAfxu3duy56dW6V6rTo2vfan1Svk5PGj0mvA6+Iq7l4+Tygjy8dtyH3QuTTvj4GiGVD/zmEjR46U2bNnq6CGxAlboTkUNcK9xy9J3rxpkzFS27n1D/li3gxJSTHIk892lhaPPSHjh70hL786RNVoZkwZK/t2bZeEBKw3v3R7eaDUb/yQdGzTTApGFhb/gAC1nseefFYeatkmS5/ZE8vnCWVk+bgN9bgPYl01SkeqykGIlcQ0e93499h69Gys5HVwfQk3bkjZ6Ainlclrg2BKSoqULFlSOnXqJBMmTLDrtfYGQSIib+SqIHjsXNaCYJmi7h8EczwxBs2gZ86cUVmhRETkPjQdJMbkeBBs2bKlyogiIiI3o3n9VdPcIzGGiIhIlzVBIiJyT5r3VwQZBImISL8X0GZNkIiI0uF4Yoyn1AXZJ0hERLrFmiAREem2OZQ1QSIi0i3WBImIyCrWBImIiLwYa4JERGQVL5tGRES6pekgMYY1QSIi0u0VY5gdSkREuuXRNUHj3SduJiTkdFGIiHKM8Rjo9DvyaN5fFfToIJjw7xffqEbZnC4KEZFbHBNxM1xn0Xg/QfcWFRUlZ8+elbx584rmhF5Y3E05OjpardOd74TszrgNuQ1zmh73QdQAEQBxTHQmjYkx7s3Hx0eKFi3q9PXih6OXH4+rcBtyG+Y0ve2DzqwB6gkTY4iIKMMuQUcne82YMUNKlCghAQEBUrduXdm+fXuGy3/99ddSoUIFtXzVqlVlzZo1dr8ngyAREeV4FFy2bJkMHjxYRowYIbt375bq1atLq1at5MqVK1aX37x5s3Tq1El69uwpe/bskXbt2qnpwIEDdr0vg6AZf39/9QXgX3IMt2HWcRty+7kLLYv/2WPq1KnSu3dv6d69u1SqVElmzZolQUFBMn/+fKvLT58+XR599FF54403pGLFijJmzBh54IEH5OOPP7bvMxqcnlNLRESenlwUGhoql2LjHe5XxToiI0IlPj7zdSQlJamA980336janFG3bt3k+vXrsnLlyjSvKVasmKo5vvrqq6Z5qMSsWLFC9u3bp48hEkRE5DoJCTccvvwZXmsMhqlbOlK3tsXGxkpycrIUKlTIYj4eHz582Or6L126ZHV5zLcHgyAREVnw8/OTyMhIKVsyWrIiODhYDVcxh9rayJEjxV0wCBIRkQVkW548eVI1U2YFettSj+G2lnMREREhvr6+cvnyZYv5eIxgbA3m27N8epgYk4X0XPrP+PHj5cEHH1QXLihYsKBq1z9y5Ag3kYMmTJigDh7m/R2UufPnz8vzzz8v4eHhEhgYqNLmd+7cyU3ngICAANNYS0cn9CumnmctCKLmWatWLVm3bp1pXkpKinpcv359q+XDfPPl4eeff053+fQwCDqYnkuWNm7cKP3795etW7eqHfHevXvSsmVLuXXrFjeVnXbs2CGzZ8+WatWqcdvZIS4uTho2bCi5c+eWH374QQ4ePChTpkyR/Pnzczt6gMGDB8vcuXNl4cKFcujQIenbt686fiBbFLp27SpDhw41LT9o0CBZu3at+o7Rb4gmVpzwDBgwwL43RnYoGQx16tQx9O/f37QpkpOTDVFRUYbx48dz8zjgypUryDo2bNy4kdvPDgkJCYayZcsafv75Z0PTpk0NgwYN4vaz0ZAhQwyNGjXi9vJgH330kaFYsWIGPz8/dUzeunWr6Tn8Hrp162ax/FdffWUoV66cWr5y5cqG1atX2/2eHCLhYHouZezYsWNStmxZ2b9/v1SpUoWby0bY58LCwuSDDz6QZs2aSY0aNWTatGncfjbA2DK03pw7d061TBQpUkT69eunxp4RpYfNoZmk59qbbkv/tOWjLwtNUwyAtlu6dKlqikf/KtnvxIkTMnPmTHXy9eOPP6rmtFdeeUU1rxGlh9mh5HToG8Sli/744w9uXRvhjgfo40B/KhISyLGTr9q1a8u4cePU45o1a6r9EFceQQ2byBrWBB1MzyXr0Cm9atUqWb9+vUvu8OGtdu3apZKwcNmnXLlyqQlNeh9++KH6Gy0VlLHChQurJlFzuJzWmTNnuOkoXQyCDqbnUtrxQAiAy5cvl19//VVKlizJTWSH5s2bq/7TvXv3mibUarp06aL+xkkaZQzN76mH5fz9999SvHhxbjpKF5tDzdJz0WSCA0+dOnVUMoJ5ei5l3gS6ZMkSlUSEsYLGvlSME8J4LcoYtlnq/tM8efKo8W7sV7XNa6+9Jg0aNFDNoc8995wa5ztnzhw1EaWH2aFmcPXxSZMmqQM4svLQFIVB85S51FeFMPrss8/kxRdf5CZ0ALND7YemeIwlO3r0qGqNwMkts0MpIwyCRESkW+wTJCIi3WIQJCIi3WIQJCIi3WIQJCIi3WIQJCIi3WIQJCIi3WIQJCIi3WIQJCIi3WIQJHKxDRs2qCvq4N6U2alEiRK8FyFRJhgESfdOnTqlghQuVO2OwcxRO3bskJdeeimni0Hk1ngBbSIvVaBAgZwuApHbY02QPOJC0rhNEybclQL3fxw2bJi6fZPRF198oe4Agrsx4B6QnTt3VvfnM4qLi1O3JUJgwF0tcPdxXNwbjLd9wk1YUdPD+1mrLT700EPq7/z586vljBcGT0xMVHcwL1iwoLohbqNGjVQtLD23b9+W1q1bq1v/GGuV8+bNU/e+w+srVKggn3zyicV74/2+++47VYagoCCpXr26bNmyJcPtxuZQoswxCJJHWLhwobq5LG6PM336dJk6daoKHEb37t2TMWPGyL59+2TFihUqcJjfvQJB8+DBg/LDDz/IoUOHZObMmSqYAtYJv/zyi1y8eFEFm9Sio6Pl22+/VX/jnnVYDuWAN998Uz2HMu7evVvKlCkjrVq1kmvXrqVZD4LeI488ou5XibvI58uXTxYvXizDhw+XsWPHqrLhVkAoL9Zn7p133pHXX39dNduWK1dOOnXqJPfv33faNibSJQORm2vatKmhYsWKhpSUFNO8IUOGqHnp2bFjB6qJhoSEBPW4bdu2hu7du1td9uTJk2rZPXv2ZFiO9evXq+Xi4uJM827evGnInTu3YfHixaZ5SUlJhqioKMP7779v8bpDhw4ZqlWrZnjmmWcMiYmJpuVLly5tWLJkicV7jRkzxlC/fn2L8s2bN8/0/F9//WVaZ3qKFy9u+OCDDzL8TER6x5ogeYR69epZ3LOwfv366p5xycnJ6vGuXbukbdu2UqxYMdUk2rRpUzX/zJkz6t++ffvK0qVL1X0iUXPbvHmzU8p1/PhxVQtF06ZR7ty51Y2ZUaszhxogaonLli0TPz8/NQ83bsY6evbsKcHBwabpvffeU/PNVatWzfR34cKF1b/mTb5EZD8GQfJ4CCRofgwJCVFNi+iPW758uXouKSlJ/Ys+uNOnT6u7j1+4cEGaN2+umhazU5s2beS3335TzbJGN2/eVP/OnTtXNXMapwMHDsjWrVstXo/gamQ8IUCzKhE5jkGQPMK2bdssHiNAILnF19dXDh8+LFevXpUJEyZI48aNVWKJtRoSkmK6desmixYtUuPn5syZo+Yba2XGWmV6rC1XunRpNX/Tpk2meagZIhBXqlTJ4vUoH94fAdgYCAsVKiRRUVFy4sQJVUs0n4wJO0TkOhwiQR4BzZqDBw+Wl19+WSWffPTRRzJlyhT1HJpAEYgwr0+fPqoWhSQZc0g8qVWrllSuXFllc65atUplYwKyOpExunbtWilatKjK0EQWamrFixdXNTC89rHHHlOvQdMlmlrfeOMNCQsLU2V5//33VQYomjhTmzx5sgqiDz/8sBp3iIA9atQolV2K93z00UdV+Xbu3KkyWvGZiciFcrpTksiWxJh+/foZ+vTpYwgJCTHkz5/f8Pbbb1skyiCxpESJEgZ/f3+VUPL9999bJLsg0QSJNIGBgYawsDDDk08+aThx4oTp9XPnzjVER0cbfHx81PulZ/To0YbIyEiDpmmGbt26qXl37twxDBw40BAREaHev2HDhobt27dnmFCD5QsXLmw4cuSIeozEmho1ahj8/PzU52vSpInhu+++SzdxB+vCPKw7PUyMIcqchv+5MsgSZRXG7SGhBU2YZDskz6BG3KtXL242onSwOZTIy6ApFn2Uly9fVs2/RJQ+JsYQeRkk/HTs2FFeffVVNZSEiNLH5lAiItIt1gSJiEi3GASJiEi3GASJiEi3GASJiEi3GASJiEi3GASJiEi3GASJiEi3GASJiEi3GASJiEj06v8BgKebVyLVB38AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 520x440 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The (T, T) averaging weights as an annotated heatmap.\n",
    "fig, ax = plt.subplots(figsize=(5.2, 4.4))\n",
    "im = ax.imshow(wei, cmap=\"Blues\")\n",
    "for i in range(T):\n",
    "    for j in range(T):\n",
    "        if wei[i, j] > 0:\n",
    "            ax.text(j, i, f\"{wei[i, j]:.2f}\", ha=\"center\", va=\"center\",\n",
    "                    fontsize=7, color=\"white\" if wei[i, j] > 0.5 else \"#333\")\n",
    "ax.set_xlabel(\"past token j\"); ax.set_ylabel(\"current token t\")\n",
    "ax.set_title(\"step-2 averaging: an equal share of the past\")\n",
    "fig.colorbar(im); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m5c1c",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>ax.imshow(wei, cmap=\"Blues\")</code>: draw the 8&times;8 weight table as an image; darker means a larger share.</li>\n",
    "<li>The double loop writes each nonzero weight into its cell, so you can read the exact numbers off the picture.</li>\n",
    "<li>Row 0 is a lone <code>1.00</code> (token 0 has only itself); row 1 is <code>0.50, 0.50</code>; row 7 is eight equal <code>0.12</code>s.</li>\n",
    "<li>The blank upper triangle is the causal rule: no token draws anything from the future. Attention keeps this exact triangular shape but replaces the equal shares with learned ones.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8091ae3c",
   "metadata": {},
   "source": [
    "## Step 3: let the data set the weights, which is attention itself\n",
    "\n",
    "Equal averaging is crude: a pronoun should lean heavily on its noun, not on every random word. So we let the **data** set the weights. The clearest way to picture this is a search engine, in which each token produces three things:\n",
    "\n",
    "- a **query** `q`, meaning \"what am I looking for?\" (like the words you type into a search box),\n",
    "- a **key** `k`, meaning \"what do I contain?\" (like the tags a document advertises),\n",
    "- a **value** `v`, meaning \"what I will pass on if you attend to me\" (like the document's actual content).\n",
    "\n",
    "These are three different jobs, so each gets its own learned projection (`W_q`, `W_k`, `W_v`, which are just matrices the model trains). To score how much token `i` should attend to token `j`, take the **dot product** `q_i . k_j`, the query-meets-key similarity from notebook 01. A high dot product means a good match, so the token pays more attention there.\n",
    "\n",
    "The rest is machinery you already know: hide the future by setting those scores to negative infinity, apply softmax to each row so the weights add up to 1, and return the weighted sum of the values. The whole thing is one compact formula:\n",
    "\n",
    "$$\\text{attention}(Q,K,V) = \\text{softmax}\\!\\left(\\frac{Q K^{\\top}}{\\sqrt{d}}\\right) V$$\n",
    "\n",
    "In plain words, reading from the inside out: match every query against every key ($QK^{\\top}$, where $K^{\\top}$ just means the table of keys turned on its side so the multiplication lines up), tone the scores down by dividing by $\\sqrt{d}$ (explained in the next cell), turn them into weights with softmax, and use those weights to blend the values ($\\cdot\\, V$). Unlike the bigram, the token itself decides where to look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4c6d4d67",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.082743Z",
     "iopub.status.busy": "2026-07-01T09:02:50.082688Z",
     "iopub.status.idle": "2026-07-01T09:02:50.085201Z",
     "shell.execute_reply": "2026-07-01T09:02:50.084838Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attention output: (4, 8, 16)\n",
      "causal? upper triangle is zero: True\n",
      "rows sum to 1: True\n"
     ]
    }
   ],
   "source": [
    "head_size = 16\n",
    "# learned projections (random here; in the GPT these are trained)\n",
    "W_q = np.random.randn(C, head_size) * 0.1\n",
    "W_k = np.random.randn(C, head_size) * 0.1\n",
    "W_v = np.random.randn(C, head_size) * 0.1\n",
    "\n",
    "q = x @ W_q          # (B,T,head_size) - queries\n",
    "k = x @ W_k          # (B,T,head_size) - keys\n",
    "v = x @ W_v          # (B,T,head_size) - values\n",
    "\n",
    "# affinities: every query dotted with every key\n",
    "scores = q @ k.transpose(0, 2, 1)            # (B,T,T)\n",
    "scores = scores / head_size**0.5             # scale (step 4, below)\n",
    "\n",
    "mask = np.tril(np.ones((T, T)))              # causal mask\n",
    "scores = np.where(mask == 0, -np.inf, scores)  # future -> -inf -> 0 after softmax\n",
    "attn = softmax(scores, axis=-1)              # (B,T,T) each row sums to 1\n",
    "out = attn @ v                               # (B,T,head_size) weighted sum of values\n",
    "print(\"attention output:\", out.shape)\n",
    "print(\"causal? upper triangle is zero:\", np.allclose(np.triu(attn[0], k=1), 0))\n",
    "print(\"rows sum to 1:\", np.allclose(attn[0].sum(1), 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd46bc30",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>head_size = 16</code>: each attention head will boil a token down to a 16-number query/key/value.</li>\n",
    "<li><code>W_q = np.random.randn(C, head_size) * 0.1</code> (and W_k, W_v) -- the three learned projections. Multiplying a token's 32 numbers by a (32,16) grid gives its 16-number query / key / value. (Random here; in the GPT these are trained.)</li>\n",
    "<li><code>q = x @ W_q</code> (and k, v) -- apply each projection to every token in every sequence at once: shape (B, T, 16).</li>\n",
    "<li><code>scores = q @ k.transpose(0, 2, 1)</code>: <code>transpose</code> swaps the last two axes so the matmul lines up; the result is <b>every query dotted with every key</b> -- a (B, T, T) grid of match scores.</li>\n",
    "<li><code>scores = scores / head_size**0.5</code>: divide by &radic;16 = 4: the scaling trick (next cell shows why).</li>\n",
    "<li><code>mask = np.tril(np.ones((T, T)))</code>: the same lower-triangle mask: 1 where attending is allowed, 0 for the future.</li>\n",
    "<li><code>scores = np.where(mask == 0, -np.inf, scores)</code>: wherever the mask is 0, overwrite the score with negative infinity; everywhere else keep it. After softmax, -inf becomes exactly 0 weight.</li>\n",
    "<li><code>attn = softmax(scores, axis=-1)</code>: turn each row of scores into weights that sum to 1.</li>\n",
    "<li><code>out = attn @ v</code>: blend the value vectors by those weights: each token's output is its personalized weighted sum of the past.</li>\n",
    "<li><code>np.triu(attn[0], k=1)</code>: the strictly-upper triangle (future positions); checking it's all zeros proves no information leaked backward in time.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m5c2a",
   "metadata": {},
   "source": [
    "### Open up one row of the attention matrix\n",
    "\n",
    "`attn` is a `(B, T, T)` block, which is a lot to take in at once. So follow a single query token all the way through -- token `t = 4` in the first sequence -- and watch its score against every other token become a set of weights. The recomputed weights match row 4 of the full `attn`, and their weighted sum of values matches `out` for that token, so this one row is the whole mechanism in miniature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "m5c2b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.086167Z",
     "iopub.status.busy": "2026-07-01T09:02:50.086109Z",
     "iopub.status.idle": "2026-07-01T09:02:50.112582Z",
     "shell.execute_reply": "2026-07-01T09:02:50.112249Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token t=4: query shape (16,), scored against 8 keys\n",
      "raw scores  q.k/sqrt(hs): [ 0.31  0.08 -0.38  0.45 -0.41 -0.13  0.17 -0.32]\n",
      "after causal mask       : [ 0.31  0.08 -0.38  0.45 -0.41  -inf  -inf  -inf]\n",
      "after softmax (weights) : [0.255 0.202 0.128 0.292 0.124 0.    0.    0.   ]  sum = 1.0\n",
      "matches row 4 of attn   : True\n",
      "blended value == out[0,4]: True\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAEdCAYAAADaR7fMAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQQVJREFUeJzt3Qd0FGX7NvAbAoTeDL2FJiH0LggiRYqogMpLEanCK0V6MUiTIk2QKr2KUixgeZFeBAkgINIRBKQnhF4kIMx3rvs7s//dzSbZTTbZ7Ob6nTOQnZ2dXu55agrDMAwhIiIiSmQpE3uBRERERMAghIiIiDyCQQgRERF5BIMQIiIi8ggGIUREROQRDEKIiIjIIxiEEBERkUcwCCEiIiKPYBBCREREHsEgxAeNHDlSUqRIIREREeKLlixZotu3f/9+T69KknL+/HndL9g/9udCYhwPLJ9iFxgYKB06dOCuigPst4wZM3Lf+RAGIZToPvnkE1m7dq1P7vldu3bpAzkpB4GJsf8///xzm2AosVy5ckUDr0OHDkX57quvvpKpU6cmynrs3r1b1+P27duJsjxf8vDhQ91327dvl+TkSgznri9jEEKJzleDkGfPnskHH3wgGTJkkKRi6NCh8s8//yTo/n/33Xd1GYUKFUoSQcjHH3+cJIIQrIejIOTUqVMyf/78RFkPbw1CsO+SYxDycTTnri9jEELkJvPmzZOLFy/Ke++9l2T2aapUqSRt2rQJugw/Pz9dRkJn+/gKf39/SZ06tSRnjx490qCdSNCLLvmWESNGoGdk4/Tp00b79u2NLFmyGJkzZzY6dOhgPHjwwGbaJ0+eGKNGjTKKFClipEmTxihUqJAREhJiPHr0yDJN3759jezZsxvPnj2zjOvZs6cuY9q0aZZx165d03Gff/55tOuG7+0HrKPp4MGDRqNGjYxMmTIZGTJkMOrWrWuEhobazGPx4sX6u99++80y7ubNm0aVKlWMfPnyGSdPntRx2Ibhw4cbRYsW1W3Lnz+/MXDgQJttM9epR48expo1a4xSpUrptMHBwcbPP//s9D6/ceOG8dxzzxmzZs2y7P/r16/H+ru7d+8avXv31v2O5ebIkcOoX7++ceDAAcs0tWvX1vXav3+/Ub16dSNt2rRGYGCgMXv2bJt5nTt3TpeL/WMy18WZ/e/MujhiHg8sH/B7+2VgG+Dx48fGyJEjjWLFihn+/v56Xr344ovGxo0bY92//fv3N0qXLq3nBc4PnCeHDh2yTLNt2zaH24f1w/Ltx2M9Te48V8x9bj9Y7x/rcx7++usv4+233zayZctmpEuXzqhWrZrx008/2Uxjbt+qVauMMWPG6LmOfYhrBNe6M2K7vnBNYRlLliyJ8tv169frdz/++KNl3KVLl4yOHTsaOXPmtOyLhQsXOlzvFStWGB999JGRN29eI0WKFMatW7eiLMM8h+0H7FPAfsN6Y7lNmzbVvwMCAvTc+Pfff23m9fTpU+Ozzz7TdcJ+wjp27dpV7xWxMZeD49KgQQMjffr0Rp48eYyPP/7Y5j4IkyZN0usS5zKuzYoVKxpff/11lHniHMe5jvsx5v3888/rvTa2c9fXMQjxQeZNsEKFCsabb76pQcF7772n4wYNGhTlYsN43ADxAG3Xrp1+btasmWWa7777TscdOXLEMq5cuXJGypQp9XcmXHiY7ujRo9Gu2xdffKE3hFq1aunfGHbv3q3f4Xe4OHGxjx492hg/frxRuHBhnX7Pnj3RBiF42JcvX94oWLCgcebMGcsNyLx59OnTx5g7d64GTqlSpdKblzXMC9tjLnfq1KkalOG3ERERTu3z7t2760MJN0JXgpA2bdrozbtfv37GggULjAkTJhivv/66sXz5css0eIDixo2bKLZh+vTpRs2aNXUZ1jd8Z4KQmPa/M+viTBCCBzQe4kFBQZZlmEHGkCFD9AHUpUsXY/78+cbkyZON1q1b67GOCY41AoQPP/xQjyUCZzyEcUO/fPmyJQjGeKwLHjbmsvEgwfJxjuCBZY7HeibEufLHH3/oNmFaPATN5d2/f99hEIL1zpUrlwYGeEhPmTLFcn3h2jOZDypc15UqVdJ5I6DDsqtWrWrExtnrC9vz6quvRvk9gg0ESQgkzfXGcS5QoIDudwTFb7zxhmW77dcbwQCOAbZv3LhxUV6IAPsI88H0zZs3t+w77FPAfsODHtdap06ddNq33nrL4csP7nk4hjjX5syZYwwePFi3Hy8r5jZEx1xO8eLFjXfffdeYOXOm8dprr+lyhg0bZjMt9gGuf0yDbcOxwHTWQST2Pa6typUr64sb1mfAgAHGSy+9FOu56+sYhPgg88GDi9QaLmq8rZvwFonpcLFaw8WB8Vu3btXP4eHhNhf57du39QbZokULvXmaevXqFSXFxBHcCOzfBAGBDy5U6wvvypUrenM2L1b7IOTq1at6Q8KN8/z585ZpcAFjHXfu3GmzDFz8+O2vv/5qGYfPWK4ZwABuehg/Y8aMGLfFnNbPz8/YsGGDfnYlCMFDFG/WMTHf4vHANkVGRuoNHYGJeUN1JgiJaf87sy7OBCGAY2KmfljDw7VJkyYuLwMpEggWrGF5eIDi5m3/Ju/oDRLLtU79SMhzBW/H9vvEZB+EIPDBtNbLv3fvngYISPEyt9t8mJcsWVKPvwkPNfuXBEecvb7wdp46dWqbFAMsL2vWrDb3lM6dO2tAYx+ot2rVSs+lhw8f2qw3rlFzXExw3Vinfjh6abI+5mAGZibsS0z35ZdfOkzNsR8f3XI++OADyzjc13AOYR9aX9v224TrESl2SGUyISiL7Z7wWwznri9jmRAf9v7779t8rlWrlty4cUPu3r2rn9etW6f/9+vXz2a6/v376///+9//9P8cOXJIUFCQ/PLLL/r5119/1XIAAwcOlLCwMDl9+rSO37lzp9SsWTNOZQOePn0qGzdulGbNmkmRIkUs4/PkySNt2rTRWifmepsuXboktWvXlidPnui6WReM/Prrr6VkyZK63qilYg5169bV77dt22Yzr/r160vRokUtn8uWLSuZM2eWs2fPxrruvXr1ksaNG0uDBg1c3u6sWbPK3r17tVBabGU7/vvf/1o+p0mTRj+Hh4fLgQMHXF5ufNYlvss4duyY5ZxxpRxFypQpLecKzmNU1SxRooQcPHgwXuuUmOeKI7gOq1atqteOCdvWtWtXrfZ8/Phxm+k7duyox9/6uoaYlu/K9dWyZUu9pr777jvLdPgtCtniO0A89u2338rrr7+uf1vvt4YNG8qdO3eiHJf27dtLunTpJKHubdbbj2OaJUsWeeWVV2zWrVKlSrpv7Y9pdHr27Gn5G/c1fH78+LFs3rzZMt56m27duqXbjvWx3n6c9/D999+zLIwdBiE+rGDBgjafs2XLZrlQ4O+//9Ybe7FixWymy507t140+N6EiwpBBuD/ypUr65A9e3b9jBvYH3/8Ybkhuur69etaKh4PFXt4QKAQGwp92tfKwEN4x44dki9fPpvv8JDDww4BlPXw/PPP6/f4XUz7ytxf5r6KzqpVq7QmxOTJkyUuJk6cKEePHpUCBQrogwhV9Bw9TPLmzRul1o25Le5qn8PZdYmPUaNG6cMM616mTBkNZA8fPhzr73D8P/vsMylevLgGJAEBAXo88Vvc9OMjsc6V6OA6i+68N7935bqO7/VVrlw5DchwbpvwN/a5GZhhfjiOKIxtv98QJDnab4ULFxZ3QCFoLCem/Y9jivMiZ86cUdbv/v37UdbNEdwbrQO26K65n376SV544QVdL9wPsYzZs2fbnJcI3l588UUttJ4rVy5p1aqVrF69mgEJXrDiczJQ0obUCkf+f6ry/3Em5QJvaahWiIcSgg4EG/gdxuMzHpK4kcU1CImLN998U5YtWybTpk2TcePG2XyHdcFDbsqUKQ5/iwdtXPaVPTxEW7RooW+m5o3JrJaJmzremrBvovOf//xH99maNWv0bXPSpEkyYcIEfQtF6kpiSox1eemll+Svv/7SN0IsY8GCBRpczJkzJ8ZaRahWPGzYMOnUqZOMHj1ab/Z4SPTp0yfeN/LEOlfcJTGWj4fm2LFjNfUgU6ZM8sMPP0jr1q01RQ7Mfd62bVtN4XAEKUTW3JUKEt32W8P6IQD58ssvHX5vH8TEFe59b7zxhp7XqJaOlCXUfFq8eLFWCbfedqTWIgUGKczr16/XwK5u3bp6HTizTb6KQUgyhuwLXKx4azDfugBZLHiQWmdvmMHFpk2b5LfffpMPP/xQP+PiQ9RvvqkjuTM2joIe3BTSp0+vbSjYO3nypD5w7B8GaJMDqTjDhw/XpFdznQDJ5UiZqVevXoJWHUWggZuN9Q3HVLFiRX2rjK3eP25c3bt31wFvaPgdHgDWD35kkTx48MAmNeTPP/+0tMDpipj2hzPrEt9lIIDA2zIGvJXiHEKqS0xByDfffCN16tSRhQsX2ozHeYo3dGeWG913CXGuuDIfXGfRnffm9/Hl6vWFIARtViDLBW/uSOnE27v1/BCcIJsH2VPu5I5jgGOKLBOkPsQ1+MG9ES9dZuqHo2sO+wcpIBs2bNAUOhOCEHvYxzjHMCDgRWD90UcfaWCCfZhcq7gzOyYZe/XVV/V/+waczDfCJk2a2CSlIssDb63IL8bFbQYneLPFQwJJkuabUkzwILVvxAlvAihTgTdk66ROBER4wCPFBfnu9vB2PGDAAAkJCdFgyPqt/vLlyw4bhULDWniguwNSDewHM98cqTTYX9HBDdw+KwFvbwjoIiMjbcb/+++/MnfuXMtnpLDgMx4GzgR+se1/V9YlrssAlOWwhvx5BJKxLQPnh/2bPvL9cYztlwuOlo3vHGXdJMS5EtN6OLoO9+3bJ6GhoZZxWCayOvCwCw4Olvhy9frCSwlSh/C2jgHBKYJF6/m99dZb+hBGFp49ZNfEFYIliE9rszimOKeRamYP15Kz8545c6blb5x/+IyUDgQS5n5A8IBlmbB/7RsDvHnzZpR5ly9fXv83z31XzhlfwpSQZAxv6UhKxc0OJz4KeeJmuHTpUi3AhjdPawg4Vq5cqTcnMx8ab8q4ePCGgAJuzsBDE28pCHbwkEOAU61aNRkzZoymtOCGiDdxBDR40OIiRXmF6CDbAA+XHj166NsZkohRXgR5rijAhjcNBE24UeCtD+Px5oIyLfGF/WTPTPlA6oH1W7q9e/fuSf78+eXtt9/WY4EHMvYLUprsy5hgPyFrBDc4vJnhwYDl4Ni52vCVo/2PsgLOrouzy0BQiGOKIAMBDZKe8UB9+eWX9XukiKD/HwSw1gUAHXnttde0PAlST2rUqCFHjhzRpHb7PHu8AaM8E7J3cC7g3MS5hW3EMrHfUBC7SpUquo0oWJkQ54oZGOJNFykIOEZYlqPWdJGCt2LFCj1fUMgZ+wXX4Llz5/QhbxbIjS9Xry8E00hlxJt+586do6zH+PHjdX9h/3bp0kWPLR62KJCJc8fRg9cZSLnAvHCscK5jf5QuXVoHZ+FehoLbyKbFdYIADMcAqb4IXpGFi3M9JthuZJvgHolt/PnnnzUrZciQIZbsHLyo4Tpq1KiR3v+Qejhr1iw9563LOuHcRXYMpkfKFqZD9k3+/PktBZJjOnd9mqer55D7RVdF1FFVSjRWhgZ4UB0Q1fJQ59++sTIT2hHB77t162YzHg1aYfyWLVucWj80JoYqgWiUyVFjZQ0bNjQyZsyo7R/UqVPH0o5FTI2VoRoj2mZAuwBr1661VJVDWxeoLoqqnGjjANX4sL137tyJ0gCVPUeNSjnD2Sq6qPaIBrFQbdVsPAp/27d34KixMqwb2iWw5mwVXUf739l1ccTReYV2D1CdEfOybqwMjWyhHQVU98Ty0ZbI2LFjY223AecjGqRClVD8Do0+oZEtzNe+KvD333+vbVLgXLDeH2iDAm2hYNn2jZUlxLmCtjjQlgmq/zrbWBnWDccX+yi6xsrsG8JydNyj48z1ZUIDaGajWbt27XI4TVhYmO4P3Ddw/8idO7dRr149Y968ebGud0ywTtj/qA7rqLEye47Oc8B6YD44Z3AulilTRttKQtVkVxsrQ3MEWI59VXG01YP2RHDe4HzGcbBfH9wb0eYM2vvBNuF/3K/+/PNPp85dX5YC/3g6ECKi6CHlAAUEHSV7E1HC9NaLFDqUWaKExTIhRERE5BEMQoiIiMgjGIQQERGRR7BMCBEREXkEU0KIiIjIIxiEEBERkUewsbJomutFM9loMCa5NqVLREQUF2j5A40xojHEWBvbM5IANLqEBnzQ2Asa6dm7d2+003777bfa+EyWLFm0ARk0qLRs2TKbaZ49e2YMGzZMG85Bwz9oPMe+UZiYXLx40dJIDwfuA54DPAd4DvAc4DkgLu8DPEtj4/GUELMZZTRViyZq0Y9Jw4YNtaMlNPVsD034oilkdDWNnkvRjTKacsa0+B2gCeLp06dr08do8hb9i+C748ePa1O8sUEKiNk5maP+SoiIiMgxdHiIDhHNZ2mSrh2DwAP9OJgdBSErBCuPHlKte0WNCfovQZv86KwIm4MkoP79+2vHZoB+RdAT5JIlS2x6goxpB6JXVvyOQQgREZHzXHmGerRgKnoCPXDggE1X0Mg/wmfrHiWjg4Bjy5Ytmmpi9vCITp+uXbtmM0/sDAQ70c0THThhp1kPRERElLA8GoSgPwz0VolUCmv4jEAiOoiu0AMmsmOQAjJjxgx55ZVX9Dvzd67MEz0tIlAxB6TEEBERUcLyyiq6yGdC98zoZnzs2LFapmT79u1xnl9ISIgGNuaAsiBERESUsDxaMDUgIED8/PwkLCzMZjw+586dO9rfIcumWLFi+nf58uXlxIkTmpqB3kbN32EeefLksZknpnXE399fByIiIkomKSHITqlUqZKW6zChYCo+V69e3en54Dco1wGoDYNAxHqeKOOxd+9el+ZJRERECcvjVXSRldK+fXupXLmyVK1aVavoPnjwQKvdQrt27SRfvnya0gH4H9MWLVpUA49169bJF198IbNnz9bv0bhYnz59ZMyYMVK8eHFLFV3UmGnWrJlHt5WIiIiSUBDSsmVLuX79ugwfPlwLjiLLZP369ZaCpRcuXLBpcQ0BSvfu3eXSpUuSLl06bS9k+fLlOh/ToEGDdLquXbvK7du3pWbNmjpPZ9oIIUoO2rY9Id5i+fKSnl4FIkogHm8nJCliOyHk6xiEEJEk93ZCiIiIKPliEEJEREQewSCEiIiIPIJBCBEREXkEgxAiIiLyCAYhRERE5BEMQoiIiMgjGIQQERGRRzAIISIiIo9gEEJEREQewSCEiIiIPIJBCBEREXkEgxAiIiLyCAYhRERE5BEMQoiIiMgjGIQQERGRRzAIISIiIo9gEEJEREQewSCEiIiIPIJBCBEREXkEgxAiIiLyCAYhRERE5BGpJAmYNWuWTJo0Sa5duyblypWTGTNmSNWqVR1OO3/+fFm2bJkcPXpUP1eqVEk++eQTm+k7dOggS5cutfldw4YNZf369eJJbdueEG+yfHlJT68CERH5MI+nhKxatUr69esnI0aMkIMHD2oQgoAhPDzc4fTbt2+X1q1by7Zt2yQ0NFQKFCggDRo0kMuXL9tM16hRI7l69aplWLFiRSJtEREREXlFEDJlyhTp0qWLdOzYUYKDg2XOnDmSPn16WbRokcPpv/zyS+nevbuUL19egoKCZMGCBfLs2TPZsmWLzXT+/v6SO3duy5AtW7Zo1yEyMlLu3r1rMxAREZEPByGPHz+WAwcOSP369f9vhVKm1M9I5XDGw4cP5cmTJ5I9e/YoKSY5c+aUEiVKSLdu3eTGjRvRzmPcuHGSJUsWy4DUFSIiIvLhICQiIkKePn0quXLlshmPzygf4ozBgwdL3rx5bQIZZMWg3AhSRyZMmCA7duyQxo0b67IcCQkJkTt37liGixcvxnPLiIiIyCsKpsbV+PHjZeXKlZrqkTZtWsv4Vq1aWf4uU6aMlC1bVooWLarT1atXL8p8kHWDgYiIiJJJSkhAQID4+flJWFiYzXh8RjmOmHz66acahGzcuFGDjJgUKVJEl3XmzBm3rDcRERF5eRCSJk0arWJrXajULGRavXr1aH83ceJEGT16tFa5rVy5cqzLuXTpkpYJyZMnj9vWnYiIiLy8dgyq56LtD7TrceLECS1E+uDBA60tA+3atdMyGyaU8Rg2bJjWngkMDNSyIxju37+v3+P/gQMHyp49e+T8+fMa0DRt2lSKFSumVX+JiIgoafB4mZCWLVvK9evXZfjw4RpMoOotUjjMwqoXLlzQGjOm2bNna62at99+22Y+aGdk5MiRmr1z+PBhDWpu376thVbRjghSTljug4iIKOnweBACPXv21MERFCa1htSNmKRLl042bNjg1vUjIiKiJJAdg6qvaNzLHlIn8B0RERFRggQhKKuBtjTs3bt3z1KOg4iIiMjtQYhhGJIiRQqHNVDQ2igRERGRW8uEVKhQQYMPDGjwK1Wq//spWiI9d+6ctlRKRERE5NYgpFmzZvr/oUOHtKprxowZbdr7QHXZt956y9nZERERUTLndBCCKrCAYAPVaq2bSSciIiJK8Cq67du3t9SGCQ8P1xZOrRUsWNDllSAiIqLkx+Ug5PTp09KpUyfZvXu3wwKr0fVUS0RERBSvIKRDhw5aKPWnn37Svlgc1ZQhIiIicnsQgoKpBw4ckKCgIFd/SkRERBT3dkKCg4MlIiLC1Z8RERERuR6E3L171zKgF9tBgwZpny43btyw+Q4DERERkduyY7JmzWpT9gOFUNFgmTUWTCUiIiK3ByHbtm1zaaZEREREbglCateu7cxkRERERAlXO+bw4cMOxyO7Bq2oorEyf39/V2dLREREyYzLQUj58uVjbBskderU2qz73Llz2bQ7ERERua+K7po1a6R48eIyb948bTMEA/4uUaKEfPXVV7Jw4ULZunWrDB061NVZExERUTLickrI2LFjZdq0adqTrqlMmTKSP39+GTZsmOzbt08yZMgg/fv3l08//dTd60tERETJNSXkyJEjUqhQoSjjMQ7fmVk2V69edc8aEhERkU9yOQhBc+3jx4/XXnRNT5480XFmU+6XL1+WXLlyuXdNiYiIKHkHIbNmzdLO65D9Ur9+fR3wN8bNnj1bpzl79qx0797dpXkGBgZqQdZq1applk505s+fL7Vq1ZJs2bLpgOXbT4+G04YPH64d7KVLl06nQe+/RERE5MVBSI0aNeTcuXMyatQoKVu2rA74G+NeeOEFnebdd9+VgQMHOjW/VatWSb9+/WTEiBFy8OBBKVeunJY3CQ8Pdzg9motv3bq1NqAWGhoqBQoUkAYNGmjqi2nixIkyffp0mTNnjuzdu1fLqGCejx49cnVziYiIKIGkMJBs4EFI+ahSpYrMnDlTPz979kwDiw8++EA+/PDDWH//9OlTTRHB79u1a6epIHnz5tWCsQMGDNBp7ty5o9lDS5YskVatWsU6T/SBkyVLFv1d5syZxV3atj0h3mT58pLJert8mTcdMx4vIu/iyjPUqdoxP/zwgzRu3FjbAMHfMXnjjTecXlGUKzlw4ICEhIRYxqVMmVKzT5DK4YyHDx9qmZTs2bPrZ6TIXLt2Tedhws5AsIN5OgpCIiMjdTCxIz4iIqKE51QQ0qxZM32w58yZU/+ODhoxQ8qEsyIiInR6+0Ks+Hzy5Emn5jF48GBN+TCDDqynOQ/7eZrf2Rs3bpx8/PHHTq83ERERJVKZEGSRIAAx/45ucCUAcQfUyFm5cqU2oIZCrXGFlBgkG5nDxYsX3bqeRERE5IaCqdbiW9AzICBA/Pz8JCwszGY8PufOnTvG36IhNAQhGzdu1MKxJvN3rswTfd0g38p6ICIioiQWhCC1Y/To0ZIvXz7JmDGjVscFtJaKJttdkSZNGqlUqZJs2bLFMg4pKvhcvXr1aH+H2i9Yh/Xr10vlypVtvitcuLAGG9bzRBkP1JKJaZ5ERESUxIMQNNuOWiYIBBBEmEqXLi0LFixweQVQPRdtfyxdulROnDgh3bp1kwcPHkjHjh31e9R4sS64OmHCBA14Fi1apG2LoJwHhvv371vKpfTp00fGjBmjhWjRiivmgXIjMZVnISIioiTed8yyZcu0w7p69erJ+++/bxmP9j2cLUxqDT3uXr9+XRsXQzCBJt+RwmEWLL1w4YLWmDGhQTTUqnn77bdt5oN2RkaOHKl/Dxo0SAOZrl27yu3bt6VmzZo6z/iUGyEiIiIPByFoFKxYsWJRxiMbBVVl46Jnz546RNc4mbXz58/HOj+khqABNQxERETkI9kxwcHBsnPnzijjv/nmG6lQoYK71ouIiIh8nMspIcg2ad++vaaIIPXju+++k1OnTmk2DfqPISIiIkqQlJCmTZvKjz/+KJs3b9Y+WRCUoEApxr3yyiuuzo6IiIiSKZdTQgC92G7atMn9a0NERETJhsspIUj5QA+27JGWiIiIEjUIQSdwr7/+umTNmlVTRIYOHapZM//880+8VoSIiIiSF5eDEGTDoO0NtEj66quvyv79++XNN9/UoATtcRARERElWJmQVKlSyYsvvig5cuSQ7NmzS6ZMmWTt2rVxaqyMiIiIkieXU0LQWmqbNm2075gaNWpoS6RIAUGKCFo+JSIiIkqQlBA01Y4UkP79+0v37t21EzsiIiKiBE8JQeNk77zzjqxcuVKDEaSGDBkyRDZu3CgPHz50eQWIiIgoeXI5JQQ90Zq90d65c0ebcP/666/ltdde047mWHWXiIiIEqxg6o0bN2THjh3auRyGY8eOSbZs2bTKLhEREVGCBCFlypTRZtoRdLz00kvSpUsXqV27tpQtW9bVWREREVEyFqeCqQg6SpcunTBrRERERMmCy0FIjx49EmZNiIiIKFlxuXYMERERkTswCCEiIiKPYBBCREREHsEghIiIiLynnRD0ortv3z4JDw+XZ8+e2XzXrl07d60bERER+TCXg5Aff/xRm22/f/++ZM6cWVKkSGH5Dn8zCCEiIqIEyY5Bx3WdOnXSIAQpIrdu3bIMN2/edHV2MmvWLAkMDJS0adNKtWrVNIUlOmiZ9a233tLpEfBMnTo1yjQjR47U76yHoKAgl9eLiIiIklgQcvnyZenVq5ekT58+3gtftWqV9OvXT0aMGCEHDx6UcuXKScOGDTWbxxF0kFekSBEZP3685M6dO9r5lipVSq5evWoZdu3aFe91JSIiIg8HIQgS9u/f75aFT5kyRZt979ixowQHB8ucOXM0uFm0aJHD6atUqSKTJk2SVq1aib+/f7TzTZUqlQYp5hAQEOCW9SUiIiIPlglp0qSJDBw4UI4fP679yKROndrm+zfeeMOp+Tx+/FgOHDggISEhlnHohbd+/foSGhoq8XH69GnJmzevZvFUr15dxo0bJwULFox2+sjISB1Md+/ejdfyiYiIKAGCEKRcwKhRo6J8h/IXT58+dWo+EREROm2uXLlsxuPzyZMnJa5QrmTJkiVSokQJzYr5+OOPtXffo0ePSqZMmRz+BkEKpiMiIqIknB2DKrnRDc4GIAmpcePG0qJFC+3VF1lH69at0wK0q1evjvY3SI25c+eOZbh48WKirjMREVFyFKd2QtwB5TT8/PwkLCzMZjw+x1To1FVZs2aV559/Xs6cORPtNChfElMZEyIiIkoiLabu2LFDXn/9dSlWrJgOKAeyc+dOl+aRJk0aqVSpkmzZssUyDqkp+IxyHO6CqsR//fWX5MmTx23zJCIiIg8EIcuXL9fCo6jFgqq6GNKlSyf16tWTr776yqV5oXru/PnzZenSpXLixAnp1q2bPHjwQGvLABo+sy64isKshw4d0gF/o7ow/rZO5RgwYIAGSefPn5fdu3dL8+bNNcWldevWrm4qERERJaXsmLFjx8rEiROlb9++lnEIRFDddvTo0dKmTRun59WyZUu5fv26DB8+XK5duybly5eX9evXWwqrXrhwQWvMmK5cuSIVKlSwfP700091qF27tmzfvl3HXbp0SQOOGzduSI4cOaRmzZqyZ88e/ZuIiIi8OAg5e/asZsXYQ5bMkCFDXF6Bnj176uCIGViY0FKqYRgxzm/lypUurwMRERF5QXZMgQIFbMpxmDZv3qzfERERESVISgj6jkH2C8pi1KhRQ8f9+uuv2jbHtGnTXJ0dERERJVMuByEoPIoqtJMnT7a0vVGyZEntB6Zp06YJsY5ERETkg+LUTghqnGAgIiIiStR2QoiIiIgSJSUke/bs8ueff2orp9myZdM+YqJz8+bNeK8UERER+T6ngpDPPvvM0vkb/o4pCCEiIiJyWxDSvn17y98dOnRwasZEvqBt2xPiLZYvL+npVSAiStgyIWgCPTw8PMp4tFCK74iIiIgSpHZMdC2WRkZGaqd0RESewpQrIh8NQqZPn67/ozzIggULJGPGjJbvnj59Kr/88osEBQUlzFoSERFR8g1CUCDVTAmZM2eOTdYLUkDQrwvGExEREbk1CDl37pz+X6dOHfnuu++0qi4RERFRopUJ2bZtW5wXRkRERBTnIATlP9BZHXrSRS2ZZ8+e2Xy/detWV2dJREREyZDLQUjv3r01CGnSpImULl2aDZcRERFR4gQhK1eu1N5zX3311bgtkYiIiCgujZWhJkyxYsW484iIiChxg5D+/fvLtGnTom20jIiIiChBsmN27dqlNWR+/vlnKVWqlKROndrme1TfJSIiInJ7EJI1a1Zp3ry5qz8jIiIiil8QsnjxYld/QkRERBT/MiHw77//yubNm2Xu3Lly7949HXflyhW5f/++y/OaNWuWNvmeNm1aqVatmuzbty/aaY8dOyZvvfWWTo8+bKZOnRrveRIREZGXBCF///23lClTRpo2bSo9evSQ69ev6/gJEybIgAEDXJrXqlWrpF+/fjJixAg5ePCglCtXTho2bKiNoDny8OFDKVKkiIwfP15y587tlnkSERGRlwQhaKyscuXKcuvWLUmXLp1lPMqJoBVVV0yZMkW6dOkiHTt2lODgYO0AL3369LJo0SKH01epUkUmTZokrVq1En9/f7fMEyIjI+Xu3bs2AxERESWxIGTnzp0ydOhQbS/EGrI/Ll++7PR8Hj9+LAcOHJD69ev/38qkTKmfQ0NDXV2teM1z3LhxkiVLFstQoECBOC2fiIiIEjAIQV8x6D/G3qVLlyRTpkxOzyciIkLnkytXLpvx+Hzt2jVXVyte8wwJCZE7d+5YhosXL8Zp+URERJSAQUiDBg1sCoSigCgKpKIMhrc25Y6sncyZM9sMRERElMSq6E6ePFkLeqK8xaNHj6RNmzZy+vRpCQgIkBUrVjg9H0zv5+cnYWFhNuPxObpCp56YJxERESWRlJD8+fPLH3/8IR999JH07dtXKlSooLVVfv/9d8mZM6fT80GZkkqVKtkUZkVWDz5Xr17d1dVKsHkSERFREkkJ+eWXX6RGjRryzjvv6GDddgi+e+mll5yeF6rStm/fXmvbVK1aVbN5Hjx4oDVboF27dpIvXz4tOGoWPD1+/LjlbxSEPXTokGTMmNHSqV5s8yQiIiIvDULq1KkjV69ejZLqgQKd+M5RodXotGzZUtsZGT58uBYcLV++vKxfv95SsPTChQtau8WEBtGQ8mL69NNPdahdu7Zs377dqXkSERGRlwYh6D0XhVHt3bhxQzJkyODyCvTs2VMHR8zAwroasDO998Y0TyIiIvKyIOTNN9/U/xGAdOjQwaaxMKR+HD58WLNpiIiIiNwahKARL0BKBNoDsW4tFQVCX3jhBW2plIiIiMitQYjZey6yRAYOHKhNoRMRERElWhXdHTt2aM0Ue+hvpW7dunFeESIiIkpe3BaEoOEy9CtDRERE5NbsGBQ8NcuEoK0O675YUDAV1WDRpgcRERGRW4MQtLeBmjEYHGW7oKDqjBkznJ0dERERJXNOByHnzp3TVJAiRYrIvn37JEeOHDa1Y9B4GfptISIiInJrEFKoUCFLXyxEREREid5iqgnlQtCsun0h1TfeeCPeK0VERES+z+Ug5OzZs9K8eXM5cuSIlg8xm1E3m3J3pe8YIiIiSr5crqLbu3dvKVy4sISHh2uDZceOHdPec9FrrX1fL0RERERuSwkJDQ2VrVu3SkBAgPZwi6FmzZoybtw46dWrl/z++++uzpKIiIiSIZdTQpDdgr5jAIHIlStXLAVXT5065f41JCIiIp/kckpI6dKl5Y8//tAsmWrVqsnEiRO1iu68efO0+i4RERFRggQhQ4cOlQcPHujfo0aNktdee01q1aolzz33nKxatcrV2REREVEy5XIQ0rBhQ8vfxYoVk5MnT8rNmzclW7ZslhoyRERERAnWToi17Nmzu2M2RERElIy4XDCViIiIyB0YhBAREZFHMAghIiKi5BuEzJo1SwIDAyVt2rRa7Re99Mbk66+/lqCgIJ2+TJkysm7dOpvvO3TooIVkrYdGjRol8FYQERGRVwUhqNbbr18/GTFihBw8eFDKlSunNXDQLLwju3fvltatW0vnzp21ddZmzZrpcPToUZvpEHRcvXrVMqxYsSKRtoiIiIi8IgiZMmWKdOnSRTp27CjBwcEyZ84c7ZNm0aJFDqefNm2aBhgDBw6UkiVLyujRo6VixYoyc+ZMm+n8/f0ld+7clgFViImIiCjp8GgQ8vjxYzlw4IDUr1///1YoZUr9jD5qHMF46+kBKSf206MzvZw5c0qJEiWkW7ducuPGjWjXIzIyUu7evWszEBERkQ8HIREREdoXTa5cuWzG4/O1a9cc/gbjY5seKSXLli2TLVu2yIQJE2THjh3SuHFjXZYj6HwvS5YslqFAgQJu2T4iIiJK4MbKkppWrVpZ/kbB1bJly0rRokU1daRevXpRpg8JCdFyKSakhDAQISIi8uGUEPTC6+fnJ2FhYTbj8RnlOBzBeFemB3Ssh2WdOXPG4fcoP5I5c2abgYiIiHw4CEHvu5UqVdJsE9OzZ8/0c/Xq1R3+BuOtp4dNmzZFOz1cunRJy4TkyZPHjWtPREREXl07Btkg8+fPl6VLl8qJEye0ECl66UVtGWjXrp1ml5h69+4t69evl8mTJ2vneSNHjpT9+/dLz5499fv79+9rzZk9e/bI+fPnNWBp2rSpdrZn3fkeERERJfMyIS1btpTr16/L8OHDtXBp+fLlNcgwC59euHBBa8yYatSoIV999ZUMHTpUhgwZIsWLF5e1a9dK6dKl9Xtk7xw+fFiDmtu3b0vevHmlQYMGWpUX2S5ERESUNHg8CAGkYpgpGfZQmNReixYtdHAkXbp0smHDBrevIxEREflYdgwRERElTwxCiIiIyCMYhBAREZFHMAghIiIij2AQQkRERB7BIISIiIg8gkEIEREReQSDECIiIvIIBiFERETkEQxCiIiIyCMYhBAREZFHMAghIiIij2AQQkRERB7BIISIiIg8gkEIEREReQSDECIiIvIIBiFERETkEQxCiIiIyCMYhBAREZFHMAghIiIij2AQQkRERB7BIISIiIiSbxAya9YsCQwMlLRp00q1atVk3759MU7/9ddfS1BQkE5fpkwZWbdunc33hmHI8OHDJU+ePJIuXTqpX7++nD59OoG3goiIiLwqCFm1apX069dPRowYIQcPHpRy5cpJw4YNJTw83OH0u3fvltatW0vnzp3l999/l2bNmulw9OhRyzQTJ06U6dOny5w5c2Tv3r2SIUMGneejR48SccuIiIgoSQchU6ZMkS5dukjHjh0lODhYA4f06dPLokWLHE4/bdo0adSokQwcOFBKliwpo0ePlooVK8rMmTMtqSBTp06VoUOHStOmTaVs2bKybNkyuXLliqxduzaRt46IiIiik0o86PHjx3LgwAEJCQmxjEuZMqVmn4SGhjr8DcYj5cQaUjnMAOPcuXNy7do1nYcpS5Ysms2D37Zq1SrKPCMjI3Uw3blzR/+/e/euuNOTJ/fFmzi7/b66Xd62bdwu3z1eRN7EPLeRKJCkg5CIiAh5+vSp5MqVy2Y8Pp88edLhbxBgOJoe483vzXHRTWNv3Lhx8vHHH0cZX6BAAUnOVq8Wn8Tt8i48XkTe6d69e5oIkGSDkKQCKTHWqSvPnj2TmzdvynPPPScpUqSQpB5xIli6ePGiZM6cWXwFt8u7+Orx8uVt43Z5l7tedB4iBQQBSN68eWOd1qNBSEBAgPj5+UlYWJjNeHzOnTu3w99gfEzTm/9jHGrHWE9Tvnx5h/P09/fXwVrWrFnFm+CkTOonZlxwu7yLrx4vX942bpd3yewl52FsKSBJomBqmjRppFKlSrJlyxabVAh8rl69usPfYLz19LBp0ybL9IULF9ZAxHoaRJCoJRPdPImIiCjxeTw7Btkg7du3l8qVK0vVqlW1ZsuDBw+0tgy0a9dO8uXLp+U2oHfv3lK7dm2ZPHmyNGnSRFauXCn79++XefPm6ffIPunTp4+MGTNGihcvrkHJsGHDNFkIVXmJiIgoafB4ENKyZUu5fv26Ni6GgqPIMlm/fr2lYOmFCxe0xoypRo0a8tVXX2kV3CFDhmiggZoxpUuXtkwzaNAgDWS6du0qt2/flpo1a+o80biZr0E2EtpYsc9O8nbcLu/iq8fLl7eN2+Vd/H30PExhOFOHhoiIiMjXGisjIiKi5IlBCBEREXkEgxAiIiLyCAYhRERE5BEMQrzYrFmzJDAwUGv9oG+cffv2ibf75Zdf5PXXX9cq1ahu7SudDqKKeZUqVSRTpkySM2dOrS5+6tQp8XazZ8/WTiLNBpTQFs/PP/8svmb8+PGW6v/ebuTIkbot1kNQUJD4gsuXL0vbtm21tet06dJJmTJltAkHbxYYGBjleGHo0aOH+AIGIV5q1apV2sYKqmwdPHhQypUrpx35hYeHizdD1WpsCwIsX7Jjxw69aezZs0cb13vy5Ik0aNBAt9eb5c+fXx/Q6IgSN/u6detq79XHjh0TX/Hbb7/J3LlzNdjyFaVKlZKrV69ahl27dom3u3Xrlrz44ouSOnVqDYSPHz+u7Ully5ZNvP38u2p1rHD/gBYtWohPQBVd8j5Vq1Y1evToYfn89OlTI2/evMa4ceMMX4HTc82aNYYvCg8P1+3bsWOH4WuyZctmLFiwwPAF9+7dM4oXL25s2rTJqF27ttG7d2/D240YMcIoV66c4WsGDx5s1KxZ0/B1vXv3NooWLWo8e/bM8AVMCfFCjx8/1jfP+vXrW8ahQTd8Dg0N9ei6kXPu3Lmj/2fPnt1ndhl6xEYLxkjd8ZUuEpB6hZaZra81X3D69GnN8ixSpIi888472iikt/vhhx+05W2kECDLs0KFCjJ//nzxtXv/8uXLpVOnTkm+c1VnMQjxQhEREXrDN1uVNeEzWp2lpA39I6FsAZKOrVv69VZHjhyRjBkzakuO77//vqxZs0aCg4PF2yGgQlan2WWEr0D5sSVLlmgr0ijTc+7cOalVq5b2eurNzp49q9uDVrQ3bNgg3bp1k169esnSpUvFV6xdu1ZbAe/QoYP4Co83206U3ODt+ujRoz6RDw8lSpSQQ4cOaerON998o31BoQyMNwci6C4d/VQh/93Xunto3Lix5W+Uc0FQUqhQIVm9erV07txZvDm4R0rIJ598op+REoLrbM6cOXpO+oKFCxfq8UMqlq9gSogXCggIED8/PwkLC7MZj8/oQZiSrp49e8pPP/0k27Zt00KdvgC9YRcrVkx7xEaqAQoWT5s2TbwZsjtRyLtixYqSKlUqHRBYTZ8+Xf9GSqSvyJo1qzz//PNy5swZ8WZ58uSJEviWLFnSJ7Ka4O+//5bNmzfLe++9J76EQYiX3vRxw9+yZYvNWwA++0pevK9BOVsEIMiq2Lp1q/bu7KtwLkZGRoo3q1evnmYzIYXHHPCWjfIT+BsvAb7i/v378tdff+lD3Jshe9O+2vuff/6pqTy+YPHixVrWBWWUfAmzY7wUquciiRE3xqpVq8rUqVO1QGDHjh3F22+I1m9kyK/GTR8FOAsWLCjenAWD3p+///57bSvELLuTJUsWbc/AW4WEhGjyMI4NyhRgG7dv36558t4Mx8i+vE6GDBm0/QlvL8czYMAAbYsHD+crV65oNX8EVa1btxZv1rdvX+1lHdkx//nPf7TdpHnz5ungC4H94sWL9Z6PlDif4unqORR3M2bMMAoWLGikSZNGq+zu2bPH63fntm3btOqq/dC+fXvDmznaJgyLFy82vFmnTp2MQoUK6TmYI0cOo169esbGjRsNX+QrVXRbtmxp5MmTR49Zvnz59POZM2cMX/Djjz8apUuXNvz9/Y2goCBj3rx5hi/YsGGD3i9OnTpl+JoU+MfTgRARERElPywTQkRERB7BIISIiIg8gkEIEREReQSDECIiIvIIBiFERETkEQxCiIiIyCMYhBAREZFHMAghIiIij2AQQsnayy+/LH369EnUZaIbdXQa5qobN25o3xHnz58XX4R2E7t27apN9KdIkUKb6yfPnsvoMr5Zs2YeOQwffvihfPDBBx5ZNiUeBiHkEzwRTMRVy5YttWMtV40dO1aaNm0qgYGB4ovWr1+vARp6Gb569arTfbQgYFm7dm2Crx8lDATVjoJO9HGzdOlSOXv2LHe9D2MQQpTI0GEdUjRc8fDhQ1m4cKF07tw5XstGF/ToDCspMntyRSdkuXPnTvSOuh4/fpyoy6OYBQQESMOGDWX27NncVT6MQQh5PSQZ79ixQ6ZNm6ZvVBjMLAuMRy/D/v7++oBDEu+///4b7bz+97//ac+2X375pX6+ePGi9siJ7BNkEyAlwjo7xEyu/vTTT3X+6GUVPeY+efLErdkx69at02144YUXbMb/8MMPUrx4cUmbNq3UqVNH3xyx/bdv37ZZFqYLDg7WeVy4cEEiIyP1TTNfvnzaO2y1atW091tAb8yZM2eWb775xmZZSG3AtOgt192wH5H0jnXD+pupPfgfPURbK1++vIwcOdLyPTRv3tzmd46yEZBShhQzE/7u2bOnjjcfeHD06FHtGThjxoySK1cueffddyUiIsKl7cG6zJ07V1577TVJnz69lCxZUkJDQ7WHaCwX+xHBFgIvE/7G+YVlYtlVqlSRzZs328z3888/txxvTPf222+77VxGgIreufE9zuNBgwZpFllsvv32WylVqpSeW9j/kydPjjWlCsvAuQmFCxfW/ytUqKDTWh8j9Pa7cuXKWNeBvBeDEPJ6CD6qV68uXbp00WR8DAUKFJDLly/Lq6++qjfzP/74Q9+okJowZswYh/NBN/Tozhw37XfeeUcDCTyY0K37zp075ddff9WHQ6NGjWzemrdt26YPEPyPIAA3V/MG6y5YfqVKlWzGnTt3Th9CeNhi+/773//KRx995DAVZcKECbJgwQI5duyYpsLg4YuHIm7whw8flhYtWuh2nT59Wh+QrVq10q7DreEzlof94Qi6UMf+iWlAkBHdMRw1apTkz59fj99vv/3m1H4xp8O6ufI7E45XmjRp9NjOmTNHg7e6devqA3H//v2aRRQWFqYPb1eNHj1a2rVrp9kMQUFB0qZNGz1GISEhOm884HEcTPfv39fzdcuWLfL777/r8cBD2Nxn+E2vXr10P506dUrX7aWXXnLbuYzgAeftokWLZNeuXXLz5k1Zs2ZNjNt44MAB3Tc4X44cOaLB4bBhw1w6//ft26f/I+DCMfzuu+8s3+EF4tKlSz5bDor+f2EwIp/sZn3IkCFGiRIljGfPnlnGzZo1y8iYMaPx9OlTm9/NnDnTyJIli7F9+3bLtF988UWU30dGRhrp0qXTrrWhffv22pX9v//+a5mmRYsW2j16dBYvXqzLckXTpk2NTp062YwbPHiwdltu7aOPPtIuv2/dumVZFj4fOnTIMs3ff/9t+Pn5GZcvX7b5bb169YyQkBD9e+/evTrNlStX9HNYWJiRKlUqm/1j78aNG8bp06djHJ48eRLt7z/77DPdl9bwGeOtlStXzhgxYoTlM7ZvzZo1NtPguGCfWcNxxvE24e8KFSrYTDN69GijQYMGNuMuXrzocjfqmH7o0KGWz6GhoTpu4cKFlnErVqww0qZNG+N8SpUqZcyYMUP//vbbb43MmTMbd+/edThtfM/lPHnyGBMnTrR8j2OVP3/+KPvRWps2bYxXXnnFZtzAgQON4ODgGI8P1g/nJpw7d06n+f3336PM/86dO/pdTOcdebfEzXQlSkQnTpzQFBIk8ZpefPFFfePE21XBggV1HLIdwsPD9e0QqSYmpC4g+dz+zf/Ro0c2yehIivbz87N8RrYM3grd6Z9//tEkeGt4G7ZeX/PN0R7e9MuWLWv5jHVD0vvzzz9vMx2yaJAMb84H24WUAmRhLV++XAoVKhTtmzcgiR+DN7FPXcIxR4oWUgns4Zjb77OYWO9zZJ1AmTJlbMbhXLp7965mf+G8REoCslGQIoBsQxx3MyXklVde0WNQpEgRTcHAgGwoZPeY4nou37lzR5eJbDkTyuRUrlw5xiwZXGPI1rGGawxZaDjHrK+LuJafMlPzyDcxCKFkD0nvBw8e1GRo3HTNoAUPBTykzDx1azly5LD8nTp1apvv8Ht3F/5EmYVbt27F+UZuHYhhu/BwQFK6/UPC+uH73nvvyaxZszQIQXZHx44dbebjKDsGQ0yOHz9uCf6ckTJlyigPwZjK27j6O2Q9WcO+QRYIsq/sIbh0hfV5Ye43R+PMcwVldDZt2qTli4oVK6bHDdlfZnYJAgicpyi7s3HjRhk+fLgGLciCMssYxfdcTghYh7gcQ0CWUGKsI3kOgxDyCXjbx5uXNRQGRKE53ADNmzHeEHEzR9kDU9GiRTU/HAXi8FCeOXOmjq9YsaKsWrVKy1DgTdWT8HBBaoS1EiVKaIFVa86UicC8sK/wxlyrVq1op2vbtq0WTpw+fboGD+3bt49xvu+//36sZSfy5s0rrsDDB2/oJqQaoCyMNTzY7Y89focCptZQNsM+YLSHY45zBgUsE7t2Ds5NFKhF6oYZONiXhcA61a9fX4cRI0Zo8LF161Z58803430uI8jau3evJbULKTEIVPHb6OAaw3rbbwdSjMwA1/4YotyRdcoGrl2wP4aAY4hjhlQ58k0smEo+AQ8N3EBx00ZNBrxddu/eXWsEoNbFyZMn5fvvv9cbN2oA4E3ZGm6aSIbHA8hsbwQF+pACgeRmFObDww9voSgciOycxIRChShUap0agkKO2K7BgwdruyOrV6+2FAiMKcUC24ptQ6FJFALEdqFw4Lhx4zQrwJQtWzZ9uA0cOFAaNGhgE7g5gqwYvMHHNLj6YEch0S+++EL3P7KREAjZp97g2KMw57Vr1yz7B79DQc5ly5bpQw/H3T4ocQQ1m/D2jUKdCOiQVbFhwwZNBXL0kHQn1HrB8UCwhOwTFGS1TlFD+ykICPH933//rduG7xGMuuNc7t27t4wfP15rsuC8wvVj1rKKTv/+/XXfoxAuzkFk3yHwQaqOCccC41DYFscEwap1MIjACKk+ZiFgZA2ZsK4IlM1sGfJBni6UQuQOKDT4wgsvaEE7nNYo7AYo0FalShUjTZo0Ru7cubUwp3XhSPsCrcePHzdy5sxp9OvXTz9fvXrVaNeunREQEGD4+/sbRYoUMbp06aIF5pwtAOmOgqlQtWpVY86cOTbjvv/+e6NYsWK6bi+//LIxe/Zs3f5//vknxmU9fvzYGD58uBEYGGikTp1aCyU2b97cOHz4sM10W7Zs0fmtXr3aSGiOCqZiP6OQLwpkFihQwFiyZEmUgqk//PCD7gMUnLX+PbYvV65cuv19+/Y1evbsGaVgqn1hZvjzzz91X2TNmlXPp6CgIKNPnz6WQp1mYd+Y2BfGdFT4ctu2bTaFiDFNnTp1dJnYVhQwtV7HnTt36uds2bLpNGXLljVWrVrltnMZ1wV+j32NbcfvMH1MBVPhm2++0YKoOI8KFixoTJo0yeZ7FIBGYd8MGTIYxYsXN9atW2dTMBXmz5+v25wyZUqbY4TCtCjAS74rBf7xdCBElJyg/Qi8ObqamoJUCqRK4I3ePiXHulVVVDVFCpA7IBWib9++cuXKFUuyeXKHVBW0P2O2q0IJ4+eff9aUFlQhT+ysMUo8PLJEiQjBAcpxxCWPu0mTJpq1gPZP0A6K2XgVakGgVgvy4idNmmTT9kRcIc8e+fhInke2DwMQ24ejWdaCEg4azUOBaAYgvo0pIUSJCIX00Eopym6g5c/4QioFChyiHANqnaB1TzSGFd8bN2pdIFUFhRRRlsZRlVUiovhiEEJEREQewdoxRERE5BEMQoiIiMgjGIQQERGRRzAIISIiIo9gEEJEREQewSCEiIiIPIJBCBEREXkEgxAiIiIST/h/6EL0ehOtXQoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x260 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Follow token t=4: query . keys -> scale -> causal mask -> softmax -> weights.\n",
    "t = 4\n",
    "qt = q[0, t]                                   # this token's query vector (head_size,)\n",
    "raw = (k[0] @ qt) / head_size**0.5             # score against every token's key (T,)\n",
    "masked = raw.copy(); masked[t+1:] = -np.inf    # a token may not look at the future\n",
    "w = softmax(masked)                            # weights over tokens 0..t (future = 0)\n",
    "\n",
    "print(f\"token t={t}: query shape {qt.shape}, scored against {T} keys\")\n",
    "print(\"raw scores  q.k/sqrt(hs):\", np.round(raw, 2))\n",
    "print(\"after causal mask       :\", np.round(masked, 2))\n",
    "print(\"after softmax (weights) :\", np.round(w, 3), \" sum =\", round(w.sum(), 3))\n",
    "print(\"matches row 4 of attn   :\", np.allclose(w, attn[0, t]))\n",
    "print(\"blended value == out[0,4]:\", np.allclose(w @ v[0], out[0, t]))\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(6, 2.6))\n",
    "colors = [\"#5b5bd6\" if j <= t else \"#d9d9e0\" for j in range(T)]\n",
    "ax.bar(range(T), w, color=colors)\n",
    "ax.set_xticks(range(T)); ax.set_xlabel(\"token j  (grey = future, masked out)\")\n",
    "ax.set_ylabel(\"attention weight\"); ax.set_title(f\"how token {t} splits its attention over the past\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m5c2c",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>qt = q[0, t]</code>: pull out token 4's query vector (16 numbers).</li>\n",
    "<li><code>raw = (k[0] @ qt) / head_size**0.5</code>: dot that query against every token's key, then apply the step-4 scale -- one score per token.</li>\n",
    "<li><code>masked[t+1:] = -np.inf</code>: blank out tokens 5, 6, 7 (the future); softmax turns <code>-inf</code> into exactly 0.</li>\n",
    "<li><code>w = softmax(masked)</code>: the five surviving scores become weights that sum to 1.</li>\n",
    "<li>The two <code>allclose</code> checks prove this hand-trace equals row 4 of the full <code>attn</code> matrix, and that blending the values with these weights gives the same <code>out</code> the batched code produced.</li>\n",
    "<li>The bar chart shows where token 4 actually looks: taller bars are tokens it weights more, the greyed bars are the masked future.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4838fb8a",
   "metadata": {},
   "source": [
    "## Step 4: why divide by the square root of the head size?\n",
    "\n",
    "One small detail keeps attention healthy at the start of training. A dot product adds up `head_size` separate little products (`head_size` is just the length of the query and key vectors), so with longer vectors the scores naturally grow larger. Feed large numbers into softmax and it becomes **peaky**, meaning almost all the weight piles onto a single token. That is a problem early on, because a peaky softmax produces very small gradients (it is already acting \"certain\"), so the model can barely learn, and what it is certain about is random nonsense, since nothing has been trained yet.\n",
    "\n",
    "Dividing every score by the square root of `head_size` cancels that growth and keeps the scores moderate, so attention starts out **soft**, meaning spread out, and therefore trainable. The demo makes it concrete: with no scaling the top weight is already 0.87 (for head size 16) or 0.94 (for head size 256), which is dangerously peaky, while the scaled version sits at a gentle 0.32 or 0.24."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f0465edc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.113532Z",
     "iopub.status.busy": "2026-07-01T09:02:50.113479Z",
     "iopub.status.idle": "2026-07-01T09:02:50.116501Z",
     "shell.execute_reply": "2026-07-01T09:02:50.116198Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "head_size=  16:  unscaled max weight = 0.87    scaled max weight = 0.32\n",
      "head_size= 256:  unscaled max weight = 0.94    scaled max weight = 0.24\n",
      "\n",
      "Bigger head_size -> unscaled softmax gets peaky (toward 1.0, starving gradients);\n",
      "the 1/sqrt(head_size) scale keeps it soft and trainable.\n"
     ]
    }
   ],
   "source": [
    "# Dot products grow with head_size, which makes softmax collapse toward one-hot.\n",
    "# Compare one query against 8 keys, for two head sizes, with and without the scale:\n",
    "rng = np.random.default_rng(0)\n",
    "for hs in [16, 256]:\n",
    "    qq = rng.standard_normal(hs); kk = rng.standard_normal((8, hs))\n",
    "    s = kk @ qq\n",
    "    print(f\"head_size={hs:4d}:  unscaled max weight = {softmax(s).max():.2f}    \"\n",
    "          f\"scaled max weight = {softmax(s / hs**0.5).max():.2f}\")\n",
    "print(\"\\nBigger head_size -> unscaled softmax gets peaky (toward 1.0, starving gradients);\")\n",
    "print(\"the 1/sqrt(head_size) scale keeps it soft and trainable.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4305113",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>rng = np.random.default_rng(0)</code>: a seeded random generator so the demo numbers are repeatable.</li>\n",
    "<li><code>for hs in [16, 256]:</code>: try a small and a large head size.</li>\n",
    "<li><code>qq = rng.standard_normal(hs); kk = rng.standard_normal((8, hs))</code>: one random query and 8 random keys of that size.</li>\n",
    "<li><code>s = kk @ qq</code>: the 8 match scores. With more dimensions added up, the scores naturally swing bigger.</li>\n",
    "<li><code>softmax(s).max()</code>: the largest attention weight <i>without</i> scaling. Near 1.0 means all the attention piled onto one token (peaky -- bad early in training).</li>\n",
    "<li><code>softmax(s / hs**0.5).max()</code>: the same scores divided by &radic;(head size) first: attention stays spread out and trainable.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6c300548",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.117409Z",
     "iopub.status.busy": "2026-07-01T09:02:50.117355Z",
     "iopub.status.idle": "2026-07-01T09:02:50.151651Z",
     "shell.execute_reply": "2026-07-01T09:02:50.151400Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbAAAAGJCAYAAADrO2ZPAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAASAJJREFUeJzt3Qd4FNXaB/B3k5CEjkgNvUmRDoKAiiJF4GJHBBQE5FpoggVRKUpX4YJSVcCKoILApwIKSpEiVQXpvShNkFAkQPZ8z//oLLObTdjZbNidyf93n7myky2zszPzzjnnPee4lFJKiIiIbCYq3BtAREQUDAYwIiKyJQYwIiKyJQYwIiKyJQYwIiKyJQYwIiKyJQYwIiKyJQYwIiKyJQYwIiKyJQYwHy6XSwYNGiRO89hjj0nJkiWDfm2OHDkknJ5++mlp0qSJ5/G+ffv0b/X+++9LZoTvje+P/WAnmf13C6dBgwbpfW+4dOmSFCtWTCZMmCB2dU0D2JYtW/RO9HfSYSdeq4P6m2++cWSQCrfz58/r/bpkyZKQvu/evXvlvffek5deekmc6Foe+0SGLFmySJ8+fWTo0KFy4cIFsaNrHsBeffXViAhg2A5//v77b3nllVfEad59913Zvn17hgcw7NdQB7CxY8dKqVKl5I477hAnCubYf/TRR/WxWqJEiQzbLnK+Tp06yYkTJ2T69OliR6xC9BEfHy8xMTHixLutuLg4sRtUc3zyySfy0EMPiZ2dO3cupO8THR2tj1VzlVBmFKr9mlnlyZNHmjZtatsagJAEsP379+s2ivLly0vWrFnl+uuvl9atW3uVtLCDsA5wJ40TDwvu1tE289tvv8nSpUs962+//XbPa//66y955plndH0tLsJly5aVkSNHitvtTlG3/uabb8o777wjZcqU0c+96aabZO3atV7tOePHj9f/Nj7LfBHw1wa2ceNGad68ueTKlUu3Bd15552yevVqv20SK1as0MXy/PnzS/bs2eW+++6T48ePp7n/5s2bp1/766+/etbNmjVLr7v//vu9nluxYkVp06aN17qPP/5YatWqpfd93rx55eGHH5aDBw9etQ3szz//1Hfy+F44kDt27Ci//PJLqm0Uhw8flnvvvVfvA3y/5557TpKTkz37H+sApTBjvxr78siRI/pur2jRovp3KVy4sNxzzz1XbcP58ccf9R1i48aNJRDff/+93HrrrXrf4zvhM7Zu3er5O/Yxtgv73LB+/Xq9rmbNml7vhd+8bt26Xuvmz5/vef+cOXNKy5Yt9bHrr81w9+7d0qJFC/289u3b+93etI5945jC33B+FShQQO8/89/M+2/u3Ll6exISEvQ+xjkwePBgz29kwPtXrlxZ14jgXMyWLZsUKVJEXn/9db/n9t13362/Lz6/d+/esnDhQs+5a/4e+N6+8Fnmc9kf/CZ4benSpXVQLlSokHTu3Fkfn/7acLDd7dq1k+uuu05uueUWuVZQzYZtuOGGG/R24hjG+Ynf2YDrT/369fU1EOcjzssvvvgi4HZA3+vPmTNn9LUP+xe/KX4DtAVv2LDB85zly5fra2vx4sX1c3CdxO+EEnog8H44z06ePCl2E5KiBgLEypUr9YUTJxh+oIkTJ+oDFwcbTpDbbrtNevbsKW+99ZZuy8CFGPDfMWPGSI8ePfRJ//LLL+v1BQsW9FRLNWzYUF88n3jiCf0j4bP69esnf/zxh36tGYrC+NHxXBwMOClxkO3Zs0eXQrD+999/l++++04++uijq343XFxwwcJF/oUXXtDvMXnyZP3dcGHxvcDhe+DEGjhwoN4P2L7u3bvLzJkzU/0MnITY1mXLlknVqlU9B2VUVJQ+sAwIhNu2bdPvZ0D9df/+/XUJ5fHHH9fPefvtt/X+RuDFRdwfBP9WrVrJmjVr5KmnnpIKFSroCyCCmD+4CDZr1kx/X5ykixYtklGjRumLJF6P4IXfHP9G0DYCr/F9HnjgAb0vsX9wMh47dkz/BgcOHEgzuQS/NfZNjRo15GqwTQg6uBDiIoATGPuiQYMG+oTH5+DCjX2CfY0Ls3lfI3gnJibq3xr7B5/93//+1/P+OF6wf7AfcAOFYxPfGb8f9rX5e1y+fFk/D3/D/sI54E9ax74BwQv7d8CAAWmWOHBBxPvgBgr/RTDHa/Cd3njjDa/nnjp1Su666y79O+HYwUW2b9++UqVKFb0PAZ/VqFEjfZ716tVLBxacXz/88IOEEo4DnJ+4wcFn4DjBTSj+ixtF31ImLtblypWTYcOGSVqzQeE3DPSinDt3bn1upwbH/3/+8x9ZvHixvs5hf+A6g23fvHmzPg+M6m4cV7hhuXjxosyYMUNv71dffaVvLqx68skn9W+Dc75SpUo6qOOagJuymv/ecH3++ef6WMS5h8CJcxrH/aFDh/TfrgZBFvsRxzu+o62oEDh//nyKdatWrcKRpT788EPPus8//1yv++GHH1I8/8Ybb1QNGzZMsX7w4MEqe/bsaseOHV7rX3zxRRUdHa0OHDigH+/du1e/9/XXX69Onjzped7cuXP1+v/7v//zrOvWrZte5w/WDxw40PP43nvvVbGxsWr37t2edb///rvKmTOnuu222zzrpk2bpl/buHFj5Xa7Pet79+6tt/Ovv/7y+3nm7//QQw95HtesWVO1bt1av+fWrVv1utmzZ+vHv/zyi368b98+/d5Dhw71eq9NmzapmJgYr/UdO3ZUJUqU8DyeNWuWfq8xY8Z41iUnJ6tGjRrp9fg+5tdi3Wuvveb1OTVq1FC1atXyPD5+/HiK/QenTp3S69944w1l1SOPPKJ/U1/G723ezurVq6sCBQqoP//807MO+yoqKkp16NDBs65ly5aqTp06nsf333+/XrAv58+fr9dt2LBBvz+OHzhz5ozKkyeP6tq1q9d2HDlyROXOndtrvbG/cIwGIrVj3zimbrnlFnX58mW/f8N+SOs8fOKJJ1S2bNnUhQsXPOvwWb7nZlJSkipUqJB64IEHPOtGjRqlnzdnzhzPur///ltVqFAhxXmMYwvf2xc+y/zd/P1u/rb7008/1c9btmyZZx2OK6xr27atCoTxWYEs/q5JZlOnTtXPGz16dIq/mc933+9y8eJFVblyZX1epbUPDL7nD44tXK/Sct7P/hs+fLhyuVxq//79KfafL1zPsH7kyJHKbkJShYiisrnNAncJqObDna65qBsM3EGgBIRSDaqSjAVVSrgrwp20GarX8FwDXgu4w7MK7//tt9/qajPc1RtQdYAqDNwJ4e7WDHfs5jtGfD7eB1UxacHzUBIA3NmhNID3ypcvn2c9/ot9ilIEzJ49W99l4g7avG9wF4s71LTulBcsWKDvOLt27epZh1JIt27d0rwb9N3mQPYrjo/Y2Fhd5YQ7fytwLJl/z9SglPDzzz/rqihUoxpQAkQVCRJ3zNuN49IozeB3RFVf9erVvfY1fkejigp32qjKbtu2rde+RlsUSqX+9jXuiEMBvxE+x8p5iGMI24fvirtzlNzNUEJ75JFHPI/x+9SpU8fr98QxgqpFo6QKqDozHzOhYN5uVNNhu2+++Wb92N/1w/c4TA3OA/xugSzVqlVL871QpY9zEaVlX+bz3fxdcKyfPn3ac7wFA+f7Tz/9pGuNUpPV9Jk4prH/UI2JeIiagasxzi+8LlNWIaKqZvjw4TJt2jRd1Wcu1uMHTI+dO3fqOnKjfcUXqqLMUMXo78exeuEEVMfh5Efbni9UfSJ4oK3pxhtvTPfn4yCfNGmS7Nq1S9ep46SoV6+eJ7DhooH/ojoMgcbYN9jXCFb+pFUlgoCKQOxbtYUbD39w4fL9DfDdAtmvqJdHlduzzz6rq8dwcUJVRYcOHfRF5moCmTTcuEFI7bdCuw1ObrTlYJ+iim/VqlW6vQDHENahysocwFBlYwRD7GtAlZo/qHY0QyKQ0V4FZ8+e1YsBASm1Y9oXMjADge1HBi2qDn1vrHzPQ2ybb9Ucfk9zOyz2KarGfJ+X2jESLFTzod0U1W2+57O/60eg+wPHbKBtp1eDcxLH1tUSvFBVOGTIEH0zlZSU5FkfbLINmkBQbY3jFFV9uNHCeWO+oT5w4ICuKka7ru/5GMj11zi/7JgQFJIAhrsSBC80NuKii/pk7AzUFZsTLYKB1+MOGu1P/qBB1Sy1O9VALoKhEOznG3f6KFHiLhj128bFFu2GuPjhbgptXuZ9g/2MxAJ/nxvKzseBlADSgmMDbW5z5szRwQTtdrjpwcU2rfYt1OkHc/ORltq1a+uLG/Y1bjjQMI7jCPsaKe248CCAoS3PYBzHaAfzF3R9L2wI2saNBqAdzNx1A+nvgXZCNt9hpwalQ7QVI5C+9tprOvDgO+LOH21bvudhqM+T1C5+qH242rGDGgS0vzz//PO6FIzjFtuLNjp/149A9ofx2VdLoDLgRgWl0PTAMYPSKtqfcRzhBhE3kbg2mtPU09pX/vYNjssvv/xS1wahLRM3g6h9ad68uX4Nro+4CcDvjLZsXDdQkEBtRCDXX+P8QgkzUwYwNDLiLgGN+uaqAJxUZmlF+NT+hhMRF+9Q3UldbTvMcIeMEoq//lOoksEFCndGoYALKRacBAhgRtUnTgY0yqMqFQcrHpv3DS44uCP1DeRXgwsoqr1QwjSXwlACzKj9iu1FKQwLSjS4WOGYQRZlanBCIo0ed5K4MUqN0R8qtd8KJydObHN1GfY19rmxr/FfBC983tGjR1Psa0CwC+ZYxF2zOWPOfBEOxZ0vqmdR3YoLm3m70Qk8WNinSMLCMWbeRn/HCEpvvue7UYozlxb8XTyRGIHgjlKEwSjxpgdqRwItreFcSCtbEr8/qvLQRJJazQaqGXHTgBs0c5cVBDB/tTK++yu1ZgYEQiTyYEEJFTe3uJFt3ry5bNq0SXbs2CEffPCBPsYMqBYNlHGMGIl1dhKSNjDcYfneuSELxveOwriA+DvQ8Td/63EHgqoeHBS+8HxUBVmV1nb4fi/0kUB2nvluGRc33FHhguRbdZQeuICiRIIsIuOiios80rBHjBjhScs1IIMM24iT33f/47FvGrIZMuRwMqKDswF3a0YXg2AYgdB3vyJI+vb0xwUB38tczeIPSvT4Lkh1TwtOcuwrnMjmz0eGGO5cUfVihv2LCxIuXMa+RpDDSYw7XOM55v2F3xqZb9hvvq52p4+LOAKfsaAq+GrHvhVGKcd8HCALLj3DBOE7407e3OUAv6P5mDH/nsgYxGeaq9N8u3MEst3gm10cjFC2gSGLFm1E48aNS/E3Y9vxXRDozdc9XDdQ62CG4wjHmm/7ve9vhffxrQLEDRS6SRjnTbSf/Yd/IxsyUEY3EpxrmbIEhvYMVK3gDhntBgg4SGlG9Y8ZLjDY4bhA4IfBXQraFPCj4MKMlGTUH6OOHevwN1Qr4ATCZ6BIjOehLQN3Hij54QCxWvQ1ggDS+nGSYptQ3ekPtgcHOIIV7oBQVYQ0ehxA/vrNpAcumLj7NycPYNvQIIsAjjtEczUHLhrYPnQpwH5AsgmCAu6oUOWAJBD01fIHz0UpBKUh3FGjpIP9bKQdB1MqQIDF748uAygRoloGCSe4yUDfOdyM4O/Yh9g+3Aiktt8N2A84jnA8pdb+ZED1Cu5KcSJ26dLFk0aP49K3bx/2Ne5icYE1ByqUXvD7IiXe3IaFiw6OT/Sbwx0wthsldLQ/fP311zog+bu4BSK1Y98KHCO4s0dNCI5r/H44J9NTdY4uJ/hOSFxB2jhuEnB8opThe4ygCwfOR1T74XdGmxFK1kbJNTXYr9jnOJdwY4CkEdxwpKfkmBFtYCjdfPjhh7o2xLjBxHUIxyWuC+hviDT50aNH632AJC+UlnBDiN/U3LZo7C/clOK/qNJGMENJygyJODgGH3zwQR1gUbWKz0O3JaO2q0KFCnof4zzHzQb2J0qCVqrdcX3D8et7vbaFUKQyIk26U6dOKl++fCpHjhyqWbNmatu2bX5Ta999911VunRpnbJsTl9FOjLSm5GejvXm1FukMPfr10+VLVtWp7Tjc+rXr6/efPNNnaZqTk31l6rtm5qKlOQePXqo/Pnz61RT827wlwaOlGp8J3w3pCTfcccdauXKlX7TmteuXeu1Ht8vkDRd+O233/RzK1as6LV+yJAhen3//v39vg4p8Ui1RncDLEhzRurt9u3bU02jN9Le27Vrp/c50nUfe+wxtWLFCv1ZM2bM8Hot3teXv7Rc7Bek1uN3MvbliRMn9PZgu/A++Ky6deuqzz77TAWiZ8+e+rc3Sy0VedGiRapBgwYqa9asKleuXKpVq1Zqy5YtKd4zMTFRH4P47uYU9Y8//li/76OPPup3W/A74ljAd4iPj1dlypTR+23dunVX3V+pSe3YT+2YSi2NHr/dzTffrL97QkKCeuGFF9TChQtTHH94f6Tu+/J3jOzZs0dvG94T58uzzz7r6YKxevVqr+ci7b5IkSIqLi5O/wbYJ4Gk0R86dEjdd999upsC9iu6jxip3eZz0TjecNyGA9LVX375ZVWqVCmVJUsW3e3gwQcf9OpiM2XKFFWuXDm9D3C843v6O0/wXl26dNHfF787utAcO3bM6zuja8Pzzz+vqlWrpp+DYwr/njBhgtd7bdmyRXffwfUJ10Z06UD3Ed/97G870L0H5+p7772n7MiF/wt3EKXIgeoOJC8gtdxczRVOaBPEnSaSVVCSo/BC9R5GekBHWZSYyN6/5euvv65LzIEmx0QSBrBMDFVs5oMWde5o81u3bp0e+imSDmj0qUJVp5XGaQr9MYI2MGSN4ljxrfIie7l06ZKufnzxxRd1NagdOW/UWrLU/QEXKLQZoU0PGWxIZ0aiQiQFL0AbEV17SBRCpibar9FujXYtZHWiLYzsLUuWLLoN185YAsvEkEmJxmCUbHBnjcZmlHTMYy1S5oYqJszFhiQhlLqQhIM+mb4DShOFAwMYERGlC7IokQWMlHwM64YsY2Q6X63vIrI6MYIM+tNiFBl/MxqkhfOBERFRuqBLAVL9A+1Him4S6HaA6Xww7BZG6kGXAn/9fdPCEhgREYUM+gderQSGYa/QfxIDDRjQtxId+jGIdKZI4sDIERilGZ137TgQJRGRAT2a0HkZI22Yx9FMjwsXLniNjmKF7xBigMEnQjGzOwa78O1kjkElUBKzwtYBDMErVGMREhFFAowOYx4FJj3Bq1SJHHLkWMpBggOBkT/MMygAJur1HdUmGOim4ztxKx5jFgXfrhuODWAoecH+DSUlV47Ia86774Yq4d4EIrKJy3JJfpRvPNe19Lp48aIOXnvXl5BcOa1dHxPPuKVUrf06mJrHew1F6SuUbB3AjOItgpfVH+haiHGlPh8XEZGXf8dECnVzSPYc/yxWJP+7LQheoRyw3DzQMsZCNcNjfJaVPqi2DmBERJQ2tyi9WGH1+VZh8ATzLOmAUXasjogfecUWIiKylbNnz+p0eCxGmjz+bYz0gRkzzPOVPfnkk3qMU3SKx8gumErms88+02NsWsESGBGRg7n1/6y/xgqMn4o+XQZ0UAZM7/P+++/rzs3mYasw0SjS6BGwMHcZklYw4gsyEa1gACMicrBkpfRi9TVWYK7CtCY2QRDz95qNGzdKejCAERE5mDsC28BChQGMiMjB3KIkmQGMiIjsxu3gEhizEImIyJZYhUhE5GDJ1yCJI1wYwIiIHMz972L1NXbAAEZE5GDJQSRxWH1+uDCAERE5WLK6MrahldfYQUQkcWAWz5IlS0p8fLzUrVtX1qxZE+5NIiJyVBWi2+JiB2EPYDNnztTDjmCemQ0bNuhpqTGcyLFjx8K9aUREFMHCHsBGjx4tXbt2lU6dOkmlSpVk0qRJki1bNpk6dWq4N42IyPbc4pJkiwteYwdhDWCYcG39+vVeU0tjKm08xpTTvpKSkvSMneaFiIhS51bBLXYQ1gB24sQJSU5O9ju1NKac9jV8+HDJnTu3ZylWrNg13FoiIvtJDqIEhsUOwl6FaAXmlDl9+rRnwXTXRESUOQNYWNPo8+XLJ9HR0X6nlsaU077i4uL0QkREgXErl16ssPr8TFkCi42NlVq1asnixYs969xut35sdWppIiLKXMLekRkp9Ji1s3bt2lKnTh0ZM2aMnDt3TmclEhFR+iQHUSXIKsQAtWnTRo4fPy4DBgzQiRvVq1eXBQsWpEjsICIi65IlSi/WXmMPYS+BQffu3fVCREShpYJoA8Nr7CAiAhgREWWMZFYhEhGRHSWrKL1Ye43Ygq36gRERERlYhUhE5GBuPbahtbKKm/OBERFRuCWzDYyIiDJPG5gSO2AVIhGR46sQXZZfYwcMYEREDuYOoiOzXdrAmIVIRES2xBIYEZGDJbMNjIiI7FqF6HZoFSJLYEREDpasXHqx+ho7cEQAq/+/xyU6Ll4iTfLs0xKJitz/W7g3gYgiejR6JXbgiABGRET+uVWUXqxw26QfGLMQiYjIllgCIyJysGRWIRIRkR25g0jKwGvsgCUwIiIHcweVRm+P1iUGMCIiB0sOqiMzAxgREYWZ28GD+dojzBIREflgFSIRkYMlswqRiIgyTxp9lNgBS2BERA7mVi69WH2NHTCAERE5mDuoCS1ZAiMiIluOhRgldmCPrSQiIvLBKkQiIgdLFpderL7GDsJaAlu2bJm0atVKEhISxOVyyZw5c8K5OUREjq1CdFtc7CCsW3nu3DmpVq2ajB8/PpybQUTkWMmmUljgiz2EtQqxefPmeglUUlKSXgyJiYkZtGVERM7gZhJHZBg+fLjkzp3bsxQrVizcm0REZIuROJItLsFAbVrJkiUlPj5e6tatK2vWrEnz+WPGjJHy5ctL1qxZ9fW8d+/ecuHChYA/zx4Vnf/q16+fnD592rMcPHgw3JtEREQiMnPmTOnTp48MHDhQNmzYoJuHmjVrJseOHfO7f6ZPny4vvviifv7WrVtlypQp+j1eeuklZwawuLg4yZUrl9dCRESpU/+ORm9lwWusGj16tHTt2lU6deoklSpVkkmTJkm2bNlk6tSpfp+/cuVKadCggbRr106X2po2bSpt27a9aqnNtgGMiIiuXRViYmKi12LOQTC7ePGirF+/Xho3buxZFxUVpR+vWrXK72vq16+vX2MErD179sg333wjLVq0CPi7sR8YEZGDudMxFqJvngGq+wYNGpTi+SdOnJDk5GQpWLCg13o83rZtm9/PQMkLr7vllltEKSWXL1+WJ5980lIVYlgD2NmzZ2XXrl2ex3v37pWff/5Z8ubNK8WLFw/nphERSWYfjf7gwYNeTTVoxgmVJUuWyLBhw2TChAk64QOxoFevXjJ48GDp379/5AewdevWyR133OF5jAZA6Nixo7z//vth3DIiImdwp6MEFmiuQb58+SQ6OlqOHj3qtR6PCxUq5Pc1CFKPPvqoPP744/pxlSpVdN/g//73v/Lyyy/rKsiIbgO7/fbbddHRd2HwIiKyj9jYWKlVq5YsXrzYs87tduvH9erV8/ua8+fPpwhSCIKAOBAItoERETmYW6IsT48SzHQqqEFD7Vnt2rWlTp06uo8XSlTISoQOHTpIkSJFdH9ewDCCyFysUaOGpwoRpTKsNwLZ1TCAERE5WLJy6cXqa6xq06aNHD9+XAYMGCBHjhyR6tWry4IFCzyJHQcOHPAqcb3yyit6DFz89/Dhw5I/f34dvIYOHRrwZ7pUoGW1CIS0TozIUfGpYRIdFy+RJrnhaYlERe7/LdybQEQ+LqtLskTm6kEaQtHHNfHf6+MTyx6QuBxZLL026ewlmXzbrJBtS0ZhCYyIyMFUEGMh4jV2YI+tJCIi8sESGBGRgyU7eEJLBjAiIgdzqyv9uqy8xg4YwIiIHMzt4PnAGMCIiBzM/e8I81ZfYwcMYEREDpZ8jfqBhYM9yolEREQ+WAIjInIwN9vAIluO35MlJkuyRJpzi3JLJPr9ufoSqRLeXBnuTSByXhuYYhsYERHZjAoiiQOvsQNWIRIROZg7HfOBRToGMCIiB3M7uA3MHltJRESU3hLY3r17Zfny5bJ//349oybmcMGEZJh1Mz4+8qY0ISLKzNysQhT55JNPZOzYsbJu3To9QVlCQoJkzZpVTp48Kbt379bBq3379tK3b18pUaJEuH8zIiISjsShS1ixsbHy2GOPyaxZs6RYsWJeB0ZSUpKsWrVKZsyYoaeTnjBhgrRu3ZoHDxFRmLkzewlsxIgR0qxZs1T/HhcXJ7fffrteMB30vn37QrmNREQUJHdmD2BpBS9f119/vV6IiCj83Jk9gPlyu92ya9cuOXbsmP632W233RaqbSMiIgpdAFu9erW0a9dOZyEq5T3rmcvlkuTkyBvSiYgos3KzBHbFk08+qRM1vv76aylcuLAOWkREFJlUEPN7KaeWwHbu3ClffPGFlC1bNmO2iIiIQsbt4BKY5ZE46tatq9u/iIjIPgHMbXFxZAmsR48e8uyzz8qRI0ekSpUqkiVLFq+/V61aNeD3Gj58uMyePVu2bdumO0XXr19fRo4cKeXLl7e6WURElMlKYJYD2AMPPKD/27lzZ886tIMhocNqEsfSpUulW7ductNNN8nly5flpZdekqZNm8qWLVske/bsVjeNiIgykaDGQgyVBQsWeD1+//33pUCBArJ+/Xqm4xMRhYCbJbArMnKcw9OnT+v/5s2b1+/fMWQVFkNiYmKGbQsRkRMohRoyi1mINqlCDGo6lY8++kgaNGigB/RFfzAYM2aMzJ07N+gNQYfoZ555Rr9v5cqVU20zy507t2fxHZORiIi8IYU+mMWRAWzixInSp08fadGihfz111+eNq88efLoIBYstIVt3rxZDwicmn79+ulSmrEcPHgw6M8jIsoM3A7OQrQcwN5++21599135eWXX5bo6GjPenRu3rRpU1Ab0b17d/nqq6/khx9+kKJFi6Y5aHCuXLm8FiIiunoVotXFsUkcmF7FX3A5d+6cpfdC5iLS8r/88ktZsmSJlCpVyurmEBFRJmU5gCHI/PzzzymSOZBRWLFiRcvVhtOnT9dtZzlz5tR9ywDtW+gXRkRE6eNmFuIVaP9C4Llw4YIuQa1Zs0Y+/fRTnWDx3nvvWW5PA8wjZjZt2jQ9eSYREaWPcnAWouUS2OOPP65LR6+88oqcP39ej0yPbMSxY8fKww8/bOm9fEezJyKi0FJBlMAcG8DQ96p9+/Z6QQA7e/as7nwMGCORg/wSEUUOpQOS9dc4MguxZcuWns7E2bJl8wSv7du3p6gKJCKi8HKzH9gVOXLkkPvuu0+PXWjYunWrDl7GOIlEREQRVwLD6PHoRIwqRLRhofMxglfbtm11OxgREUUO5eB+YJYDGBI4MBszqgwfeughufPOO6VDhw4yevTojNlCIiIKmtvBI3EElMThO2huVFSUzJw5U5o0aaKrDfv37+95DkfHICKKHEoFkcRhkyyOgAIYxjnEXF++UIU4adIkmTx5clDzgRERUcZSmb0fGMYoJCIi+1GZPYA1bNgw47eEiIgoIzsyA6ZRmTJlik6fhxtvvFE6d+6sxzAkIqLI4VYucVksUbmdmoW4bt06KVOmjPzvf/+TkydP6gUZiFi3YcOGjNlKIiJKVxKHsrg4sgTWu3dvufvuu/WcYDEx/7wcnZoxRiJmVF62bFlGbCcREQVB6YBktQ3MoQEMJTBz8NJvEhMjL7zwgp7UMhyyHr8oMTGWC5MZLilXvESi2LMSsS43qiWRKOb79eHeBKKgKAcncVi+6qOf14EDB1KsP3jwoJ7Ti4iIImwwX7G+ODKAtWnTRrp06aI7MiNoYZkxY4auQsRwUkRERBFZhfjmm2/qDssYPsoY0DdLlizy1FNPyYgRIzJiG4mIKEiKVYhXxMbG6kF7T506JT///LNekImIrMS4uDgeZEREmbQOcfz48VKyZEmJj4+XunXrypo1a67aJatbt25SuHBhHT9uuOEG+eabbzKuChH9vc6cOaPnAqtSpYpe8O9z587pvxERUQRRQYxEH0QSB5qV+vTpIwMHDtRdqqpVqybNmjWTY8eO+X3+xYsX9Xi6+/btky+++EIPEI8EwSJFimRcAPvggw/k77//TrEe6z788EOrb0dERA7oBzZ69Gjp2rWrdOrUSSpVqqTHyUXhZurUqX6fj/WovZszZ440aNBAl9ww6hMCX8gDGEabxzxgGLQXJTA8NhZUJ6LYZ8zOTERE9p8PLNF0nceSlJSUamlq/fr10rhxY69ZS/B41apVfl8zb948qVevnq5CLFiwoFSuXFmGDRtmaUD4gJM4jBHpsaCe0hfWv/rqqwF/MBERRbZixYp5PUb14KBBg1I878SJEzrwIBCZ4fG2bdv8vveePXvk+++/15MjowC0a9cuefrpp+XSpUv6c0IawDAiPUpfjRo1klmzZknevHm9EjtKlCghCQkJgb4dERFdCyqINq1/n49uUuY5HkOZqOd2u3Wt3TvvvCPR0dFSq1YtOXz4sLzxxhuhD2DGiPR79+6V4sWL+50fjIiInDOhZa5cuQKapDhfvnw6CB09etRrPR4XKlTI72uQeYguWHidoWLFinLkyBFdJYmCUciTOFDSYvAiIrIJlfFp9Ag2KEEtXrzYq4SFx2jn8geJG6g2xPMMO3bs0IEtkOAFkTeAIBERRUQShxVIoUcaPDLVMdUWBrdA9ypkJQIGv+jXr5/n+fg7shB79eqlA9fXX3+tkziQ1JGh84EREZGNqIz/CAwzePz4cRkwYICuBqxevbosWLDAk9iBMXSRmWhOEFm4cKGe4aRq1aq6/xeCWd++fQP+TAYwIiIKie7du+vFnyVLlqRYh+rF1atXB/15lqsQkYWI4T98oY8A/kZERJmvCjEcLAcwRFFkiPi6cOGCLF++3NJ7TZw4URcdjUwXROP58+db3SQiIsqE86kEXIX466+/ev69ZcsWXcdpQAc21HVaGcMKihYtqkewL1eunO5jhsa/e+65RzZu3Cg33nijpfciIiJ/UJqyWqJyOSuAoUHOGInDX1Vh1qxZ5e2337b04a1atfJ6PHToUF0qQ52ovwCGYUzMQ5mg2pKIiNIQTInKaSUwdGBGKal06dJ6iPz8+fN7/oacffSoNndIswqluM8//1ynXabWb2D48OEcroqIyArFAKY7MIO501kobNq0SQcstKHlyJFDvvzySz2SsT/oQ4C+BuYSmO9YXURElDkElUa/c+dOPTYi5nnxDWjoA2BF+fLl9aSYGOkec8J07NhRli5d6jeIYRwuTppJRGRBOsZCdFwAQ09r9KDG2FcY48o8rBT+bTWAofqxbNmy+t8YimTt2rV6xufJkydb3TQiIgrhWIiOC2BDhgzRyRZWektbgRJdanPOEBGRRYptYB6YvLJ169YhOYbQptW8eXM9uj0myZw+fbruZ4bhRYiIKASUc6sQLXdkRvD69ttvQ/LhaEPDAI9oB7vzzjt19SGCV5MmTULy/kREmZ1LBbc4sgoR7VX9+/fXfbWqVKmi53Mx69mzZ8DvNWXKFKsfT0REFFwAw+yZSHdHpiAWMyRxWAlgRESUwRTbwLw6NBMRkU0otoGlgAF9t2/fLpcvXw7HT0JERJl8MF/LSRznz5+XLl26SLZs2fR4hZikDHr06KEH5iUiogiiGMC8Ut9/+eUXne4eHx/vWd+4cWOZOXNmeH4gIiLKdCwnccyZM0cHqptvvtlrFA6Uxnbv3h3q7SMiovRQTOLwOH78uB553hdGkTcHNCIiigCKSRwetWvXlq+//trz2Aha7733XqrToBARUXi42JH5imHDhunhnzArMzIQMfAu/r1y5coU/cKIiCjMlHOrEC1nId5yyy06iQPBCyNxYFgpVCmuWrVKjyZPREQUcUkcly5dkieeeEIPJYVpVYiIiGxRAsO4h7Nmzcq4rSEiopByBdMO5tQ0+nvvvVen0vfu3VsixYV8sRKTJVYiTa79FyUSxR09K5HqZPXrJBLlrXSDRKrkLTvCvQkUyZRzsxAtB7By5crJa6+9JitWrNBtXtmzZ/f6OwfzJSKKIMq5SRyWAximQMmTJ4+sX79eL2YcjZ6IKMIoBrB/9oNSeggpZB1mzZo13D8LERFdRTATVNplQssoqwEMVYiHDh3KuC0iIiIKdQCLiorSAezPP/+08jIiIgoXxdHoPTBlyvPPPy+bN2/mAUlEFOmUcwOY5SSODh066DnBqlWrJrGxsSnawk6ePBnK7SMionRwObgNzHIAGzNmTMZsCRERhZ5iPzCPjh078hAjIrILxTR6jwMHDqS5r4oXL57xPwgREWV6lqsQS5YsmebElcnJyZl+pxIRRQoX28Cu2LhxY4oR6rFu9OjRMnTo0Gv+4xARURpYhXgFsg/9zdKckJAgb7zxhtx///1p7UoiIrqWVBBZhU7NQkxN+fLlZe3ataF6OyIiCgXl3CQOyzMyJyYmei2nT5+Wbdu2ySuvvKJH6QgWOkijbe2ZZ54J+j2IiMgHOzJfgZHofZM4MEZisWLFZMaMGRIMlNwmT54sVatWDer1RESU+ViuQvz++++9AhjGR8yfP7+ULVtWYmKs10iePXtW2rdvL++++64MGTLE8uuJiCh1zEI0uf322yWUunXrJi1btpTGjRtfNYAlJSXpxYAqTCIiypwst4ENHz5cpk6dmmI91o0cOdLSe6HKccOGDfo9A/3s3LlzexZUWxIRUeZsA7McwNBWVaFChRTrb7zxRpk0aVLA73Pw4EHp1auXfPLJJxIfHx/Qa/r166eTRowF70FERFevQrS62IHlRqsjR45I4cKFU6xHO9gff/wR8PusX79ejh07JjVr1vQaxWPZsmUybtw4XVUYHR3t9Zq4uDi9EBGRBTYJSBkewFBtt2LFCilVqpTXeqxDZ+ZA3XnnnbJp0yavdZ06ddKlu759+6YIXkREROkKYF27dtV9tTCEVKNGjfS6xYsXywsvvCDPPvtswO+TM2dOqVy5ste67Nmzy/XXX59iPRERBUk5tyOz5QCG2Zj//PNPefrpp+XixYt6HdqwUGpCGxUREUUOFwfzNe0Ml0tnG/bv31+2bt2qZ2TGCByhaJtasmRJut+DiIhMWAJLKUeOHHLTTTf5+QsREUUKl4NLYAGl0T/55JNy6NChgN5w5syZOjWeiIgyVz+w8ePH6zkj0axUt25dWbNmTcB9glG7d++994a+DQwp8ujn1aBBA2nVqpVn+hRs5KlTp2TLli3y448/6o3A+nfeecfSRhARkb3NnDlT+vTpo/sDI3iNGTNGmjVrJtu3b5cCBQqk+rp9+/bJc889J7feemvGlMAGDx4sO3bs0AFswoQJcvPNN0vx4sX1RmEalQ4dOsiePXt04Fq9ejUH5SUiymQlsNGjR+ssdXSHqlSpkg5k2bJl8ztyk7nvL8bCffXVV6V06dIZl4VYsGBBefnll/WCUteBAwfk77//lnz58kmZMmVSjFBPRET2bgNL9BlvNrXBJJCRjsEpzJnoGOgdY9yuWrUq1c957bXXdEGoS5cusnz58mszoeV1112nFyIicm4WYjGf8WYHDhwogwYNSvH0EydO6NIUCjpmeIz5Iv1Bs9OUKVPk559/lrDPyExERM4KYAcPHpRcuXJ5VodqKL8zZ87Io48+qqfRQi1esBjAiIgczJWOKkQEL3MASw2CEIb/O3r0qNd6PC5UqFCK5+/evVsnbyAp0OB2u/V/Ma8kEj/QNBXy0eiJiIjMYmNjpVatWnpYQXNAwuN69eqJL4x5i7FwUX1oLHfffbfccccd+t+BTpXFEhgRkZOpazMWIlLoO3bsqLtZ1alTR6fRnzt3TmclArLVixQpoud1RBcs3zFv8+TJo/9rZSxcywEMjXidO3eWEiVKWH0pERE5dCSONm3ayPHjx2XAgAF62q3q1avLggULPIkdyFxHZmIoWQ5gc+fOlaFDh0rDhg116uMDDzzAObqIiCKVunaj0Xfv3l0vwYx1+/7772d8AEP95MaNG2XatGl6RuVu3brJww8/rEtl4RobMcfeMxIT/c/I+BElOTIHFLucK7AZsMMh174LEokuFL16Q3a4xOSuJpHKteqXcG8CKedOpxJUea5GjRry1ltvye+//67z+DFOIkbpqFq1qowdO1ZOnz4d+i0lIiLLXEEudpCuCkmllJ7YEr2w8W90bh43bpzOIMG4WERERBEVwDBkCOo5CxcuLL1799YlMswNtnTpUtm5c6duI+vZs2fot5aIiCJ2NPprzXIbWJUqVfTQIE2bNtXVh+iIhg5sZm3bttXtY0REFF4uB88HZjmAPfTQQzphA/n8afXKNnpVExFRGCkmcWho70Kqo+8IxUREFMGU86oPLZfAsmTJIhcuRGaaMxERZa4qRMtJHOj3NXLkSLl8+XLGbBEREVFGtIGtXbtWD9D47bff6oSO7Nmze/199uzZVt+SiIgyinJuG5jlAIYBFzF8FBERRT6Xg6sQLQcwDCFFREQ2oZxbAguqIzPavxYtWiSTJ0/WM2sChpU6e/ZsqLePiIhCUAJzWVwcWQLbv3+/3HXXXXpo/KSkJGnSpInkzJlTJ3bg8aRJkzJmS4mIyDrFEpgHRtjAhGWnTp2SrFmzetbfd999XrNxEhERRVQJbPny5bJy5Uo9hbRZyZIl5fDhw6HcNiIiSi/FEpgHhohKTk5OsY8wpQqqEq0YNGiQuFwur6VChQo8YImIQsTl4DYwy0kcGMR3zJgxnscIOkjeGDhwoLRo0cLyBtx4443yxx9/eJYff/zR8nsQEVEqOBr9FaNGjZJmzZpJpUqV9LBS7dq101OoYADfTz/9VKyKiYmRQoUKBfRcJIlgMXBMRiKitLmU0osVVp9vmzawokWLyi+//CIzZsyQX3/9VZe+unTpIu3bt/dK6ggUgl9CQoLEx8dLvXr1ZPjw4VK8eHG/z8XfXn31VcufQUSUaSnntoHFBPWimBh55JFH0v3hdevW1aPbly9fXlcfIjjdeuutsnnzZr/taf369ZM+ffp4lcAw+zMREWU+lgPYhx9+mObfO3ToEPB7NW/e3PPvqlWr6oBWokQJ+eyzz3SpzldcXJxeiIgoMBxKysR3pmXMEXb+/HmdVp8tWzZLAczfOIs33HCD7Nq1K+j3ICKizFGFaDkLER2YzQvawLZv3y633HJLUEkcZniv3bt3S+HChdP1PkRE9A+m0V9FuXLlZMSIESlKZ1fz3HPPydKlS2Xfvn26czRG84iOjpa2bdtaeh8iIkoF0+gDS+zAgL5WoPMzgtWff/4p+fPn16W41atX638TEVH6uTidyhXz5s3z2jlKKZ1BOG7cOGnQoIGlHYtUfCIiomuShXjvvfd6PcZIHCgxNWrUSHdyJiKiCKKcm8QRE8xYiEREZB8umwSka9KRmYiIbEKpfxarr3FiADOPhHE1o0ePtvr2REQUQi4mcVyxceNGvaADM4aAgh07duj095o1a17ZaS4XD0IiIoqcElirVq30OIUffPCBXHfddXodOjR36tRJj2P47LPPZsR2EhFRMJRzkzgsj8SBTEOMCm8EL8C/hwwZwixEIqII43IHtziyBIYR4I8fP55iPdadOXMmVNtFREShoFgC88BwT6gunD17th5JA8usWbP06PH3338/DzgiogjiUsEtjiyBTZo0SY9hiJmYkcih3yQmRgewN954IyO2kYiIgqWYRu+BKVMmTJiggxVGjocyZcpI9uzZeYAREVHkd2RGwMIklEREFLlc7AcW2S4UzCYxMfESabLuOSmRKNpy7um1c754LolE2XefkkjlOn9BIlbpkhKJLu/ZJ5mGcm4SB4eSIiJyMBdLYEREZEuKSRxERGRDLgeXwCK4NYSIiCh1bAMjInIyxSQOIiKyIZeDqxBZAiMicjK3+mex+hobYAAjInIyxSpEIiKyIVcQVYJ2mY6YWYhERGRLDGBERJmhI7OyuARh/PjxUrJkSYmPj5e6devKmjVrUn3uu+++K7feequeEBlL48aN03y+PwxgREQO5rpG84HNnDlT+vTpIwMHDpQNGzZItWrVpFmzZnLs2DG/z1+yZIm0bdtWfvjhB1m1apUUK1ZMmjZtKocPHw74MxnAiIgyQxKHsrhYNHr0aOnataue8LhSpUp67khMvzV16lS/z//kk0/k6aeflurVq0uFChXkvffeE7fbLYsXLw74MxnAiIgczKVUUAskJiZ6LUlJSX4/4+LFi7J+/XpdDWiIiorSj1G6CsT58+f1JMl58+a1TwBDcfGRRx6R66+/XrJmzSpVqlSRdevWhXuziIicwR3kIqKr9XLnzu1Zhg8f7vcjTpw4IcnJyVKwYEGv9Xh85MiRgDazb9++kpCQ4BUEI7of2KlTp6RBgwZyxx13yPz58yV//vyyc+dO3aBHREThdfDgQcmV68ocfXFxcRnyOSNGjJAZM2bodjEkgNgigI0cOVJH+GnTpnnWlSpVKpybRETkKC5TlaCV1wCClzmApSZfvnwSHR0tR48e9VqPx4UKFUrztW+++aYOYIsWLZKqVata2s6wViHOmzdPateuLa1bt5YCBQpIjRo1dGplalD/6lsnS0RE4U3iiI2NlVq1anklYBgJGfXq1Uv1da+//roMHjxYFixYoGOBVWENYHv27JGJEydKuXLlZOHChfLUU09Jz5495YMPPvD7fNS/mutjUXojIqLw9wNDCj0KILh+b926VV/Pz507p7MSoUOHDtKvXz+vGrj+/fvrLEX0HUNbGZazZ8/aowoRERpRd9iwYfoxSmCbN2/W6ZcdO3ZM8Xx8eewkA0pgDGJEROEfjb5NmzZy/PhxGTBggA5ESI9HycpI7Dhw4IDOTDSg8ILsxQcffNDrfdCPbNCgQZEfwAoXLqz7C5hVrFhRZs2a5ff5aEDMqEZEIiJHUkGUqIIciaN79+568QcJGmb79u2T9AprFSIyELdv3+61bseOHVKiRImwbRMREdlDWANY7969ZfXq1boKcdeuXTJ9+nR55513pFu3buHcLCIix3C5g1vsIKwB7KabbpIvv/xSPv30U6lcubLORhkzZoy0b98+nJtFROQc6toN5pvpJrT8z3/+oxciIsoAihNaEhFRJuvIHOnCXgIjIiJnZCFea2EfzJeIiCgYLIERETmZujK6vKXX2AADGBGRg7nYBkZERPbNQlTWX2MDLIERETmZcm4SBwMYEZGTuVGPGMRrbIBZiEREZEssgREROZiLSRxERGRLim1gRERkR4oBLKJl23FMYqIib6LLpFL5JRLFHj0jkSrHpj8kErlzZpdI5XJHbsaYio/MVgpVv5pEGnX5gshPczPgjRWzEImIyIbczEIkIiKKKJFZviciopBwMQuRiIhsSbENjIiI7MitUAyz/hobYBUiEZGTKZbAiIjIllQQg/PaowTGsRCJiMiWWIVIRORkilWIRERkR249o2UQr4l8LIERETmZcv+zWH2NDTCAERE5mWIVIhER2ZHbuVWIzEIkIiJbYhUiEZGTKedWIYa1BFayZElxuVwplm7duoVzs4iInEOZgljAi9hCWEtga9euleTkZM/jzZs3S5MmTaR169bh3CwiIudQzi2BhTWA5c/vPWPxiBEjpEyZMtKwYUO/z09KStKLITExMcO3kYjI1txIiXcH8ZrIFzFJHBcvXpSPP/5YOnfurKsR/Rk+fLjkzp3bsxQrVuyabycRka0oq9WHwYydmMkD2Jw5c+Svv/6Sxx57LNXn9OvXT06fPu1ZDh48eE23kYiIIkfEZCFOmTJFmjdvLgkJCak+Jy4uTi9ERBQgtoFlrP3798uiRYtk9uzZGfxJRESZjNu5HZkjogQ2bdo0KVCggLRs2TLcm0JE5ChKufVi9TV2EPYA5na7dQDr2LGjxMSEfXOIiJxXhehmGn2GQNXhgQMHdPYhERGFmAqiCtEmWYhhL/I0bdpUlE12FhERRY6wBzAiIspAbreIi/OBERGR3ShWIRIRkQ0pt1uUxRIYsxCJiCj8lHNLYBEzlBQREZEVTOIgInIytxJxObMExgBGRORkCsHIahYiAxgREYWZcitRFktgdumbyzYwIiInU+7gliCMHz9eSpYsKfHx8VK3bl1Zs2ZNms///PPPpUKFCvr5VapUkW+++cbS5zGAERE5vQTmtr5YNXPmTOnTp48MHDhQNmzYINWqVZNmzZrJsWPH/D5/5cqV0rZtW+nSpYts3LhR7r33Xr1s3rw54M9kACMionQbPXq0dO3aVTp16iSVKlWSSZMmSbZs2WTq1Kl+nz927Fi566675Pnnn5eKFSvK4MGDpWbNmjJu3LjMkcRh1NNedl+USHT58gWJRFHJSRKpXO7I3DZ3cuSeKlERus9AJbskEl2+HC2R5vLlpAxpf7qskixXCV6WS/q/iYmJAU0qfPHiRVm/fr3069fPsy4qKkoaN24sq1at8vsZWI8SmxlKbHPmzAl4OyP3rAzAmTNn9H+XHHpPItKBcG8AEdnxupY7d+50v09sbKwUKlRIfjxirV3JkCNHDilWrJjXOlQPDho0KMVzT5w4IcnJyVKwYEGv9Xi8bds2v+9/5MgRv8/H+kDZOoAlJCTIwYMHJWfOnOJype9OD3ca+LHwfrly5QrZNjoZ9xn3GY+z0EHJC8EL17VQiI+Pl7179+rSUbDb43td9Vf6CidbBzAUUYsWLRrS90TwYgDjPstoPM64z/wJRcnLN4hhyWj58uWT6OhoOXr0qNd6PEYp0B+st/J8f5jEQURE6a6urFWrlixevNizzu1268f16tXz+xqsNz8fvvvuu1Sf77gSGBERRQYkZHTs2FFq164tderUkTFjxsi5c+d0ViJ06NBBihQpIsOHD9ePe/XqJQ0bNpRRo0ZJy5YtZcaMGbJu3Tp55513Av5MBjBT3S4aKCOtjjeScZ9xn/E4I0ObNm3k+PHjMmDAAJ2IUb16dVmwYIEnUePAgQO62cdQv359mT59urzyyivy0ksvSbly5XQGYuXKlSVQLmWXMUOIiIhM2AZGRES2xABGRES2xABGRES2xABGRES2xAAW5DQAmRnSYG+66SY9AkqBAgX0CNLbt28P92bZyogRI/QoB88880y4NyWiHT58WB555BG5/vrrJWvWrHrKDaRaEwEDWBDTAGR2S5culW7dusnq1at1x8NLly5J06ZNdZ8Purq1a9fK5MmTpWrVqtxdaTh16pQ0aNBAsmTJIvPnz5ctW7boPkPXXXcd9xtpTKMX0SUulCiMYfzRgxzjIvbo0UNefPHFf/YUpQp9P1ASQ2C77bbbuKfScPbsWT1lxIQJE2TIkCG6rww6fFJKOPdWrFghy5cv5+4hvzJ9CcyYBgDD/gc6DQB5O336tP5v3rx5uWuuAiVXjDpgPt7Iv3nz5ulRHVq3bq1vkGrUqCHvvvsudxd5ZPoAltY0AFaG9c+sUFpFOw6qeqz0oM+MMFQOqqiNoXQobXv27JGJEyfqERoWLlwoTz31lPTs2VM++OAD7jrSOJQUpbtEgSnAf/zxR+7JNGCaHoz9hjbDazE6uFNujlACGzZsmH6MEhiONcz0izH3iDJ9CSyYaQDoH927d5evvvpKfvjhh5BPa+M0qKZGUhDav2JiYvSCNsO33npL/xu1AOStcOHCemp6M0w9jzH1iCDTB7BgpgHI7DB8JoLXl19+Kd9//72UKlUq3JsU8e68807ZtGmT/Pzzz54FpYv27dvrf+MmiryhWtq3e8aOHTukRIkS3FWksQoxgGkAKGW1IUaRnjt3ru4LZrQVYjI+9NWhlLCffNsIs2fPrvs3se3Qv969e+sRy1GF+NBDD+m+mZhqw8p0G+RwGI2elHr77bdV8eLFVWxsrKpTp45avXo1d0sqcNj4W6ZNm8Z9ZkHDhg1Vr169uM/S8H//93+qcuXKKi4uTlWoUEG988473F/kwX5gRERkS5m+DYyIiOyJAYyIiGyJAYyIiGyJAYyIiGyJAYyIiGyJAYyIiGyJAYyIiGyJAYyIiGyJAYyCdvvtt+upVK41jFOJQV2dNACuy+WSOXPmhG1OvJIlS8q6devC8vlEwWIAI9t54YUX5JVXXrmmA+DiAh/IzMnhDETpGdD6ueeek759+4Z7U4gsYQAjW8G8Y7t375YHHngg3JviKBgVH/v2t99+C/emEAWMAYxC5uuvv9Yj0n/yySeeSRwxiniePHkkb968cs8998i+ffv035YtWyZZsmRJMes1qiRvvfXWNGc1btKkidekkIMGDZLq1avL5MmTpVixYpItWzb9uadPn/Y8Z+3atfp1mP8N29iwYUM9O7IBYxTjfYoXLy5xcXGSkJCgZ/81qkr379+vR0dHCQtLaqU0uO+++/RzjMeAmYXLlCmjSzvly5eXjz76KM19OXDgQD0f1q+//qofI7hgv2C0f3xHbBtmTDB/NkZt79y5sx75Ht/DPGo7qgkxBQ7eE/sOU5KYZ4a+7rrr9PQl2L9EtnFlXF+i4EdT/+STT1TOnDn16OFw8eJFVbFiRdW5c2f166+/qi1btqh27dqp8uXLq6SkJP2cG264Qb3++uue98Nr8uXLp6ZOnZrqZ1atWlWNGDHCa93AgQNV9uzZVaNGjdTGjRvV0qVLVdmyZfXnGRYvXqw++ugjtXXrVr0tXbp0UQULFlSJiYn6759//rnKlSuX+uabb9T+/fvVTz/95Bn5/M8//1RFixZVr732mvrjjz/04s+xY8c8o/LjOXgMs2fPVlmyZFHjx49X27dvV6NGjVLR0dHq+++/97wWr/vyyy+V2+1W3bt3VyVLllQ7d+7Uf9u1a5f+fv/73//Ujh071IoVK1SNGjXUY4895nl9iRIlVN68efVn4HXDhw9XUVFRatu2bfrvb7zxhipWrJhatmyZ2rdvn1q+fLmaPn261/b37dtX/6ZEdsEARukOYOPGjVO5c+dWS5Ys8fwNwQLBChdkAwJX1qxZ1cKFC/XjkSNH6iBnmDVrlsqRI4c6e/Zsqp+Jz/nwww9TBDAEhEOHDnnWzZ8/X1/AUws2ycnJXgEXQQUBFUHUHwQIBJCrMQKRWf369VXXrl291rVu3Vq1aNHC63UIogi62Cfm74Jg+9///tfr9QhA+H5///23Z/seeeQRz9+x3wsUKKAmTpyoH/fo0UMHePPv4Wvs2LE6cBLZBasQKV2++OILXbX23Xff6Wo5wy+//CK7du3S1Vk5cuTQC6oRL1y4oNuw4LHHHtPPWb16tX78/vvv66o/TPSYmr///tur+tCAKrMiRYp4HmM2bcysbczoe/ToUenatauUK1dOVyHmypVLzp4965mevnXr1vq9S5curZ+H2aYvX74ckqNj69atunrODI+x3gz78aefftLVq+bvgn2JfWPsRyzNmjXT32/v3r2e51WtWtXzb1RhFipUSI4dO+bZ15j5GdWXqH789ttvU2wnqifPnz8fku9MdC0wgFG61KhRQ/Lnzy9Tp07V7UgGBIdatWrpi6Z5wZTw7dq1088pUKCAtGrVSqZNm6YDzPz583UbTlrQhnXq1CnL24kZt/H5Y8eOlZUrV+p/YzZktA0B2pUQ7CZMmKAv5E8//bTcdtttcunSJblW0EZ3+PBhWbhwodd67MsnnnjCaz8iqO3cuVO3qxnQpmiGIIYgBzVr1tTBbvDgwTpQ40bhwQcf9Hr+yZMn9W9JZBcx4d4AsjdcQEeNGqUTHZDWPm7cOM8Fc+bMmTpIobSTmscff1zatm0rRYsW1e/lW1LxFzC3bNmSYj1KUr///rtOvgCU6qKionSJA1asWKGDU4sWLTwJJidOnPB6DwQuBFQs3bp1kwoVKsimTZv0d0HyRSD9zhBEfJ+HPmv4fARRAx5XqlTJ63l33323/mwEeOzLhx9+WK/H5+M7ly1bVtIDv0ObNm30guB111136aCFkjFs3rxZ718iu2AJjNLthhtukB9++EFmzZrl6diMtGyUlpB5uHz5cn33v2TJEl19dejQIc9rURWGC+uQIUOkU6dOV/0sPB8Zeb5QrYgAgZIJPg+fg1IGqtEAVYfI/EO1HarpsH0IWAZU0U2ZMkVfxPfs2SMff/yx/juy9YwsP1TtoYTkG/jM8Dx0tEZ2pVFSfP755/X7IxMRpabRo0fL7Nmzdd8rX8hgxHZiX6B6FtA/C6VGZBGi9IX3mDt3rn4cKHzmp59+Ktu2bdOl4M8//1zvG2SIGrDfmjZtGvB7EoVduBvhyBlZiIDsPiQO9OnTRz9GAkWHDh10ZmFcXJwqXbq0TmY4ffq01/v0799fJ2H8/vvvV/1MZATGx8d7suuMJI5q1aqpCRMmqISEBP33Bx98UJ08edLznA0bNqjatWvrv5UrV04nTJgTM5B4UbduXZ2JiIy/m2++WS1atMjz+lWrVukMSHyPtE6befPm6QzImJgY/f4GbBu+P7IRkSzim4jim/wxc+ZMva1IbIE1a9aoJk2a6CQXbB+2ZejQoWkmmWCfYN8AMiqrV6+uX4vveOedd+p9Yli5cqXKkyePOn/+/FV/A6JI4cL/hTuIUubWpUsXOX78uMybNy+g56NEk5iYqPt9AfpvYfQLlE4oOKhWrFatmrz00kvchWQbrEKksEFHY1QHTp8+XXr06BHw615++WVdtWckKFD6IJGlSpUqOguSyE6YxEFhg/axNWvWyJNPPqkz8AKFdhuWFEIHCSoYW5LIbliFSEREtsQqRCIisiUGMCIisiUGMCIisiUGMCIisiUGMCIisiUGMCIisiUGMCIisiUGMCIiEjv6fxV/fgBPtPdTAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize who attends to whom (row = query token, col = key token)\n",
    "fig, ax = plt.subplots(figsize=(5,4))\n",
    "im = ax.imshow(attn[0], cmap=\"viridis\")\n",
    "ax.set_xlabel(\"key (past tokens)\"); ax.set_ylabel(\"query (current token)\")\n",
    "ax.set_title(\"attention weights (lower-triangular = causal)\")\n",
    "fig.colorbar(im); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15f0a0a1",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>fig, ax = plt.subplots(figsize=(5,4))</code>: make a single plot panel to draw the heatmap on.</li>\n",
    "<li><code>ax.imshow(attn[0], cmap=\"viridis\")</code>: draw the first sequence's (T,T) attention grid as colors: brighter = more attention.</li>\n",
    "<li><code>ax.set_xlabel(...); ax.set_ylabel(...)</code>: label the axes: columns are the past tokens being looked at, rows are the token doing the looking.</li>\n",
    "<li><code>fig.colorbar(im); plt.show()</code>: add the color scale and render. The bright lower triangle and dark upper triangle <i>is</i> causality, made visible.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7411eceb",
   "metadata": {},
   "source": [
    "## Package it as a function\n",
    "\n",
    "Those few lines are one **attention head**: query, key, value, score, mask, softmax, blend. Here it is wrapped into a single function that returns exactly the same output. We will reuse this precise logic in PyTorch later, where it shrinks to about six lines and the gradients are handled for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "14835fe6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-01T09:02:50.152769Z",
     "iopub.status.busy": "2026-07-01T09:02:50.152712Z",
     "iopub.status.idle": "2026-07-01T09:02:50.154648Z",
     "shell.execute_reply": "2026-07-01T09:02:50.154387Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output: (4, 8, 16)  same as before: True\n"
     ]
    }
   ],
   "source": [
    "def attention_head(x, W_q, W_k, W_v):\n",
    "    B, T, C = x.shape; hs = W_q.shape[1]\n",
    "    q, k, v = x @ W_q, x @ W_k, x @ W_v\n",
    "    scores = (q @ k.transpose(0,2,1)) / hs**0.5\n",
    "    scores = np.where(np.tril(np.ones((T,T))) == 0, -np.inf, scores)\n",
    "    return softmax(scores, -1) @ v\n",
    "\n",
    "out2 = attention_head(x, W_q, W_k, W_v)\n",
    "print(\"output:\", out2.shape, \" same as before:\", np.allclose(out, out2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb0766ca",
   "metadata": {},
   "source": [
    "<details class=\"code-read\"><summary>Line by line: what each line does</summary>\n",
    "<ul>\n",
    "<li><code>def attention_head(x, W_q, W_k, W_v):</code>: the previous cells folded into one reusable function.</li>\n",
    "<li><code>B, T, C = x.shape</code>: read the input's three sizes back out (tuple unpacking).</li>\n",
    "<li><code>hs = W_q.shape[1]</code>: read the head size off the projection grid instead of hard-coding it.</li>\n",
    "<li><code>return softmax(scores, -1) @ v</code>: the same project &rarr; score &rarr; scale &rarr; mask &rarr; softmax &rarr; blend, compressed.</li>\n",
    "<li><code>np.allclose(out, out2)</code>: the function reproduces the step-by-step result exactly.</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ec4ed5",
   "metadata": {},
   "source": [
    "## Recap\n",
    "\n",
    "Self-attention in one sentence: each token builds query, key, and value vectors, scores every earlier token by query-key similarity, hides the future, turns the scores into weights with softmax, and returns the weighted sum of the values. One head mixes information in one particular way, for example \"pronouns, find your nouns.\"\n",
    "\n",
    "Next, notebook 06 runs several heads in parallel (so the model can look for several kinds of relationship at once), adds position information and a small feed-forward layer, and wraps it all into the **Transformer block** that we stack to build a GPT."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (.venv)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}