{ "cells": [ { "cell_type": "markdown", "id": "7d22be6a", "metadata": {}, "source": [ "# 06 - The Transformer block\n", "\n", "A GPT is just a stack of identical **Transformer blocks**, one on top of the next. You can picture each block as a short team meeting with two phases:\n", "\n", "1. **multi-head self-attention**, where everyone shares information (each token gathers context from the others),\n", "2. a **feed-forward network**, where everyone goes back to their desk and thinks it over alone.\n", "\n", "Two supports keep the meeting orderly: **residual connections** (keep the original notes and just add edits on top) and **LayerNorm** (keep everyone's numbers on a similar scale). And because attention on its own has no sense of word order, we first stamp each token with its **position**. We will build every piece in NumPy and run one full block's forward pass." ] }, { "cell_type": "code", "execution_count": 1, "id": "4dc377e4", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.173507Z", "iopub.status.busy": "2026-07-01T09:08:58.173301Z", "iopub.status.idle": "2026-07-01T09:08:58.221299Z", "shell.execute_reply": "2026-07-01T09:08:58.220808Z" } }, "outputs": [], "source": [ "import numpy as np\n", "np.random.seed(1337)\n", "def softmax(x, axis=-1):\n", " x = x - x.max(axis=axis, keepdims=True); e = np.exp(x); return e / e.sum(axis=axis, keepdims=True)\n", "\n", "B, T, C = 4, 16, 64 # batch, time, embedding dim\n", "n_head = 4" ] }, { "cell_type": "markdown", "id": "079258f1", "metadata": {}, "source": [ "
Line by line: what each line does\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "68225a11", "metadata": {}, "source": [ "## Positional encoding\n", "\n", "Attention is **order-blind**. It is just a weighted sum, so if you shuffled the tokens it would return the same outputs (shuffled the same way); it has no idea which token came first. But order carries meaning, since \"dog bites man\" and \"man bites dog\" are very different. So before anything else we stamp each token with its position, adding a position signature onto its embedding (the embedding being the list of numbers that represents the token).\n", "\n", "The classic method uses **sinusoids**, which are simply waves of different speeds. Think of a car's mileage counter: a fast-spinning ones wheel, a slower tens wheel, and a slower-still hundreds wheel. Read together, the mix of fast and slow wheels gives every position a unique fingerprint, and nearby positions get similar fingerprints. (A GPT actually learns its own table of position signatures instead of using fixed waves, but both achieve the same thing.)" ] }, { "cell_type": "code", "execution_count": 2, "id": "437e3be0", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.222674Z", "iopub.status.busy": "2026-07-01T09:08:58.222582Z", "iopub.status.idle": "2026-07-01T09:08:58.399218Z", "shell.execute_reply": "2026-07-01T09:08:58.398783Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAE8CAYAAABD1oIBAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAT7dJREFUeJzt3Qd4FMX7B/D3LpWEJEBC702KQECaFBsgRUTABohSpEhTmgj4A0FRQq8iKAjIXxRFBQWlV8HQRem9SW8JkJB2t//nHbPnlbnLbsiR3N338zwrubnJZnfvvH1vZt4Zg6IoCgEAAABkwJhRBQAAAAAEDQAAAKAZWhoAAABAEwQNAAAAoAmCBgAAANAEQQMAAABogqABAAAANEHQAAAAAJogaAAAAABNEDT4uC5dulCpUqXIU5w9e5YMBgMtXLjQrefGv8e/nxPpOa/Ro0eL6+Vpr50n4vPi8+PzVD399NNiA/AWCBoAPFxiYqIIDjZv3pzdhwIAXs4/uw8AstfcuXPJbDZ7zMtQsmRJun//PgUEBJCvsn/NOGj48MMPxc/232pHjBhBw4YNe+jHCP9au3YtLgV4FQQNPs7Tbr7c/BscHEy+TM9r5u/vLzbIHoGBgbj04FXQPeHF7t69SwMGDBD930FBQVSgQAF69tlnad++fU77x9V+50mTJtEXX3xBZcuWFb9bu3Zt2r17t83+nfXXyvrclyxZQjVr1qSwsDAKDw+nqlWr0vTp023qnD59ml555RXKly8fhYSE0OOPP06//vqrpn7x5cuXU5UqVURAwf8uW7ZMek34vOrXr0+RkZGUK1cucUw//PADZYb1tZo6dapoBeF9PvXUU3Tw4EGH+hs3bqQnnniCQkNDKU+ePNS6dWs6cuTIA71mfAz58+cXP3NrAx8Pb9xd4WxMQ1paGo0ZM8by2vK+3n//fUpOTrapx+XPP/88bdu2jerUqSOubZkyZWjRokU29W7dukXvvvuueE1z584tXt8WLVrQX3/9RZkVFxcnrkPx4sXFMZYrV47Gjx9v08Ki573Kjh49Sq+++qq4Xvw6VahQgf73v//Z1Pnzzz/FsfM58Lk0btyYduzY4bCvQ4cOUaNGjcR+ihUrRh9//LG0xc7+/xHuQuJj/v777+mTTz4Rv8vXlf/OyZMnHX5/1qxZ4prz3+HX4Pfff8c4CchW+ArixXr16iVuiP369aPKlSvTzZs3xQ2Ab1SPPfaYy9/95ptvxA3srbfeEh9yEyZMoBdffFHc2PW2Tqxbt446dOggPhj5g5/xMWzfvp369+8vHl+9elXczLmp/Z133hE39a+++opeeOEFcQ5t27Z12QT80ksviXOMiYkR59m1a1fxgWyPAxXeZ8eOHSklJUUEMxyorFy5klq2bEmZwTdRvlZ9+/alpKQk8Tf4hnLgwAEqWLCgqLN+/XpxM+IbAN/IuYtl5syZ1KBBAxEQqEGA3teMb4CzZ8+m3r17i2vErxGrVq2a0+Pt3r27uLYvv/wyDR48mHbu3CmuG/8N+2CLb2Rcr1u3btS5c2eaP3++CFo42Hr00UdFHX5PcNDG17F06dLitfz8889F8HT48GEqUqSIruvJ7wH+3YsXL4r3X4kSJeiPP/6g4cOH0+XLl2natGm636t///23CNj4cc+ePcX1PnXqFK1YsULcvNVAgOtwwPDee++JunwefNPfsmUL1a1bV9S7cuUKPfPMMyL44q4fDgI5aOEbu1bjxo0jo9Eogq34+HhxzPye5NdCxa8rvw/4mAYOHCiCpDZt2lDevHml722Ah0IBrxUREaH07dvXZZ3OnTsrJUuWtDw+c+aMwm+LyMhI5datW5byn3/+WZSvWLHCUvbUU0+JLaN99u/fXwkPD1fS0tKcHseAAQPE/n///XdL2d27d5XSpUsrpUqVUkwmk83xLViwwFKvevXqSuHChZW4uDhL2dq1a0U96+NgiYmJNo9TUlKUKlWqKI0aNbIp59/j83BFPZZcuXIp//zzj6V8586donzgwIE2x1igQAHl5s2blrK//vpLMRqNSqdOnR7oNbt+/br4e6NGjXKoy2XW/5vv379fPO7evbtNvXfffVeUb9y40eYacNnWrVstZdeuXVOCgoKUwYMHW8qSkpIsr4/1teF6H330kcP1sn7tZMaMGaOEhoYqx48ftykfNmyY4ufnp5w/f173e/XJJ59UwsLClHPnztns02w2W35u06aNEhgYqJw6dcpSdunSJfF7/Pv271V+na2vC792XM7H5ez/kU2bNok6lSpVUpKTky3l06dPF+UHDhwQj/k5Pq/atWsrqamplnoLFy4U9WT/3wE8DOie8GLcBM7fXC5duqT7d9u1aye+0aj42w7jb2+ZOY6EhATR4uDMb7/9JppfGzZsaCnj5mH+VsjfsPgbqwx/89y/f7/4FhwREWEp5yZ9/qZuz/rb4O3bt8W3PD436+Z/vfjbX9GiRS2P+Tz4Wymfk/Ux8jd07npRcWsAH6da70FfMy3UvzVo0CCbcm5xYPbdQXwN1ddebdngZn3r9wF3CfC3ZmYymUTrCL92XC8z13Xp0qXib/L778aNG5atSZMmYv9bt27V9V69fv26+J0333xTtFpYU7tueL/cYsWvJbcGqQoXLkyvvfaaaO25c+eO5Rpy1xm/ztbXhVsKtOKWMOvxDvbHvGfPHnEde/ToYTMmhf+G9bkCPGwIGrwYN3ly3zr3C/MHHDeLa73p23+4qh9UfKPVq0+fPvTII4+I5nluVuUP79WrV9vUOXfunLjJ2KtUqZLleRm1vHz58g7PyfbH3RD8gc/9yHwDV5v3OXjILNnf5vNV8/XVY3R2fnxD5KDqQV8zLfhY+AbPYwSsFSpUSAQs9tfZ/n2gvhes3wfcl89jOvg6cAARFRUlrit3CWTmup44cUK8P3gf1hsHDezatWu63qvq9eOxLs5wYMHdIs5eIz7HCxcuiMd8jbS+35zJ6JjV18H+deIAwpPmVQHvg6DBi/GgL/7A5L5z7leeOHGi6IdetWpVhr/r5+cnLVcUbh39l7NJg/hbmzUezMfftH/55RcxnmDTpk0igODWgYeJB5Hx3+eA4bPPPhPfGLn1g79JWp+Xp75memid8EnL+2Ds2LGi5eLJJ5+kr7/+mtasWSOuKx93ZtJ5+Xe4BYb3Idt4/IreY8xpPPGYARgGQno5bl7lb/q88Tc0HkzHA7/4pv2g+NuR7FuwrFWAm2JbtWolNr4p8PHwILORI0eKb1OceXDs2DHpiHfGz8uo5fzt1J79/n788UcRMPBNjb8RqxYsWEAPQva3jx8/bvlGqB6js/Pjb+Y8mC6zr5meGR/5WPj68zGrrTiMBy9yxoKz6+wKD9zkgYFffvmlTTnvj89NL86CuHfvnqVl4UGp3Q2yjBYVt2Rwxo6z14hbZ7j1h/E10vJ+exDq68ADUfnaqnjwJbdguRroCuBOaGnwUvxt375pmL/x87dX+9S6zOIPd/5A5aZdFafZcVaENe6btcYfwOqHnnoszz33HO3atYtiY2Mt9bjJnkel881XNj5BvcFWr15dZANYny9/I7UfB8Hf7vgGa90Swh/APPL/QfDv80h/FZ8Hj0tQb/LWx8g3UhXfxLgfnc/9QV4zvtkx6307o/4t+wyEKVOmiH8zk0HC19X+GzKPS7C+JnpbW/h9wMGdPT5HvnHqwQEBt4Jw5sf58+dtnlOPm8+hadOm9PPPP9tMA83BFGdn8FgbzqpQryGnYfLrrOL/BxYvXkxZpVatWiKDiCfysj5f/huZ6SIEyCpoafBSnILG4wc4XS46OloMTOO0P85fnzx5cpb8DR6bwDebZs2aiZQ8/lY8Z84c0SytDhpTU/w4l5/TEPmYuCWCm9/5Rqp+2+XUtW+//VbcaDnlkscb8E32zJkzooVAHWgnw+mCfLPjD3Y+Jv5bvH8+Dv7GquI6fLzNmzcXXRJ8vJwHzy0d3P+eWfz7/Lc57ZFv7nxD5g98TttTcTcDn1u9evXEtVJTLnnwpjqnQmZfMx7cyUHVd999J8ZS8LXj/ntZHz7vl7uFOBjjGzCnNvLNj681DwK0/larFc/l8NFHH4nBfZw2y6mmfHOzHlCox5AhQ0RXFu9XTe/kAJL3y60afFPX24IxY8YM8Rpxqw0PruXUUN4PD/zkrjPGcy1wsMn1uJWHxw9waxi/pjzWRMWv6//93/+J9xGnDKspl9w68CDvI/uWOX5fvP322+L/Gw6k+Hh5fhIO1nPKeiLggx5KjgY8dJyyNWTIECU6OlqkjHEKG//82WefaUq5nDhxosM+ZWl9X3/9tVKmTBmRqsZphWvWrHHY5w8//KA0bdpUpBxyvRIlSihvvfWWcvnyZZt9carbyy+/rOTJk0cJDg5W6tSpo6xcudKmjrO0vR9//FGksXGaX+XKlZWffvrJ4TjYl19+qZQvX17Uq1ixotiPfVqi3pRLvlaTJ09WihcvLvb7xBNPiHRKe+vXr1caNGggUjQ5BbVVq1bK4cOHH/g1Y3/88YdSs2ZNcX2tXyfZuXEK34cffijSWQMCAsRxDx8+XKRO2l+Dli1bOpyHfRoh/x6nYHLaK58bn2NsbKxDPa0pl2q6LR9TuXLlxDlFRUUp9evXVyZNmiTSZO2vv5b36sGDB5W2bdta3l8VKlRQRo4caVNn3759SrNmzZTcuXMrISEhyjPPPCOurb2///5bnBvvp2jRoiJNlN9bWlMuly5darM/Z9dmxowZ4nXg9xX//7B9+3bxOjdv3jzDawjgDgb+T3YHLgCeiL/58TdWbkXgSXoA3I3Ho3B3C09exV0XAA8bxjQAAORAPLuo/Xc6nn2Uu9+w3DZkF4xpAADIgXiwJU8fzdNz8xgZniiLM1R4rAqXAWQHBA0AADkQZw1xmicP4uTWBR7g2qlTJ7FuBVbPhOyC7gmAB/hQ5+ZjjGcAd72/OIuEF8jixdX4X04b5TRc8Dxbt24V89RwCjVnv2hJ9eZVUTnjR13p1X51X8YZYPxe4TloePp661Rgd0DQAAAA4GYJCQki5Zlv8lpwujmniXMaNKcF81LxnL5uPX8Jp1nzbKyjRo0S3Ve8f06Bt59qPSshewIAAOAhMhgMYhl6nhvFmaFDh4p5RKxnMm3fvr2YX0Vdu4dbFmrXrk2ffvqpJbuGu7R4fg+e+8YdvH5MA19EXjEwLCwME6IAAHgw7g7kSdC4id/VhG8PkrHCXUF6jsdgN9EWdyVYT1OfWTwrqv1U6tyKwC0OjI9z7969NHz4cMvzfE34d6xn1s1qXh80cMCgzhkPAACej1cc5dlTszpgyBWWjyjtvubfyZ07t82ss4y7CtRZXh8Ej2EpWLCgTRk/5tl2eUZZnk6cp56X1VHX7HEHrw8auIWBnTxxwvKzquizg6W/c3Gd45S9euq6uz6OBddF9h7Ijveis/o4FlwXd3x2KaZUMh3+3uGzPCuIFoa0+xRQpQORX0DGv2BKpXsHvxUBjLouCcuKVoaczOuDBrXpiN9k1i+seM4vUPo79vX01nV3fRwLrovsPZAd70Vn9XEsuC7u/Oxy59obxqBQp3/XmmJKsRy7s+N/EIUKFRILplnjx/y3eL0ZXmSNN1kd/l2fzp542CklAADgmwwGPzIYNWwGP7ceBy9ut2HDBpsyXlCNyxnP1cGLuVnX4TF8/Fit45NBQ3aklAAAgG8y+BnJ4OenYTPq2i+PfeDUSXVVVU6p5J/V5dp5QCNP3qXq1asXnT59WqyqymMUPvvsM/r+++/FLKEqvjfyGiS8Su2RI0fESruc2skrzvps0MBLGffo0UNcBF7+l5deDgkJEZOcAAAAZCWj0U/zpseePXuoRo0aYlNv+PzzBx98IB5fvnzZEkAwXgyPUy65dYG/LE+ePJnmzZsnvjSr2rVrR5MmTRL7qF69ughCOB3TfnCkz4xpyExKSXJysthUPNIUAABAC7X7IUNGfUEDLzLmalFp2WyP/Dt//vmny/3269dPbA9Ljm5puHHjhtOUEk5HkYmJiaGIiAjLhnRLAADQStN4BqPGwMIL5eigITO4VSI+Pt6ycToMAACAFgajUfPmi3J090RUVJTulJKsmo0LAAB8j7u6J7xFjg6VsiulBAAAfNO/rQhauieM5ItydEuDOsK0c+fOVKtWLapTpw5NmzbN7SklAADgmwz+gWQMyHhyJzOZyRfl+KCBU0quX78uUkp48COnlbg7pQQAAHyT1u4Jg492T+T4oCE7UkoAAMA3IWjwgqAhK0S//T0ZA3LZlL00oLe07hPjf3coq/Dsi9K63X84JC2PqlhXWv7ZnkvS8twFS0nLV52KcygLzJ1XWnfvlURpuV9gsLT8dLx8CVijv3yxlov30hzKnPXr3UwykR7xKfqa+u6laq9/3+Q8N1omWWd9PYeepm/XpOdQdB426ayuq747941jwXVxJ80TNxnR0gAAAODT1IGQWur5Ip9paQAAAMgIuidcQ9AAAACQDkGDawgaAAAA0iFocA1BAwAAQDqDQWPKpQEDIQEAAHyaMSBA0+ROpDhmk/kCtDQAAACkQ/eEawgaAAAA0iFocA1BAwAAQDqj0SC2DBk11PFCCBoAAADSGYwGsWXE4KNBg29OaQUAACBhMBg0b3rNmjWLSpUqRcHBwVS3bl3atWuX07pPP/209G+2bNnSUqdLly4Ozzdv3pzcCS0NAAAAVi0IWronFJ0tDd999x0NGjSI5syZIwKGadOmUbNmzejYsWNUoEABh/o//fQTpaT8tz7QzZs3KTo6ml555RWbehwkLFiwwPI4KCjIra8lWhoAAADSiW/sRg2bQV/QMGXKFOrRowd17dqVKleuLIKHkJAQmj9/vrR+vnz5qFChQpZt3bp1or590MBBgnW9vHnlCxpmFQQNAAAA6TQFDMb/xj3cuXPHZktOTna4ltxisHfvXmrSpMl/N1+jUTyOjY3VdO2//PJLat++PYWGhtqUb968WbRUVKhQgXr37i1aJNzJZ7onkm5dJoO/7RLRn5vPSetG/XzRoezC5pnSupU7TJOWDxpsGw2q5i2TL6X9SMN60vI5W047Ht8jtaV1fzt6TVqeu2BpafneS3ek5YG580nLz8UlOZT5BdouN666ed+ka9nt+GSTrpXkElO1L6acrHM96hSzvvppOuqbdO5bz3LXepeX1nkouurr3TdATuHnZxRbxhWN4p/ixYvbFI8aNYpGjx5tU3bjxg0ymUxUsGBBm3J+fPTo0Qz/FI99OHjwoAgc7LsmXnzxRSpdujSdOnWK3n//fWrRooUIRPz83DNjpc8EDQAAAFmdPXHhwgUKDw9365gCDhaqVq1KderUsSnnlgcVP1+tWjUqW7asaH1o3LgxuQO6JwAAANSbYvpASC0b44DBepMFDVFRUeKb/9WrV23K+TGPQ3AlISGBlixZQt26daOMlClTRvytkydPkrvk6KAhJiaGateuTWFhYaLPpk2bNmKkKQAAgDsYjNo3rQIDA6lmzZq0YcMGS5nZbBaP69WTd02rli5dKsZJvP766xn+nX/++UeMaShcuDD5ZNCwZcsW6tu3L+3YsUOMHE1NTaWmTZuKyAsAAMBT5mkYNGgQzZ07l7766is6cuSIGLTI9zLOpmCdOnWi4cOHS7sm+AtzZGSkTfm9e/doyJAh4v549uxZEYC0bt2aypUrJ1I53SVHj2lYvXq1zeOFCxeKFgcehfrkk09m23EBAIB34rHX2uZpIF3atWtH169fpw8++ICuXLlC1atXF/c4dXDk+fPnRUaFNW5Z37ZtG61du9Zhf9zd8ffff4sgJC4ujooUKSK+VI8ZM8atczXk6KDBXnx8vCV/1RluxrFOeeEUGAAAgOyeRrpfv35ik+HBi/Y4jVJR5KlIuXLlojVr1tDDlqO7J6xx/8+AAQOoQYMGVKVKFZfjICIiIiybfToMAADAw57cyVt4TNDAYxs4T5VHkbrCfULcIqFunA4DAACghdFg0Lz5Io/onuDmnJUrV9LWrVupWLFiLutyX467594GAAAvpbF7gnx0lcscHTRwX87bb79Ny5YtE/09POsVAACAu/j5G8WW4f3J32Ma6n0naOAuiW+++YZ+/vlnMVcDjzhlPFaBB4EAAABkJa3plAYf7Z7I0aHS7NmzxbgEXlecJ6tQN15iFAAAwBMmd/ImObqlwVmqCQAAgDtYTxGdUT1flKODhqy08fM+FBb236IibFjBatK6JftMdSi70v0lad27l4tIy997VB6Gjv1go7T8my+GSsvfmeRY/9E68rEdm/ZfkpZHlXlEWv77iRvS8pCootLyozccZ+IMDJPPmXE+7r603NmqmHFJadJyo3+gtDwh1ax5Rcz7aY51XdVP0bO0JBGl6lnlUmccbNIROOtdWVJvSO7OEB6rYoIvzNPgDXwmaAAAAMgIxjS4hqABAAAgHbonXEPQAAAAYDcjZEYMPpo9gaABAAAgnZ/RILaMKBjTAAAA4NsC/I1iy4gBkzsBAAD4Nq0tDWa0NAAAAPg2BA2uYUwDAACAelM08qZlTAP5JAQNAAAA6dDS4BqCBgAAAKt5GrSMaTD56JgGH21gAQAAcORnMJKfUcNm0H/7nDVrFpUqVYqCg4Opbt26tGvXLqd1Fy5caJmdUt349+zXZ/rggw/EQo688nOTJk3oxIkTbn1ZETQAAADYdU9o2fTg1ZkHDRpEo0aNon379lF0dDQ1a9aMrl275vR3wsPD6fLly5bt3LlzNs9PmDCBZsyYQXPmzKGdO3dSaGio2GdSUhK5C4IGAACATAYNd+7csdmSk5Ol13LKlCnUo0cP6tq1K1WuXFnc6ENCQmj+/PlOrz23LhQqVMiyFSxY0KaVYdq0aTRixAhq3bo1VatWjRYtWkSXLl2i5cuXu+31RNAAAACQLtDPqHljxYsXp4iICMsWExND9lJSUmjv3r2i+0BlNBrF49jYWHLm3r17VLJkSfE3ODA4dOiQ5bkzZ87QlStXbPbJf5+7PVzt80H5zEDIs82fp1A/P5uybs+Xk9YdNKqxQ9mUggOldUv2eUtafvjtftLy1ET5UtrPRTouO81eO77boeytdxtpXkbb1VLaB0/fkpbnLVpcWr7//G2HsuCI/NK65+PlzWMBoRHS8mv3UnQtjX03JU1z3aQ0fYs6O1sa29lS2mk61nXWs4w207Eytq5ltN29HLU7l9FmWEobsnsgpDG9zoULF0Q3giooKMih7o0bN8hkMtm0FDB+fPToUen+K1SoIFohuAUhPj6eJk2aRPXr1xeBQ7FixUTAoO7Dfp/qc+7gM0EDAABARrSOV/BLr8MBg3XQkFXq1asnNhUHDJUqVaLPP/+cxowZQ9kF3RMAAADpeGInrZtWUVFR5OfnR1evXrUp58c8VkGLgIAAqlGjBp08eVI8Vn/vQfbp9UHDuHHjxMCQAQMGZPehAACAF3JH9kRgYCDVrFmTNmzYYCkzm83isXVrgivcvXHgwAGRXslKly4tggPrffJATM6i0LpPr+6e2L17t2iW4f4dAACAnNA9oRWnW3bu3Jlq1apFderUEZkPCQkJIpuCderUiYoWLWoZSPnRRx/R448/TuXKlaO4uDiaOHGiSLns3r27eF79Av3xxx9T+fLlRRAxcuRIKlKkCLVp04Z8OmjgEaQdO3akuXPnigsEAADgDn4GjUGDQV/Q0K5dO7p+/bqYjIkHKlavXp1Wr15tGch4/vx5kVGhun37tkjR5Lp58+YVLRV//PGHSNdUvffeeyLw6NmzpwgsGjZsKPZpPwmUzwUNffv2pZYtW4rUkoyCBs6Rtc6T5eYaAAAAd2RP6NGvXz+xyWzevNnm8dSpU8XmCrc2cIsEbw9Ljg8alixZImbP4u4JLbhp58MPP3T7cQEAgPdxV/eEt8jRAyE5/7V///60ePFizc0tw4cPFzmt6sb7AAAA0CLQ30CB/kYNm8EnL2iObmngGbR4Xu7HHnvMZgTp1q1b6dNPPxXdEJzGYo0n1pBNrgEAAJARtDR4cNDQuHFjkWJijUeaVqxYkYYOHeoQMAAAADwIBA0eHDSEhYVRlSpVbMp4Fa/IyEiHcgAAgJw8ENIb5OigAQAA4KGnXBqyPuXSW3hc0GCflgIAAJBVjAaD2LTU80UeFzRk1uYzcRRksE0WeXr5b9K6YaPedCh7MipEWvfpnnWk5UtqD5aWR7Z9QVp+5bNx0vLURLNDWbPS8sVR4i4ckZa37tVAWj5p0Z/S8sKl80rLj190nPMid5R81c6T1+5KywNDnKxymSBf5dIvMJe0/Pb9VB2rXDpeQ2Yw+ula5TIrVq406VyeUc/KlToXuSTFR1bFxIqYoAd/KvhpiAf8fPSy+kzQAAAAoGWsgpbxCkaMaQAAAPBtGNPgGloaAAAA0mFMg2sIGgAAANSbop+BAjQMakjVMvDBCyFoAAAASIeWBtcQNAAAAKTjBgRN2RMG37xkCBoAAADSoaXBNQQNAAAA6bD2hGsIGgAAANKhpcE12ykSAQAAfJg6pkHLptesWbOoVKlSFBwcTHXr1qVdu3Y5rTt37lx64oknKG/evGJr0qSJQ/0uXbqQwWCw2Zo3b07uhKABAAAgHd94jRo2g861J7777jsaNGgQjRo1ivbt20fR0dHUrFkzunbtmtN1ljp06ECbNm2i2NhYKl68ODVt2pQuXrxoU4+DhMuXL1u2b7/91q2vJYIGAAAAuzENWjY9pkyZQj169KCuXbtS5cqVac6cORQSEkLz58+X1l+8eDH16dOHqlevThUrVqR58+aR2WymDRs22NQLCgqiQoUKWTZulXAnBA0AAADpAgwGCjBq2Az/Bg137tyx2ZKTkx2uZUpKCu3du1d0MVhuvkajeMytCFokJiZSamoq5cuXz6FFokCBAlShQgXq3bs33bx5062vJYIGAAAA9aaosXvCmB40cLdBRESEZYuJiXG4ljdu3CCTyUQFCxa0KefHV65c0XTthw4dSkWKFLEJPLhrYtGiRaL1Yfz48bRlyxZq0aKF+Fvu4jPZEyNXfUThobbLW+ft5vjisjdj9ziUTTu0SFpXSTsqLf/DyfLKtRrVkJbvGPmptDywQgvHwm3yPquk2/I3X+PStpGpauTFM9LyVk3LScs37rjgUBbuZMnw09cSpOXBEVHS8st3kqTlAblyS8tv30/TvDR2fLJjXVf1nS2N7Wwp7TRJfYNRHo/rXHWbTPJVvaXMepe6Jvctpe3Jy1F78rHDg/Mz/rtpqccuXLhA4eHhZN1dkNXGjRtHS5YsEa0KPIhS1b59e8vPVatWpWrVqlHZsmVFvcaNG5M7oKUBAABAvSkatLY2kMABg/UmCxqioqLIz8+Prl69alPOj3kcgiuTJk0SQcPatWtFUOBKmTJlxN86efKk215PBA0AAADqTdFgsCyP7Woz6sieCAwMpJo1a9oMYlQHNdarV8/p702YMIHGjBlDq1evplq1amX4d/755x8xpqFw4cK+GzRwesnrr79OkZGRlCtXLtEEs2ePY/cBAADAwx7ToBWnW/LcC1999RUdOXJEDFpMSEgQ2RSsU6dONHz4cEt9HqMwcuRIkV3Bczvw2Afe7t27J57nf4cMGUI7duygs2fPigCkdevWVK5cOZHK6ZNjGm7fvk0NGjSgZ555hlatWkX58+enEydOuD2lBAAAfJPeMQ1atWvXjq5fv04ffPCBuPlzKiW3IKiDI8+fPy8yKlSzZ88WWRcvv/yyzX54nofRo0eL7o6///5bBCFxcXFikCTP48AtE+4YV/FAQQNHR9zHwpENT0zBzSzWTp8+nSUHx5EWj0xdsGCBpax06dJZsm8AAICHOY10v379xCbDgxetceuBK9zyvmbNGnrYMhU0dO/eXaR2vPHGG6LvRO/MWFr98ssvopnllVdeEX+vaNGiYrILniDDGc6Rtc6T5bxZAAAALfh2puWWZsDS2NpxV8Gvv/4qug7ciVssuImG+4Lef/992r17N73zzjtiUEnnzp2lv8M5sh9++KFbjwsAALyTOtBRSz1flKmBkDymwH5WKnfgbo/HHnuMxo4dSzVq1KCePXuKVgaeftMZHkgSHx9v2TiHFgAAQHvKpbbNF2UqaOCBFjyYg6e1dCfu+uA5uq1VqlRJDBhxhgeA2OfNAgAA6Ome0LL5okyNaZg8eTKdOnVKjPrkVJCAgACb53kFr6zA3R/Hjh2zKTt+/DiVLFkyS/YPAABgzUgGsWXEqKGON8pU0NCmTRt6GAYOHEj169cX3ROvvvqqWEv8iy++EBsAAECW09qKYPDNa5+poIHzRB+G2rVr07Jly8Q4hY8++kikW06bNo06duz4UP4+AAD4Fq3jFYwIGvTjpT55Ziv26KOPisGKWe35558XGwAAgLtxLICGhixuaeAJnXh1LZ6MIk+ePKKMZ6TimRt5JS6euTGnabE1gvyDQ23KStR+Rlo3+vBqh7Lef4dJ63ac3Eda/nK9otLyiFa2AztVX/SWDyrN3/pxh7Iz3/6ftK5ilq8gWcIQJy1PvHlRWl6/TKS0fNlKxxU9y0bL5zi/cUt+PiEREdLyf5zUDwiV1791P9WhzBggX7XynpNVLp2tWnk/zaSrfqqO5SJNelei1FFf7+KMOg9F96qY7oSFKMETJ3fy2eyJt99+m+7evUuHDh2iW7duie3gwYNiIiWeRwEAAMBjWxq0ZE+Qb8pUSwPPl71+/XqR/qji1MhZs2aJua8BAAA8ESZ3ckPQwJMu2adZMi6zX4cCAADAU2AgpBu6Jxo1akT9+/enS5cu2SxhzSmSjRs3zswuAQAAsh0md3JD0PDpp5+K8Qs8sVPZsmXFxumQXDZz5szM7BIAACBH3BS1br4oU90TvFw1z/rI4xqOHv13RD2Pb2jSpElWHx8AAMBDw6s2a1m52eCj2ROZChrUC/bss8+KDQAAwBtgTEMWBQ0zZswQq0wGBweLn11B2iUAAHgq32xDyOKgYerUqWL6Zg4a+GdXLRAIGgAAwBOhpcE1zWM5zpw5Q5GRkZafnW2nT5/WuksAAIAcOaZBy6YXz2XECQT85btu3bpiEUZXli5dShUrVhT1q1atSr/99pvN84qi0AcffECFCxemXLlyiXGFJ06cIHfK1ABQXjwqMdFx2t/79++L5wAAADy5pUHLpsd3331HgwYNEgs+ciJBdHQ0NWvWTCzLIPPHH39Qhw4dqFu3bvTnn3+K1aV549mXVRMmTBDDBebMmUM7d+6k0NBQsc+kpCTKUUHDhx9+SPfu3XMo50CCnwMAAPDkGSG1bHpMmTKFevToQV27dhUzKPONPiQkhObPny+tP336dGrevDkNGTJEZCeOGTOGHnvsMTHlgdrKwKs+jxgxglq3bk3VqlWjRYsWifmTli9fTjkqaOCDlTXN/PXXX5QvX76sOC4AAIAcP7nTnTt3bLbk5GSHfaakpIhVoa2nJTAajeJxbGys9Di43H4aA25FUOvzcIArV67Y1ImIiBDdHs72+dBTLvPmzWvpy3nkkUdsAgeTySRaH3r16kU50V8rfiSDn+0qiAnLB0rrml/92KGs/5CvpHWD9lyWlk87tEhartzcIy13FrVWqVXMoezgyJPSuoEVykvLzX+ulZanJsZLy6sXzC0tT7h+waGsWvGK0robdzjWZSERQdLyy3Hy5rTAkHBp+e3EFIcy/8Bc0rr3UuSrVhr95atiJjpZttLZKpdpZkV7XZN8fUaDUR6/O6kur6tzBnc9K2hm5ouFvmMhj+XJxw6ODPylWMP715Beh+ctssbdD6NHj7Ypu3HjhrhHFixY0KacH6tzHdnjgEBWn8vV59UyZ3WyPWjgphD+MHjzzTdFNwRHNarAwEAxwKNevXruOE4AAAD3U8z/blrqEdGFCxcoPPy/LzdBQfIvRt5CV9DQuXNn8S9PGV2/fn3polUAAACeyqCYxaalHuOAwTpokImKiiI/Pz+6evWqTTk/LlSokPR3uNxVffVfLuPsCes61atXp2wf08B9NaoaNWqITAn7vhx1AwAA8OiWBi2bRtwSX7NmTdqwYYOljFeE5sfOWue53Lo+W7dunaU+f3nnwMG6Dt9/OYvCnS3+Rj3jGdTUkDx58ojH9ptanlW4D2jkyJHi4nAOKi+MxSNI9faXAgAAaML3F62bDpxuOXfuXPrqq6/oyJEj1Lt3b0pISBDZFKxTp040fPhwS31eSXr16tU0efJkMe6Bx0ns2bOH+vXrJ57nMYUDBgygjz/+mH755Rc6cOCA2EeRIkVEama2d09s3LjRkhmxadMmehjGjx9Ps2fPFhf50UcfFReMLzCPpcCskwAAkN1jGrRq164dXb9+XUzGxAMVuQuBgwJ1IOP58+dFRoWKhwB88803IqXy/fffp/Lly4tUyipVqljqvPfeeyLw4CUe4uLiqGHDhmKfPBlUtgcNTz31lPRnd+LJLTj/tGXLluIxD7T89ttvXc6ixeku1ikv6C4BAAB92RNaxjQoui8qtxKoLQX2Nm/e7FD2yiuviM3pMRgMYkLFhzmpYqbmaeBIZtu2bTZTY3LU9Nprr9Ht27ez7OA40uL+muPHj1vmgeC/26JFC6e/ExMTI1oi1M0+HQYAAMApc5r2zQdlKmjgGarUb/Dcj8J9Nc8995yYbIJ/zirDhg2j9u3bi7m3OVODB2ByHw4vnOUM9wnFx8dbNk6HAQAAyK6BkD6bcqni4ICnwWQ//vgjtWrVisaOHSvm0+bgIat8//33tHjxYtGvw2Ma9u/fL4IGHuihpn/a4xxZb8+TBQAAN+FgwJz1Yxp8Omjg9BF1war169eLEZuMB0pm5RgCbtFQWxsYr/J17tw50QXhLGgAAAB4WPM0+JpMBQ08QpO7IRo0aCAGJfLqXYzHHhQr5jjtcWZxYGI9mpTxBBmc3woAAOAp2RM+PaaBV9ny9/enH374QaREFi1aVJSvWrVKrMqVVbjb45NPPqFff/2Vzp49S8uWLRMrhbVt2zbL/gYAAIC752nw6ZaGEiVK0MqVKx3Kp06dSllp5syZYnKnPn36iImleCzDW2+9JfJcAQAAshxaGrI+aFBna+SJJnhmK8YDFV944QXRfZBVwsLCxCJZvAEAAHjyPA0+GzScPHlSZElcvHiRKlSoIMp4cCLPicBdCTzdc04zetwgCg4NsymbVFHelZL40wqHslx55ctRRwTIg6QlSWWk5dWnypcOb1wwVFre5InSDmVbb8uXkQ4v9oi0/Mr6rdJyxSx/+YuFyP9nSIq/7lBWpch/K51a+yXuhHzf5SOl5XF3HdegZ0G5bV8z1bU7jtfAP1i+pPfdFHk+tTFAvjR2cpp8KW1ny12n6FiTWs9S1//WV9y21LXejzxP/Yz00MOG7IKWhqwf08BTOHNgwHMgcJolbzwFJq8RgemdAQDAY2Fyp6xvadiyZQvt2LHDshYFi4yMpHHjxomMCgAAAE+ElEs3BA08edLdu3cdyu/duyfmcAAAAPBInNKvJa3fjJRLzZ5//nmxqhav283LVPPGLQ+9evUSgyEBAAA8ElIus35Mw4wZM8SYhnr16oklOHnjxaXKlStH06dPz8wuAQAAsh/Wnsj67ok8efLQzz//LLIoDh8+LMp4LQoOGgAAADwVxjS4aZ6GL7/8UkzmdOLEv6l15cuXF4tJde/ePbO7BAAAyF5Iucz6oIFnZOTpnN9++23RRcFiY2Np4MCBIvXyo48+ysxuAQAAcsCYBi1rTyjkizIVNPB6E3PnzqUOHTpYyngAZLVq1UQggaABAAA8kmIiMpu01fNBmRoImZqaSrVq1XIor1mzJqWlyWffAwAAyOmUtFTNm7vcunWLOnbsSOHh4WIMYbdu3cSUBq7q8xd2nqE5V65cYn0onmgxPj7epp7BYHDYlixZ4v6g4Y033hCtDfa++OILcaIAAAAeiVsZtG5uwvfRQ4cO0bp168TikFu3bhXTHDhz6dIlsU2aNIkOHjxICxcupNWrV4tgw96CBQvo8uXLlq1NmzYPbyDk2rVr6fHHHxePec4GHs/QqVMnGjRokKUej30AAADwBIrJJDYt9dyBF4HkG/7u3bstLfq84jOv98RBAa/2bK9KlSr0448/Wh7zlAiffPIJvf7666L139//v1s9t1wUKlQo08eXqaCBI5nHHntM/Hzq1Cnxb1RUlNj4ORU3fQAAAHjrjJB37txxmDGZt8zipAK+sVsPAWjSpAkZjUbx5bxt27aa9sNdE9y9YR0wsL59+4osxzJlyogJGbt27arrXp2poGHTpk3kaZ5fGUNhQQE2Zb8Hy0+/79j5DmWfTX1bWrdGxO/S8hfm7JSWD/pZvvrjKyPlK25GlHRc/XJlmvwNXahcSWn5uTULpeVG/2rS8oCL/wV+1lITbfvHWIUo+eqc9+/ckpYXjiwuLT955ra0PDjU9jVT3byX4lDmFxQsrXs3WT7Oxs9fPuV5YqpZ1yqXSZLXw1ldZ6tWOq3v5LPLYHTsWdQ7ltus8xf0rKKpd4JdnlVW37Ho/AMAuoIGk+agoXhx28+0UaNG0ejRozN9va9cuUIFChSwKeMbP6/1xM9pcePGDRozZoxDlwYnKTRq1IhCQkJET0GfPn3EWAk9C01munsCAADA2yhmk9i01GO82jN/o1c5a2UYNmwYjR8/njLqmnhQ3PLRsmVLMeGiffAycuRIy881atSghIQEmjhxIoIGAACATFE0dk8o/9bhgME6aHBm8ODB1KVLF5d1uMuAxxtcu3bNppzHJXCGREZjEXghyebNm1NYWBgtW7aMAgLkLbWqunXrihaJ5ORkzV0qmcqeyCo8IrRVq1ZiYAf3qSxfvtyhyZInkipcuLBII+F+HXUGSgAAAHe1NGjZ9MifPz9VrFjR5carRPOEiXFxcbR3717L727cuJHMZrO4ybtqYWjatKnYxy+//CLWhMrI/v37KW/evLrGYGRr0MBNI9HR0TRr1izp8xMmTBCLY82ZM0cMAAkNDaVmzZpRUlLSQz9WAADwAdmcclmpUiXRWtCjRw/atWsXbd++nfr160ft27e3ZE5cvHhRBBn8vHXAwPdUzmzkxzz+gTdTepbHihUraN68eSJZgdeN4mkTxo4dK+Z30CNbxzS0aNFCbDLcyjBt2jQaMWIEtW7dWpQtWrSIChYsKFok+AICAABkZ/aEOyxevFgECo0bNxZZEy+99JL4Am09weKxY8coMTFRPN63b5/4Ys3sF448c+YMlSpVSnRV8Bd0Xu6B769cj6dE4OBEjxw7EJJPlKMk7pJQRUREiOYZTklxFjRw3wxvKvt0GAAAAGeU1FRSUlM01XMXzpT45ptvnD7PQYB1xtHTTz+dYQYSt17w9qCytXvCFTW1hFsWrPFjV2knMTExIrhQN/t0GAAAgJw8I2ROlmODhswaPny4mNRC3TgdBgAAQAvFbNa8+aIc2z2hppZcvXpVZE+o+HH16tWd/t6DzsYFAAA+TGsrghktDTlK6dKlReCwYcMGm/EJPNiDU1IAAADctjR2Rpvim0FDtrY08PSVnPphPfiR80Z5EAgv7TlgwAD6+OOPqXz58iKI4NmsOOVE76pcAAAAWmjtelDQPfHw7dmzh5555hnLY3V1zM6dO4ulPd977z2Rd8rzZ/NkFw0bNhSrf2mZtAIAAMDda0/4mmxtacgoTYRnieQFNngDAABwO4xp8MyBkAAAAA+bYjKJTUs9X+QzQcO0WbEUaJdhOjVRvqLY260+cSh78cTX0roBU+VTYP/TTN46cvDOfxNPWevZdai03LzVcYKPqED5Msq1qtjOaaE69ql82emgmvml5Ul/b5OWm1Icp+8uESHPVEmOvyEtL1Mgt7T8wAH53Bsh4fL935IsjR0QLN93XKJ8Ehaj06WxTbqWr07WszS22X1LRuvft7768OCwpLcHSEslSvPTVs8H+UzQAAAAkNVLY/saBA0AAADpkD3hGoIGAACAdIpZIUVDX5/io31NCBoAAADSccCgKWgwIeUSAADAp6F7wjW0NAAAAKRDS4NrCBoAAADSIWhwDUEDAACA1aRNZkzu5BSCBgAAgHTmtDQyp/ppqueLbKdIBAAA8GFq94SWzV1u3bpFHTt2pPDwcMqTJw9169ZNrAqd0VpOvF6T9darVy+bOufPn6eWLVtSSEgIFShQgIYMGUJpOoMftDQAAABYz9Ngzt55Gjp27EiXL1+mdevWUWpqKnXt2lWs9vzNN47LCljr0aOHzQKPHByoTCaTCBgKFSpEf/zxh9h/p06dKCAggMaOHav52BA0AAAApDObzGLLiNlNLQ1Hjhyh1atX0+7du6lWrVqibObMmfTcc8/RpEmTqEiRIk5/l4MEDgpk1q5dS4cPH6b169dTwYIFqXr16jRmzBgaOnQojR49mgID5Wvx2EP3BAAAQCa7J+7cuWOzJSfLFyXUKjY2VnRJqAEDa9KkCRmNRtq5c6fL3128eDFFRUVRlSpVaPjw4ZSYmGiz36pVq4qAQdWsWTNxzIcOHdJ8fD7T0jDwnfoUFmQbSZV9c5G07qhRXR3KZjZvLq2b+FNbaXl40Uek5VGHt0jLl56XR63Vv13hUFY3Mpe0rqFqYWn5VsmKkCw0f3Fp+fXd2zW/XfIHyhdtSUmIl5aXjQqVliclyFeMiyoSLi1PvO9YP9CqKc5afKL8/P0Cc+la5dIYII/EU83aV7lM0fntxKSjBdRM+ppLzTqXuVR8ZAVNDz50yIaUy+LFbT9HR40aJb65Z9aVK1fEeANr/v7+lC9fPvGcM6+99hqVLFlStET8/fffogXh2LFj9NNPP1n2ax0wMPWxq/36bNAAAACQEUUxaxvToPxb58KFC2LAoiooKEhaf9iwYTR+/PgMuyYyi8c8qLhFoXDhwtS4cWM6deoUlS1blrIKggYAAIBMtjSEh4fbBA3ODB48mLp06eKyTpkyZcSYhGvXrtmUc4YDZ1Q4G68gU7duXfHvyZMnRdDAv7tr1y6bOlevXhX/6tlvto5p2Lp1K7Vq1Uo0p3B6yPLlyy3P8YhRbl7hiCk0NFTU4ZGely5dys5DBgAAL+aulMv8+fNTxYoVXW48GLFevXoUFxdHe/futfzuxo0byWw2WwIBLfbv3y/+5RYHxvs9cOCATUDC2Rkc8FSuXNkzgoaEhASKjo6mWbNmOTzHAzj27dtHI0eOFP9yvwz3z7zwwgvZcqwAAOD9TKlpmjd3qFSpEjVv3lykT3LLwPbt26lfv37Uvn17S+bExYsXRZChthxwFwRnQnCgcfbsWfrll1/El+wnn3ySqlWrJuo0bdpUBAdvvPEG/fXXX7RmzRoaMWIE9e3b12mXSo7rnmjRooXYZCIiIkQUZO3TTz+lOnXqiAkqSpQo8ZCOEgAAfMW/rQgmTfXchbMgOFDgMQmcNfHSSy/RjBkzbFri+Uu0mh3BLRScSjlt2jTxZZwHZ/LvcFCg8vPzo5UrV1Lv3r1FqwO34Hfu3NlmXgevG9MQHx8vujE4HcUZTnexTnnhdBIAAABPWRo7X758LidyKlWqFClWKUocJGzZIs/Ms8bZFb/99tsDHZvHzNOQlJQkxjh06NDB5aCTmJgY0UqhbvbpMAAAAC6DBi1jGszuCxpyMo8IGrgp5tVXXxWR1ezZs13W5QktuEVC3TgdBgAAQBOtgyBNvhk0+HtKwHDu3DkxgjSj1BYe0KFnUAcAAEBOmUY6p/P3hIDhxIkTtGnTJoqMjMzuQwIAAC+WE8Y05GTZGjTwUp888YTqzJkzIreUB4FwbunLL78s0i15xCev0KVOdcnPa11cAwAAwF2TO/mabA0a9uzZQ88884zl8aBBg8S/nAbCc3dzrinj1biscasDrx0OAACQlRSTIjYt9XxRtgYNfOO3Thux5+o5AACArMYzL2oa02BGSwMAAIBPM6UqZCKzpnq+KEcPhMxKy5sNpeDQMJuypBj55Bl9krc6lH3oZLnkhfM3Sstf6NhUWv7stZ+l5QN/Oigtf2/VaYeymq9WkdbNW8z2/FQr0+T/A+Qr+u+UpPau7DsvLTcYyzmU+d88K62blnRPWl48IlhanpIorx8ZUVRafvGy46RdAUHy5ajvJsmnezUGBOhaGtvZctdJkuvrrG6qkyZNZ/VNOlrbnFU1GOWZ1Xo/8sx6lunW2Urom9/ZIMeOaTBiTAP5etAAAACgaUyDEWManEHQAAAAkM5sUsisIWgwYyAkAACAb0P3hGtoaQAAALAaj2PWMIDH7KPZfQgaAAAAVDymwaAhIDAhaAAAAPBpYu0JA9aecAYtDQAAANbZExpaGhS0NAAAAPg2U4qJTGZDxvXS5PO5eDu0NAAAAFgtX6BoGAip+OhASPlUcQAAAL46T4PGzV1u3bpFHTt2pPDwcMqTJw9169ZNrArtzNmzZ8lgMEi3pUuXWurJnl+yZImuY0NLAwAAgPWYBg0TmytuDBo4YLh8+TKtW7eOUlNTqWvXrtSzZ0/65hv50gfFixcX9a198cUXNHHiRGrRooVN+YIFC6h58+aWxxyU6IGgAQAAwCZoyL6BkEeOHKHVq1fT7t27qVatWqJs5syZ9Nxzz9GkSZOoSBHHNYP8/PyoUKFCNmXLli2jV199lXLnzm1TzkGCfV090D0BAACQye6JO3fu2GzJyckPdC1jY2PFjV0NGFiTJk3IaDTSzp07Ne1j7969tH//ftGtYa9v374UFRVFderUofnz5+sem+EzLQ1j3p9KBr9Am7LDq6dK684r99+Lpeo/pqW07uQle6Tln73whrTcHPiStPzU7G3S8n1xSQ5lL3Z4WVrX7+ZRaXmAk4HABUrIm6Wuf3NDWu5furpDmencYWldU8p9aXmxcPkql8l3b0vLC+eR1z99+pZDWVAu+aqV8Ymp0nL/wFzS8vsp8lHRfv627x9VspNVRGVSzfrWc3Q2Hku2KqaeVSgzUz8njfvS80Gn9zxzEk8+dk+lmM2kGAya6qldA9ZGjRpFo0ePpsy6cuUKFShQwKbM39+f8uXLJ57T4ssvv6RKlSpR/fr1bco/+ugjatSoEYWEhNDatWupT58+YqzEO++8o/n4fCZoAAAAyIhoRdDQPWFOb2m4cOGCGLCoCgoKktYfNmwYjR8/PsOuiQd1//59MfZh5MiRDs9Zl9WoUYMSEhLEuAcEDQAAAJnA6ZaaxjSY/63DAYN10ODM4MGDqUuXLi7rlClTRow3uHbtmk15WlqayKjQMhbhhx9+oMTEROrUqVOGdevWrUtjxowRXSrOgp0cNaZh69at1KpVKzGwg1M/li9f7rRur169RJ1p06Y91GMEAADfYebJnTRsZifdmM7kz5+fKlas6HILDAykevXqUVxcnBiXoNq4cSOZzWZxk9fSNfHCCy+Iv5cRHveQN29ezQFDtgcN3DQSHR1Ns2bNclmPR4Hu2LFDOmoUAAAgS7MnNG7uwGMROCWyR48etGvXLtq+fTv169eP2rdvb7kHXrx4UQQZ/Ly1kydPii/j3bt3d9jvihUraN68eXTw4EFRb/bs2TR27Fh6++23dR1fto5p4PxR+xxSe3xx+KTWrFlDLVvKByMCAABk2dLYSvYujb148WIRKDRu3FhkTbz00ks0Y8YMy/M8d8OxY8dEN4Q1zoYoVqwYNW3a1GGfAQEB4gv6wIEDxUDicuXK0ZQpU0RwokeOHgjJzTFvvPEGDRkyhB599FFNv8N9M9YpL5wCAwAAoIVJUcSmpZ67cKaEs4mcWKlSpaQZRNxywJsMt15YT+rklfM08EhTTjXRM7IzJiaGIiIiLJt9OgwAAIAz3OugdfNFOTZo4EEg06dPp4ULF4oBkFoNHz6c4uPjLRunwwAAAOhpadCy+aIcGzT8/vvvIu2kRIkSorWBt3Pnzom0FW6acYZHgaopMFpTYQAAABhaGjx0TAOPZeCpM601a9ZMlPPiHQAAAFnNrLEVweyjLQ3ZGjTw9JWc+qE6c+aMyBvlQSDcwhAZGekw+pMnt6hQoUI2HC0AAHg7nn1By3gFE/mmbA0a9uzZQ88884zl8aBBg8S/nTt3FmMZAAAAHqYUs0J+hoyjhhS0NDx8Tz/9tK6FZ86ePevW4wEAAN8mxjRorOeLcuyYBgAAgIdNZEZoWHvChJYG71a99cvkHxxqU3ay/lPSumcSUhzKjr48Slq34MEV0vJLI96SlucdM09afv/jftLyO5Jll5NqtJLWNS6bIC0vFCyPDcPLR0nLT99zPH8WFJbPoSz51GGXy8bay5fLcUlnlpZ0T1peOI98+erk+47LXYfnk9e9l5QmLfdzsjT2XSf1jc6WxjaZNddNdfL1RLbUtd6lkfV+iLnzM0/vrn308xdyILPGlgazj75n0dIAAACQDi0NriFoAAAASIcxDa4haAAAALAJGrSMaSCfhKABAAAgHVoaXEPQAAAAkA5jGlxD0AAAAJCOex3kuV+2fLR3AkEDAACA9YyQBswI6RRaGgAAANKhe8I1BA0AAADpMBDSNQQNAAAA6dDS4BqCBgAAgHSYRto1BA0AAADp0NLg40GDuvS2KTnR4bkEs3xZkmTFMeEm4e5daV1zyn1p+d1kxwWVmN+dO/LjNMkXiUqRJP/ccbIP4/0kaXmS5HxYcqJ8kagkRX5dlDTH/d+5n6brfO46O/9UJ8eeID9G2euZliQ/z7T78uQoc4rjPlxdF7OTY7x/z/G9oaQlS+smSOr+W19+ve7d1f5+0VOX3dVbX8d7V09dV+/prKjvzn3jWPRdlwe9hoop1eZz3R3uk1nTbI8pmhIzvY9BcefVzwH++ecfKl68eHYfBgAAZJELFy5QsWLFsvR6JiUlUenSpenKlSuaf6dQoUJ05swZCg4OJl/h9UGD2WymS5cuUVhYGN29e1cEEPyGCw8PJ2/FUTvO03vg9fQuvvJ6uuNc+XbFn+NFihQho9FIWY0Dh5QUeWuITGBgoE8FDD7RPcFvLDUiNRgM4l9+83r7/6wM5+ld8Hp6F195PbP6XCMiIshdOADwtSBAr6wP1QAAAMArIWgAAAAATXwqaAgKCqJRo0aJf70ZztO74PX0Lr7yevraufoKrx8ICQAAAFnDp1oaAAAAIPMQNAAAAIAmCBoAAABAEwQNAAAAoInPBA2zZs2iUqVKiYk76tatS7t27SJPt3XrVmrVqpWYHY0nrlq+fLnN8zzG9YMPPqDChQtTrly5qEmTJnTixAnyJDExMVS7dm0xo2eBAgWoTZs2dOzYMYdZ3Pr27UuRkZGUO3dueumll+jq1avkSWbPnk3VqlWzTIJTr149WrVqlVedo8y4cePEe3fAgAFed66jR48W52a9VaxY0evOk128eJFef/11cS78WVO1alXas2ePV30WgQ8FDd999x0NGjRIpP7s27ePoqOjqVmzZnTt2jXyZAkJCeJcOCCSmTBhAs2YMYPmzJlDO3fupNDQUHHe/GHlKbZs2SI+WHfs2EHr1q2j1NRUatq0qTh31cCBA2nFihW0dOlSUZ+nDX/xxRfJk/CspXwD3bt3r/iwbdSoEbVu3ZoOHTrkNedob/fu3fT555+LYMmaN53ro48+SpcvX7Zs27Zt87rzvH37NjVo0IACAgJEoHv48GGaPHky5c2b16s+iyCd4gPq1Kmj9O3b1/LYZDIpRYoUUWJiYhRvwS/lsmXLLI/NZrNSqFAhZeLEiZayuLg4JSgoSPn2228VT3Xt2jVxrlu2bLGcU0BAgLJ06VJLnSNHjog6sbGxiifLmzevMm/ePK88x7t37yrly5dX1q1bpzz11FNK//79Rbk3neuoUaOU6Oho6XPedJ5Dhw5VGjZs6PR5b/0s8lVe39LAi4/wtzduDrNej4Ifx8bGkrfildd4tTbr8+Y527lrxpPPOz4+XvybL18+8S+/ttz6YH2e3ARcokQJjz1Pk8lES5YsEa0p3E3hjefIrUctW7a0OSfmbefKTfDcfVimTBnq2LEjnT9/3uvO85dffqFatWrRK6+8IroQa9SoQXPnzvX6zyJf5fVBw40bN8SHcMGCBW3K+bGeJVA9jXpu3nTevGIp931zU2iVKlVEGZ8LrzSXJ08ejz/PAwcOiL5tnj2vV69etGzZMqpcubJXnSPjgIi7CXm8ij1vOle+KS5cuJBWr14txqzwzfOJJ54QqzR603mePn1anF/58uVpzZo11Lt3b3rnnXfoq6++8trPIl/m9atcgvfgb6cHDx606Rf2JhUqVKD9+/eL1pQffviBOnfuLPq6vQkvkdy/f38xPsXbVxNs0aKF5Wcet8FBRMmSJen7778XgwG9BQfz3NIwduxY8ZhbGvj/Ux6/wO9h8C5e39IQFRVFfn5+DqOS+XGhQoXIW6nn5i3n3a9fP1q5ciVt2rTJstQ543PhLqi4uDiPP0/+5lmuXDmqWbOm+BbOg1ynT5/uVefIzfI8APmxxx4jf39/sXFgxIPk+Gf+9ukt52qPWxUeeeQROnnypFe9ppwRwS1i1ipVqmTpivG2zyJf5/VBA38Q84fwhg0bbCJjfsz9xd6qdOnS4n9I6/O+c+eOGLnsSefNYzw5YOCm+o0bN4rzssavLY/atj5PTsnkDyxPOk8Zfp8mJyd71Tk2btxYdMNwi4q68bdU7u9Xf/aWc7V37949OnXqlLjJetNryt2F9mnQx48fF60q3vRZBOkUH7BkyRIxUnfhwoXK4cOHlZ49eyp58uRRrly5ongyHoH+559/io1fyilTpoifz507J54fN26cOM+ff/5Z+fvvv5XWrVsrpUuXVu7fv694it69eysRERHK5s2blcuXL1u2xMRES51evXopJUqUUDZu3Kjs2bNHqVevntg8ybBhw0RGyJkzZ8RrxY8NBoOydu1arzlHZ6yzJ7zpXAcPHizet/yabt++XWnSpIkSFRUlMoC86Tx37dql+Pv7K5988oly4sQJZfHixUpISIjy9ddfW+p4w2cR/MsnggY2c+ZM8T9oYGCgSMHcsWOH4uk2bdokggX7rXPnzpZUp5EjRyoFCxYUQVPjxo2VY8eOKZ5Edn68LViwwFKHP3j69OkjUhT5w6pt27YisPAkb775plKyZEnx/syfP794rdSAwVvOUWvQ4C3n2q5dO6Vw4cLiNS1atKh4fPLkSa87T7ZixQqlSpUq4nOmYsWKyhdffGHzvDd8FsG/sDQ2AAAAaOL1YxoAAAAgayBoAAAAAE0QNAAAAIAmCBoAAABAEwQNAAAAoAmCBgAAANAEQQMAAABogqABAAAANEHQAJBFnn76abF0d1bj5ZXtl1C2N3r0aKpevbrlcZcuXahNmzaUE65DqVKlaNq0adlyLACQtbA0NoAX4tUx/52FO/vt3r2bQkNDs/swACALIGgA8EIRERGUU+TPnz+7DwEAsgi6J8Dn8JLTMTExYsneXLlyUXR0NP3www+W5zdv3kwGg4HWrFlDNWrUEHUaNWpE165do1WrVlGlSpUoPDycXnvtNUpMTLTZd1pamljKm2/aUVFRNHLkSJtv/LzU9bvvvktFixYV377r1q0r/p59d0SJEiUoJCSE2rZtSzdv3nQ4h3HjxlHBggUpLCyMunXrRklJSTbP23dPcJfBO++8Q++99x7ly5dPLFXMXRrWjh49Sg0bNqTg4GCqXLkyrV+/XlyH5cuXO72WCQkJ1KlTJ8qdO7dY8nny5MkOdey7J3ifn3/+OT3//PPiHPl6xsbG0smTJ8Vx8nWpX7++WEYaAHKY9IWrAHzGxx9/LFbiW716tXLq1CmxYiavvMfLGFuvHvr4448r27ZtU/bt26eUK1dOrMbYtGlT8Xjr1q1KZGSkWPJXxc/nzp1brNh49OhRsTQwr15oveJf9+7dlfr164vf5xUPJ06cKP728ePHxfO8+qrRaFTGjx8vVgGcPn26WFKYlwdXfffdd+J35s2bJ/7O//73PyUsLEyJjo621OGVTnn5YetjCw8PV0aPHi3+1ldffWWz9HZaWppSoUIF5dlnn1X279+v/P7772I1WL4Oy5Ytc7l0Oa8eu379erHk8fPPPy+OxXrVSl69c+rUqZbHvE9e9ZHPg8+xTZs2SqlSpZRGjRqJ14SXr+dr37x58wd6nQEg6yFoAJ+SlJQkbuR//PGHTXm3bt2UDh062AQNfCNUxcTEiDIOMlRvvfWW0qxZM5sbc6VKlcQywKqhQ4eKMnbu3DnFz89PuXjxos3f5mWChw8fLn7mY3juuedsnuclla2Dhnr16oklla3VrVs3w6ChYcOGNr9Tu3ZtcXxs1apVir+/v83SzOvWrXMZNNy9e1cs+/z9999bym7evKnkypUrw6BhxIgRlsexsbGi7Msvv7SUffvtt0pwcLD07wJA9kH3BPgUbgLnLoVnn31WNKmr26JFixyaw6tVq2b5mbsCuCm9TJkyNmXcZWHt8ccfF83vqnr16tGJEyfIZDLRgQMHxL+PPPKIzd/esmWL5W8fOXJEdFlY431Y01JHxvp8GHcnqMd/7NgxKl68uOi2UNWpU8fl/viYU1JSbI6Fuz4qVKig61j4OrKqVavalHGXy507dzLcFwA8PBgICT7l3r174t9ff/1VjCuwFhQUZPM4ICDA8jMHAtaP1TIeH6Hnb/v5+dHevXvFv9Y4eHC3Bz1+dx2LGmTJyrLr+ABADkED+BQe4MfBwfnz5+mpp57K8v3v3LnT5vGOHTuofPnyIkjgQZXc0sDf7p944gnp7/OgQNk+ZHV4AKKzOnpx68CFCxfo6tWrlm/+nCrpStmyZcWNno+FB26y27dv0/Hjx91ybQEg+yFoAJ/C2QacvTBw4EDxLZazBeLj42n79u0iI6Jz584PtH8ORgYNGkRvvfUW7du3j2bOnGnJKOBuiY4dO4qbPZdxEHH9+nXasGGDaK5v2bKlyHBo0KABTZo0iVq3bi0yOFavXm3zN/r37y+yI2rVqiXqLl68mA4dOmTTdaIXd9dwEMDnP2HCBLp79y6NGDFCPGfd3WLfOsKZG0OGDKHIyEgqUKAA/e9//yOjEb2eAN4K/3eDzxkzZoxIheS0S/7W3rx5c9FdwSmYD4oDgvv374vxAH379hU3+J49e1qeX7BggagzePBg8e2e0yL5G736TZ3HRMydO1dMzsSpoGvXrrXcvFXt2rUTx8/pkzVr1qRz585R7969H+i4uSWEUyu5C6V27drUvXt3EQAwTsF0ZuLEiaLVpFWrVtSkSRMRhPExAYB3MvBoyOw+CADIebj1hYMAHjzKrRAAAAgaAEBYtmyZ6HLgMRgcKHArSd68eWnbtm24QgAgYEwDAAg8jmHo0KFiXAbPZsndDbIZHgHAd6GlAQAAADTBQEgAAADQBEEDAAAAaIKgAQAAADRB0AAAAACaIGgAAAAATRA0AAAAgCYIGgAAAEATBA0AAABAWvw/SPMBvCYX5qQAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def positional_encoding(T, C):\n", " pos = np.arange(T)[:, None]\n", " i = np.arange(C)[None, :]\n", " angle = pos / np.power(10000, (2 * (i // 2)) / C)\n", " pe = np.where(i % 2 == 0, np.sin(angle), np.cos(angle))\n", " return pe # (T, C)\n", "\n", "pe = positional_encoding(T, C)\n", "import matplotlib.pyplot as plt\n", "plt.figure(figsize=(6,3)); plt.imshow(pe, aspect=\"auto\", cmap=\"RdBu\")\n", "plt.xlabel(\"embedding dim\"); plt.ylabel(\"position\"); plt.title(\"sinusoidal positional encoding\"); plt.colorbar(); plt.show()" ] }, { "cell_type": "markdown", "id": "09c1c89e", "metadata": {}, "source": [ "
Line by line: what each line does\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "71edb611", "metadata": {}, "source": [ "## LayerNorm\n", "\n", "As signals pass through many layers, the numbers can drift, either ballooning very large or shrinking towards nothing, and training then falls apart. **LayerNorm** is the fix. For each token it takes that token's list of numbers and rescales it to have an average of 0 and a **standard deviation** of 1. (Standard deviation is just a measure of how spread out a set of numbers is; forcing it to 1 keeps the spread consistent, rather like grading on a curve.) It then applies two learned dials, `gamma` and `beta`, so the model can stretch or shift the result if that helps.\n", "\n", "The cell shows it working: numbers that started with an average of 0.44 and a spread of 5.02 come out at an average of 0 and a spread of 1. This is done for each token on its own and has nothing to do with the rest of the batch, which is what separates it from an older method called BatchNorm, and why it behaves the same whether you process one sequence or a thousand." ] }, { "cell_type": "code", "execution_count": 3, "id": "0895632a", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.400332Z", "iopub.status.busy": "2026-07-01T09:08:58.400258Z", "iopub.status.idle": "2026-07-01T09:08:58.402466Z", "shell.execute_reply": "2026-07-01T09:08:58.402079Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "before: mean=0.44 std=5.02\n", "after : mean=0.00 std=1.00 (per-row normalized)\n" ] } ], "source": [ "def layer_norm(x, gamma, beta, eps=1e-5):\n", " mu = x.mean(axis=-1, keepdims=True)\n", " var = x.var(axis=-1, keepdims=True)\n", " return gamma * (x - mu) / np.sqrt(var + eps) + beta\n", "\n", "z = np.random.randn(2, 5) * 5 + 3\n", "zn = layer_norm(z, 1.0, 0.0)\n", "print(\"before: mean=%.2f std=%.2f\" % (z.mean(), z.std()))\n", "print(\"after : mean=%.2f std=%.2f (per-row normalized)\" % (zn.mean(), zn.std()))" ] }, { "cell_type": "markdown", "id": "e37194ab", "metadata": {}, "source": [ "
Line by line: what each line does\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "m6lna", "metadata": {}, "source": [ "### Watch LayerNorm flatten a token\n", "\n", "LayerNorm's job is one sentence: take a token's row of numbers and rescale it to mean 0, standard deviation 1. The cell above printed the overall stats; here it is per row, before and after, with row 0 drawn as bars. However spread out a row starts, it lands on the same scale." ] }, { "cell_type": "code", "execution_count": 4, "id": "m6lnb", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.403522Z", "iopub.status.busy": "2026-07-01T09:08:58.403453Z", "iopub.status.idle": "2026-07-01T09:08:58.449067Z", "shell.execute_reply": "2026-07-01T09:08:58.448690Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "before row 0: mean=-0.06 std=3.23\n", "before row 1: mean=+0.95 std=6.29\n", "after row 0: mean=+0.00 std=1.00\n", "after row 1: mean=+0.00 std=1.00\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAArEAAAD5CAYAAAAwTm0FAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKW9JREFUeJzt3Qm8TfX+//GvIcd8Mg+ZSSWZI3Qzllwpt1K33EJSuRSXBueWpNTRJIUkdVHdLk104yrKlBtluMqpkCInZUodQ3WI9X+8v7//Oo999pn2Zu+zz3ef1/PxWI699rCm7/qsz/qu7/quIp7neQYAAABwSNFYzwAAAAAQLpJYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJLYAmD58uWmSJEi5o033ojo765du9Z06NDBlClTxv7+xo0bTUH2ySefmBIlSphvv/021rOCKJk1a5Ytizt27CgQ6/jHH3+0+8d//vOfWM8KCrnDhw+bm2++2VSvXt3uIyNGjDAFzWuvvWYqVqxo5xUFI29Yvnx5xH7zggsuMHfffbdxCUlsBH300UfmwgsvNKVLl7aB6I477ojZzn7s2DHTt29fc+DAAfPUU0+Zl19+2dStW9cUZPfee6+57rrrCvx8RsvPP/9sbrnlFlOlShWbWHXp0sVs2LAh5BOAv/71r6Z169bmtNNOs8EtPzz77LM2MY20lStXmssvv9zUrl3blCxZ0u5Pl156qfnvf/8b0vffeustc+2115oGDRrY/fGss84yo0aNsus4UKVKlWziMGbMmIgvA+JHfsT2Rx55xO5LQ4YMsfH6hhtusNN94IEHspTbWDh+/LgZO3asuf32203ZsmVNPDuVWHyyXn31VTNp0qSI/+7hw4ftdlP81AmIjg05xex77rnHTJ061ezevds4w0NE/O9///NKlizptWzZ0ps2bZp37733egkJCd6ll16a53eXLVvmaVO8/vrrEdsaX375pf3NGTNmeK6sP83vRx995BVGx48f9zp06OCVKVPGe+CBB7wpU6Z4TZo08cqVK+dt3bo1z++PHTvWO+2007zWrVt7jRs3tusyP5x77rlep06dQv78zJkz7bxt374918+p3F5xxRXe+PHjvRdeeMF7/PHHvebNm3tFixb1Fi1alOd0KlWq5J133nnemDFj7G/dcccdXokSJbyzzz7b++WXXzJ99osvvrDz9MEHH4S8HCg8TiW2h6Ndu3Zex44dM41TuQ9lf8kP8+bN84oUKeJ99913Xjw71Vh8snr16uXVrVs35M/7ecOyZcty/ZzKjj5Xp04dr3Pnzvb/isM5LXv16tVt3HRFXCWxhw8fjtm0e/bs6dWoUcNLS0vLGKeDpwrMe++9l+9J7IoVKyL+m9Fcv0oytJOdOHHCK4zmzp2bZXvt3bvXO/30073rrrsuz+/v3r07IzkbOnSo80lsdo4cOeJVq1bN69GjR56fzS6wz549O8cTu6ZNm3o33HBD2POE/OFqbA9H/fr1bSKTH0ms9qVwXX755d6FF17oxbtTjcUFLYn97bffvB9++MH+f+3atbkmsTJs2DA7H64ci51NYlXzpI3x+eef24KlAtaiRQv73rFjx7wHH3zQa9Cgga190QZJSkqyG9P3t7/9zatYsWKmDaWNp998+umnMyUHGvfss8/mOC8KbsWLF/fuuuuuTOPT09O9smXLeoMGDQqpMM6ZM8fOpw7UpUuX9nr37u3t3Lkzy+fXrFljD+Tly5f3SpUq5V100UXeqlWrMt7v37+//b3AITDRUI2TgpGmkZiYaIOTaqNCXb/y8ssve61atbI1FBUqVPCuvfbabOc1VEpgBwwYkGW8tp12bq0j1TJqeko4/B33zTfftK9VM6L52bBhQ7a10ldddZWdT31Ov/P2229n+syPP/7ojRo1yv6WzsB11q2alo0bN2a7rRToVEt4xhln2N/s2rWr99VXX5308vft29dud50JB7rlllvsdgosu3mJVBKrwKdtomXUfqQzdJUV/4CqbZNbOUtJSfG6dOlit5l+46GHHvJefPHFUzooa/uoxupkHDx40E575MiRWd5TPFAZdyVwx7N4iu36nGq1FJsUr7UvK/YuXbo0S0wJHrKL48H7TihxWPukTjbXrVvn/eEPf7DHjOHDh3vh+PXXX+36Vs1kMM2TYs5rr73mnXPOOXZeLrjgAu+zzz6z7z/33HNew4YNbZzUvGS37+d1TJMdO3Z4Q4YMsVeaNA1t46uvvjrL7/knyvq+ykLlypXteu/Tp49NRvMzFgfGHq1zlVetxypVqnjdu3f31q9fb9/XegnezoEJbWpqqr0ypenruyNGjPDefffdkJLYQKEksTo26jPZHUsLIueTWFXza+MqEE2dOtW+5+/8KuAad+ONN9rXKsS+t956y47btGlTxjj/cqW+59PZmD6nA3JOtLP4iU0wBSwFmdz4QUyXP5s1a+ZNnDjRGz16tN1RtcMGXv5UAqqdoH379t6TTz7pPfXUU/Y7Gvfxxx/bz+iS/N///nf7m6rhVKBbvHixfW/JkiU2KOt3H3vsMW/cuHF2J1cADAwGua1fJW+6rKSAqfH+b9SrV8/76aefvHDp8pSm9cwzz2R5TzvyWWedZWtCFEC1vEqIdAB55ZVXbPI7YcIEOyghb9SoUabgo+2m8VqORx991F4aUoDU/KsMBO7cCrRa79OnT7cHSk1H3921a1eWbaVLi0qGNT+aLwWXtm3beidL860an2C6lK7p+QeE/ExidUlNy3/ffffZ+XjkkUdsUqpafv/yYq1atewlepWxwHKmBFjBVuVK60c1SmeeeaYtq+EksUoi9u3bZ09ElKzouyrbJ0OXAvV9LUcwlaXgeIDYiKfYrrKr2KUTJzVFUMxVPFPTHzVT8JNp7TuKoUrW/X1JJ9BK4jV9xRl/vF8rHWocVoKkE1Dtj7fffruNb/Pnzw9xa2ReD//+97+zvKfx2q9r166dKRYrNvuX4nWsUhzRcUoxJFAoxzR/e2k73n///d7zzz9v44Dii44RgTXLfhKrGK3KhcmTJ9sKimLFinnXXHNNvsZi3/XXX2+XR+VAv6NjkSqpFHdEcVPbXtvP386Kr6Ljv5+433333d6kSZPsscePpcsinMT6x2OtNxc4n8QGV+9rx9f4m2++OdP4O++80473z4B1RhZ4Fv7zzz/bIOefhfmUBAaf1Qfzg+HKlSuzvKffUwDJjZ8YKWnSGZtPZ7aBtQeaByUCOmMNnB8Vcl2Kuvjii7P8ZnBzAu0oVatWtTWPvk8//dQuuw4Iea1fnQ0rGDz88MOZxuuAoeQ4eHwo3n//fTutd955J8t7fm1fYFtZXcLTOJ2xf/vttxnjFZyDd+pu3brZk4PAs2etOyVoWpc+vR985q1ES7UHSmiD16tqHFTL4tM2OpUkSLW/N910U5bxCxcutL+rs+78TGJ1ENRvKPk8meYEqinQ9wMPQtrndHALJ4lVWfdrJnQQuPXWW22t0MlQrZnKbnbt2lS+ckpWkL/iKbb//vvvmeKEv29pPoL3d/+qUyjNCcKJw34tn2pET5afwGUX3zRecTJwHv1YrPUTeEzzT0T9z4ZzTAtuyy6rV6+2v/fSSy9lSWJV0xn4m6qV1TpTecivWOxT3FNcPpnmBEpaNV3lAz4l7Uq2TRSSWFGsVa23C5zvneC2227L9NrvKmfkyJGZxuvOZFm4cKH9q7sOzz77bHsXtOiu52LFipm77rrL7Nmzx3z11Vd2/IcffmjvSs3tbu9ff/3V/k1ISMjynu6s9t/Py4033mjKlSuX8frqq682NWrUyFgmdZGl+br++utt10D79++3w5EjR0y3bt3sspw4cSLH3//hhx/sbwwYMMDepehr1qyZufjii7PtZih4/equb03jmmuuyZi+Bt2xe+aZZ5ply5aZcGlZpEKFCtm+36RJE9O+ffuM1+3atbN/u3btaurUqZNl/DfffGP/qmeGpUuX2nk9dOhQxrxqej169LDrcteuXRnbrmjRohl34eozugNXd7Vnd1fqwIEDbXdgvj/84Q+Zph0ulZGcyo//fn4qVaqUXT513/LTTz+F/X2VJXXX0rZt24xx2uf69esX1u9MmDDBLF682Lz44ov2944ePWp+//33k7rzV7+hOKByGswveyofKBjiIbZrun6cUNxUTFL5bdOmzSnd7R5uHNb8K2adrLxitI4/9erVyxKLr7rqqkzHtOAYHc4xTTEpsPcdfb5Ro0bm9NNPz3ZdqneBwG2rGK3YnlcXjtGIxZrHjz/+2Hz//fdhf1flXnmA8gGfesnQ8kWLtrMrsbC4cVz9+vUzvVYBVTKiwh1IO7cKUmABVqH2A6MCmgKLBiV4el2tWjXz6aef2h0sN/7OlZ6enuW93377LdPOl5vgg6t2QC2H36emH3z79++f42+kpaXlGGj8ZVdiFuycc84x7733ng0e6lIkp/WredDJd3aJgKh7p5P1fyf1WQUmqpKYmGj/qvul7Mb7Sde2bdvsb6r7pJy6UNq7d68544wzbKB8+umnbZdR27dvt8EusBumvObJX+e5JXxKwHQQC6QDrg50KiM5lR8JtQxFioL4o48+ahME7QdKIC+77DJ7oqV9KS8qa/4BK1B2ZS83LVq0yPj/X/7yF9OqVSt7EhZOn8ralwcNGmRPXB5++OFcy15+dU2GwhPbZ8+ebZ588kmzefNmm3zltHzhCDcOK8YFnnQXlBgdzjFNyWNycrKZOXOmrXwInBd9JhIxWqIRix977DG7jFof6gbxj3/8o42l6gIwLyrXKvPBsSncWBoOrVtXYqHzSWxOBSqUDaCz8BkzZtizQgU2BT59T+P1umbNmja58WvZcqKzJL+mM5jG6XciwT8jffzxxzMd3ANFuv++4PWredA6WrRokU2+IjF9P0nMKbhkN53cxvvBzV9fd955p01gsuMfENVHoxLdm266yTz00EP2YKcDpjocz652O69pZ0d9Pqq/wUBKmFWDoTKUU/mRSJWhcGjZe/fubebPn29PcLR+dBBR7XbLli3zfX50EFbfsaqd1QEtlIOJEhV9p2nTpjbxLV48+5Dnl73KlStHfL5ReGP7K6+8Yk+6+vTpY2uCq1atamOH9qOvv/7anKxw4/CpngQHxuhatWpFPEaHckxT/7RKYBWXdGVOCbHWwZ///OeIxWiJRixWjbnK2rx58+yVJS2vKglUo96zZ09TEPvJrexILHQ+iQ2mjvJVoHWGp9pFny4jacMEdqTvB7AlS5bYp1uNHj3avr7ooovMtGnTbGFVraTOnHKjA6QOjuvWrbOFNbDmTZdLAsflxj8rDdzZVJuoy/3SsGFD+7d8+fKme/fuJlz+sm/ZsiXLe6olUKENrIXNjuZB86VahMaNG5tI0KU/P6GLJP8sV7USea0vJThKMHXJOVo7c/PmzW1ZC+TXaiqA6+Cqsus3axBdgtKlo0it63Bpe6s2VoPKp+ZTtUo6OOeWUKisBZfnnMpeOJS8qvypeUheB2YlCergW4mDauVyO8Hyy15gzEDB4mJsV1xRHFKyErivqPP5UOS0f0UjDocao88777yI/W44xzStS9VmKv4E1o5G+kEQ0YrFSo71QBoNugKoq0q6MuQnsbnF0pSUlCy1o6caS3OiWm6Vb1diofNtYoOpml6Cn3wxceJE+7dXr14Z4xQAdJlFT7TSZZ6OHTtmBEAdALXT6DJqTrU3Pp0RagfUgV0HV5+euqKnZejJWaF46aWXMn1f09fZn1/IFXC10z/xxBPZPi1m3759ee5E2kF1eStwx9cOorNDf93l5sorr7RnuOPGjctyRqvXftupcGgb6DKLDhSRpOSlc+fOZvr06dmeWQeuLy1T8PK8/vrrGW1mI0GXs1ROAge/nZXaO+lgrIOdT22SNA+qDQ1so6WyeSq1OKH45ZdfMi6f+VT21L4t8FKbEoHsDiIqS2vWrLFPEgtc3//85z9Dmr6CfDBN580337RlRdvWt3PnTnsSFkhPnLnkkkvsQUi1yGq2kZv169fb/fjcc88Naf6Q/1yM7X5tYGBsUTK0evXqkJbZr1QI3seiEYdzo2OProREOkaHc0zLLkZPnjw5U9OvSAgnFodC8xfc3EHxSydSwbE0u2YRKvdqSxvYhErx+fnnnzfRoFgoemS9C+KuJla1XTpb0wbWjt+pUyd7IFXipks6wZdzFdTmzJljzy79NjM6Q1KB2rp1a55tpnw6o9JG1/TU4Pq7776zZ4w6kKo2KBS6hK3LXWqAr51IwVqXuwcPHmzf1wH5hRdesEmtDrb6nAK1Ei015NfZ7DvvvJPrNHQZQ9/X5Ri1E1TNlgKBgrUeb5gXBZzx48ebpKQk21ZX61SJjc7QdalEy67L9+G64oor7Pcj3RZHj9DTOtX21XpUrYjWrQ4i2ka63Cxq7/nggw/adartuGnTJptwhdJmKVKBUwdVTf+LL76wtb9qn6sAqANVIN3wIH5bab/dlA6s4h9otJ38M3k9wtKnxH7FihW5XlZT2dd0VNOkG+t0sNf20brT5bvAg5BqtjQtlVUFZ91wp+dva35U9ocPH273J+2TmpfPPvssz/WhMqrLlmpXq99UoqpLiQrmc+fOzfRZtS0LXh5NV5eSNR+rVq2yg0/tIXUjYyDV2OkA5Uo7sMLIxdiuuKJk6E9/+pNNshUnn3vuObtPhfLYWr+mWI/k1n6nq0oqp9GKwznRybaW9/3337dxMlLCOaZpXSqm6Fil9acYrvnJ7p6F/IrFaiqi8uc3C8uOTn4Uy/S7KsO6IqT51hWCwFplbWvFNt24eP7559vPaVvruDVlyhQb55RgqjJK60G1wqHS97XP+DeWaZ2qHPvNNPy2yn4sVHviWDQZOyme492wqB++YOoQW33mqYsO9cen/uuCO8T2qa9B/U5wdxLqniPcR1F++OGHtusm9eemPvnUpUZg9yI58btt+te//mXnU11gqfsodbkR2IWUT/0LXnnllfbRmuraRN1yqP+7wHnN7Slg6tJKjzfUNNS5tPqry+lhB9mtX/8hA+onUd2RaFBfoVreLVu2eCdDHStrelqHeXU7E9jBdnaP1wvuFurrr7+23YepuxeVB3Vldtlll3lvvPFGxmdUNtSXoPp01HrR+lH3LeqeJrALqZzWqz/tvLouyc2BAwdsN1Darup3VtNVlyjBtE6Cu2LJqcP04AcQiPoYzKtroP3799v1q+2q7asuYvSQgcBuXvw+LrV99HCI4GmpP0W9PpmHHah/SZUv9ZuoLoO0P6mcZtfVkd+FUKCc1kV268N/RLP2C8RePMV2dfGkfom1vypWq+/SBQsW2P5ug/fhnGKd9hvtP+omLHjfCSUO+w87OFXqf1f90gY/TCGcWJxT/AzlmKauyQYOHGhjgvoJV7dcmzdvtp/V+gzuYis4dob6hKtwYrEeoqPjRW79o6uLNT0sQ33cKk5qO+n/wQ/ZUP+/6k9WD/cIftiB8gA9aEbzouXXgxPCedhB3WweTOMPgeVJ3UzqGKg+fV1RRP/EOpEGRDV/usTi1ygi8lQroBp/1fIPHTqUVfz/b2BTVz6q5aAmFsieaiJVA6qrM7r5Ff93VUc1pLrCGQ/mz59vr1CoyY1/U2NBRxKLAkNtxXQJUDduBN6kgchRX5pKXnU5NRJd7rhObQdV1l577bWQ2oQDhZkudw8ZMsQ274l0Tziu+fzzz22zPDVbcuVO/rxoeXQMVpdgriCJBQAAgHPirncCAAAAxD+SWAAAADiHJBYAAADOKdD9xOqJGerXTP3fcdcwgFjxnxKm3jMCn+ITb4i5AFyKuQU6iVUCq6fzAEBBkJqamu2z4+MFMReASzG3QCexqoH1F0JP7gCAWDh48KA9ofZjUrwi5gJwKeYW6CTWb0KgBJYkFkCsxXuzJmIuAJdibvw27gIAAEDcIokFAACAc0hiAQAA4Jx8S2InTJhg2zaMGDEivyYJAACAOJUvSezatWvN9OnTTbNmzfJjcgAAAIhzUU9iDx8+bPr162dmzJhhKlSokOtn09PTbbcKgQMAAACQ711sDR061PTq1ct0797djB8/PtfPJicnm3HjxkV7lgBE2NaHmjq1ThuPSYn1LADASWs6catTay9lZGP3amLnzJljNmzYYJPTUCQlJZm0tLSMQQ85AAAAAPKtJlYJ6PDhw82SJUtMyZIlQ/pOQkKCHQAAAICYJLHr1683e/fuNa1atcoYd/z4cbNy5UozZcoU2/61WLFi0Zo8AAAA4ljUkthu3bqZTZs2ZRo3cOBAc/bZZ5t77rmHBBYAAAAFL4ktV66cado0880eZcqUMZUqVcoyHgAAAAgHT+wCAACAc6LexVag5cuX5+fkAAAAEKeoiQUAAIBzSGIBAADgHJJYAAAAOIckFgDihPrh7t27t6lZs6YpUqSImT9/fqxnCQCihiQWAOLEkSNHTPPmzc3UqVNjPSsAEF+9EwAAoqdnz552AIDCgCQWAAopPf5bg+/gwYMxnR8ACAfNCQCgkEpOTjaJiYkZQ+3atWM9SwAQMpJYACikkpKSTFpaWsaQmpoa61kCgJDRnAAACqmEhAQ7AICLqIkFAACAc6iJBYA4cfjwYbNt27aM19u3bzcbN240FStWNHXq1InpvAFApJHEAkCcWLdunenSpUvG65EjR9q//fv3N7NmzYrhnAFA5JHEAkCc6Ny5s/E8L9azAQD5gjaxAAAAcA5JLAAAAJxDEgsAAADnkMQCAADAOSSxAAAAcA5JLAAAAJxDEgsAAADnkMQCAADAOSSxAAAAcE5Uk9jk5GRz/vnnm3LlypmqVauaPn36mC1btkRzkgAAACgEoprErlixwgwdOtSsWbPGLFmyxBw7dsxccskl5siRI9GcLAAAAOJc8Wj++Lvvvpvp9axZs2yN7Pr1681FF12U5fPp6el28B08eDCaswcAAABH5Wub2LS0NPu3YsWKOTY/SExMzBhq166dn7MHAAAAR+RbEnvixAkzYsQI07FjR9O0adNsP5OUlGQTXX9ITU3Nr9kDAACAQ6LanCCQ2sampKSYVatW5fiZhIQEOwAAAAAxT2KHDRtmFixYYFauXGlq1aqVH5MEAABAHItqEut5nrn99tvNvHnzzPLly039+vWjOTkAAAAUEsWj3YTg1VdfNW+//bbtK3b37t12vG7aKlWqVDQnDQAAgDgW1Ru7pk2bZm/Q6ty5s6lRo0bGMHfu3GhOFgAAAHEu6s0JAAAAAKf7iQUAAAAigSQWAAAAziGJBQAAgHPy7WEHQCi2PpT909wKqsZjUmI9CwBw0ppO3OrU2ksZ2TjWs4AChJpYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAOKd4rGcA4dn6UFPnVlnjMSmxngUAOGlNJ251au2ljGwc61kA8gU1sQAAAHAOSSwAAACcQxILAAAA55DEAgAAwDkksQAAAHBOviSxU6dONfXq1TMlS5Y07dq1M5988kl+TBYAAABxKupJ7Ny5c83IkSPN2LFjzYYNG0zz5s1Njx49zN69e6M9aQAolKg4AFAYRD2JnThxohk8eLAZOHCgadKkiXnuuedM6dKlzT/+8Y8sn01PTzcHDx7MNAAAQkfFAYDCIqoPOzh69KhZv369SUpKyhhXtGhR0717d7N69eosn09OTjbjxo3LMv7aa681p512WsjTPbx1uXFJ2cadw/h0A+Ocyy+P3+ULY9kol66WSWOOHTtmXBFYcSCqOFi4cKGtOBg9enSsZw8A3Ehi9+/fb44fP26qVauWabxeb968Ocvnleyq6YFPNbG1a9e2NQvly5eP26daNR7z71jPAvIB5dJdikWJiYmmoAu34kBXvzT4uPoFwCUF6rGzCQkJdgAARL/iIFJXv5Z/fdi5zdW5Ydl4vT5kLg/jYmQ8L5tr5ZIyGf7Vr6gmsZUrVzbFihUze/bsyTRer6tXrx7NSQMA8hCpq19NJ251bl3/e2TjWM8Cosy1ckmZDP/qV1Rv7CpRooRp3bq1+eCDDzLGnThxwr5u3759NCcNAIVOuBUHuvKlZDVwAABXRL13Ap3lz5gxw8yePdt8+eWXZsiQIebIkSMZNx0AACKDigMAhUnU28SqbdW+ffvM/fffb3bv3m1atGhh3n333SxttgAAkak46N+/v2nTpo1p27atmTRpEhUHAOJSvtzYNWzYMDsAAKKLigMAhUWB6p0AAHDqqDgAUBjEZRLbeExKrGcBAAqNFO70BxCPN3YBAAAAkUYSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnEMSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnEMSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnEMSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnEMSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnBO1JHbHjh1m0KBBpn79+qZUqVKmYcOGZuzYsebo0aPRmiQAAAAKieLR+uHNmzebEydOmOnTp5tGjRqZlJQUM3jwYHPkyBHzxBNPRGuyAAAAKASilsReeumldvA1aNDAbNmyxUybNo0kFgAAAAUzic1OWlqaqVixYo7vp6en28F38ODBfJozAAAAuCTfbuzatm2bmTx5srn11ltz/ExycrJJTEzMGGrXrp1fswcAAIB4TmJHjx5tihQpkuug9rCBdu3aZZsW9O3b17aLzUlSUpKtrfWH1NTUk1sqAAAAxLWwmxOMGjXKDBgwINfPqP2r7/vvvzddunQxHTp0MM8//3yu30tISLADAAAAENEktkqVKnYIhWpglcC2bt3azJw50xQtSre0AAAAKMA3dimB7dy5s6lbt67tjWDfvn0Z71WvXj1akwUAAEAhELUkdsmSJfZmLg21atXK9J7nedGaLAAAAAqBqF3fV7tZJavZDQAAAMCpoJEqAAAAnEMSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnEMSCwAAAOeQxAIAAMA5JLEAAABwDkksAAAAnEMSCwAAAOeQxAIAAMA5JLEAECcefvhh06FDB1O6dGlz+umnx3p2ACCqSGIBIE4cPXrU9O3b1wwZMiTWswIAUVc8+pMAAOSHcePG2b+zZs1ihQOIeySxAFBIpaen28F38ODBmM4PAISD5gQAUEglJyebxMTEjKF27dqxniUACBlJLAAUYKNHjzZFihTJddi8efNJ/XZSUpJJS0vLGFJTUyM+/wAQLTQnAIACbNSoUWbAgAG5fqZBgwYn9dsJCQl2AAAXkcQCQAFWpUoVOwAAMiOJBYA4sXPnTnPgwAH79/jx42bjxo12fKNGjUzZsmVjPXsAEFEksQAQJ+6//34ze/bsjNctW7a0f5ctW2Y6d+4cwzkDgMjjxi4AiBPqH9bzvCwDCSyAeEQSCwAAAOfkSxKrzrRbtGhhu4Lx22gBAAAABTqJvfvuu03NmjXzY1IAAAAoBKKexC5atMgsXrzYPPHEE9GeFAAAAAqJqPZOsGfPHjN48GAzf/58U7p06Tw/z3O8AQAAENOaWN0Rq6fM3HbbbaZNmzYhfYfneAMAACAqSWyoz/GePHmyOXTokH02d6h4jjcAAACi0pwg1Od4L1261KxevTrLc7lVK9uvX79MHXL7eI43AAAAopLEhvoc72eeecaMHz8+4/X3339vevToYebOnWvatWsX7mQBAACA6N/YVadOnUyv/ed2N2zY0NSqVStakwUAAEAhwBO7AAAA4JyodrEVqF69erbHAgAAAOBUURMLAAAA55DEAgAAwDkksQAAAHAOSSwAAACcQxILAAAA55DEAgAAwDkksQAAAHAOSSwAAACcQxILAAAA55DEAgAAwDkksQAAAHAOSSwAAACcQxILAAAA55DEAgAAwDkksQAAAHAOSSwAAACcQxILAAAA55DEAgAAwDkksQAAAHAOSSwAAACcQxILAAAA55DEAgAAwDkksQAAAHBO8Wj++MKFC82DDz5oPvvsM1OyZEnTqVMnM3/+/GhOEiiwGo9JifUsAEChkTKycaxnAa4msW+++aYZPHiweeSRR0zXrl3N77//blJSOIgDAACggCaxSliHDx9uHn/8cTNo0KCM8U2aNInG5AAAAFDIRKVN7IYNG8yuXbtM0aJFTcuWLU2NGjVMz54986yJTU9PNwcPHsw0AAAAAPmSxH7zzTf27wMPPGDuu+8+s2DBAlOhQgXTuXNnc+DAgRy/l5ycbBITEzOG2rVrR2P2AAAAUJiS2NGjR5siRYrkOmzevNmcOHHCfv7ee+81V111lWndurWZOXOmff/111/P8feTkpJMWlpaxpCamnrqSwgAAIDC3SZ21KhRZsCAAbl+pkGDBuaHH37I0gY2ISHBvrdz584cv6vPaAAAAAAilsRWqVLFDnlRzauS0S1btpgLL7zQjjt27JjZsWOHqVu3bsjT8zzP/qVtLIBY8mOQH5PiFTEXgEsxNyq9E5QvX97cdtttZuzYsbZdqxJX9VQgffv2Dfl3Dh06ZP/SNhZAQaCYpPb68YqYC8ClmBu1fmKVtBYvXtzccMMN5tdffzXt2rUzS5cutTd4hapmzZq2XWy5cuVse9pYnxUomdb8KEmPN/G8fCybmwrSdlNtgIKpYlI8K0gxt6CVgUhj2dzEditYMbeIF+/XxyJYcHU2oBvO4i2YxvvysWxuiufthtDEcxlg2dzEdisEXWwBAAAA0UQSCwAAAOeQxIZIvS3oRrV47QIsnpePZXNTPG83hCaeywDL5ia2W8FCm1gAAAA4h5pYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAOIckNkRTp0419erVMyVLlrSP0P3kk09MPFi5cqXp3bu3fbSbHjM5f/58Ew+Sk5PN+eefbx+fWbVqVdOnTx+zZcsWEw+mTZtmmjVrZp9gpKF9+/Zm0aJFJh5NmDDBlssRI0bEelaQz4i5biHmxocJjsVcktgQzJ0714wcOdL2V7hhwwbTvHlz06NHD7N3717juiNHjtjl0QEjnqxYscIMHTrUrFmzxixZssQcO3bMXHLJJXZ5XVerVi0baNavX2/WrVtnunbtaq644grz+eefm3iydu1aM336dJuwo3Ah5rqHmOu+tS7GXA95atu2rTd06NCM18ePH/dq1qzpJScnx9XaU3GYN2+eF4/27t1rl2/FihVePKpQoYL3wgsvePHi0KFD3plnnuktWbLE69Spkzd8+PBYzxLyETHXfcRctxxyNOZSE5uHo0eP2hqv7t27Z4wrWrSofb169epon2MgQtLS0uzfihUrxtU6PX78uJkzZ46tYVazgnihWvRevXpl2u9QOBBz4wMx1y1DHY25xWM9AwXd/v37baJQrVq1TOP1evPmzTGbL4TuxIkTtn1Px44dTdOmTeNi1W3atMkmrb/99pspW7asmTdvnmnSpImJB0rK1WxHl7ZQ+BBz3UfMdcsch2MuSSzins4wU1JSzKpVq0y8OOuss8zGjRttbccbb7xh+vfvb9ukuZ7IpqammuHDh9t2zLqJEoB7iLnuSHU85pLE5qFy5cqmWLFiZs+ePZnG63X16tWjuW0QAcOGDTMLFiywvTDohqh4UaJECdOoUSP7/9atW9sz6Keffto2yneZmu7ohslWrVpljNOVEG2/KVOmmPT0dLs/In4Rc91GzHXLesdjLm1iQ0gWlCR88MEHmS6V6HU8tUGMN7pPTcFUl9mXLl1q6tevb+KZyqSCjeu6detmm0qoltkf2rRpY/r162f/X5CDKSKDmOsmYq6bujkec6mJDYG619LlWm3Ytm3bmkmTJtkbaQYOHGhcd/jwYbNt27aM19u3b7cFVzdA1alTx7h8OevVV181b7/9tu0rdvfu3XZ8YmKiKVWqlHFZUlKS6dmzp90+hw4dssu5fPly89577xnXaVsFt1suU6aMqVSpUty0Z0beiLnuIea6qZzrMTfW3SO4YvLkyV6dOnW8EiVK2O5f1qxZ48WDZcuW2a6ngof+/ft7LstumTTMnDnTc91NN93k1a1b15bFKlWqeN26dfMWL17sxSuXuntB5BBz3ULMjR+dHIq5RfRPrBNpAAAAIBy0iQUAAIBzSGIBAADgHJJYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAOIckFgAAAM4hiQUAAIBzSGIBAADgHJJYAAAAGNf8P+htBxWAczhnAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# LayerNorm rescales EACH row to mean 0, std 1 -- per row, then row 0 as bars.\n", "for r in range(z.shape[0]):\n", " print(f\"before row {r}: mean={z[r].mean():+.2f} std={z[r].std():.2f}\")\n", "for r in range(zn.shape[0]):\n", " print(f\"after row {r}: mean={zn[r].mean():+.2f} std={zn[r].std():.2f}\")\n", "\n", "fig, ax = plt.subplots(1, 2, figsize=(7, 2.6))\n", "idx = np.arange(z.shape[1])\n", "ax[0].bar(idx, z[0], color=\"#d9822b\"); ax[0].axhline(0, color=\"k\", lw=.5)\n", "ax[0].set_title(\"row 0 before (mean %.1f, std %.1f)\" % (z[0].mean(), z[0].std()))\n", "ax[1].bar(idx, zn[0], color=\"#2b8dd9\"); ax[1].axhline(0, color=\"k\", lw=.5)\n", "ax[1].set_title(\"row 0 after (mean 0, std 1)\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "m6lnc", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • The two loops print each row's mean and standard deviation, first for the raw z, then for the normalized zn.
    • \n", "
  • Row 1 starts wildly spread (std over 6); after LayerNorm every row reads mean 0.00, std 1.00.
    • \n", "
  • The left bars are row 0 raw -- off-centre and uneven; the right bars are the same row after -- centred on 0 with a consistent spread.
    • \n", "
  • gamma and beta (here 1 and 0) then let the model rescale and shift if it wants, but the starting point is always this clean, fixed scale.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "0150e5e9", "metadata": {}, "source": [ "## Multi-head attention\n", "\n", "A single attention head can only look for one kind of relationship at a time. **Multi-head attention** runs several heads in parallel so the model can look for several at once: perhaps one head links each verb to its subject, another tracks quotation marks, and another watches for the previous line break. It is like a panel of specialists, each reading the sentence for their own purpose and then pooling their notes.\n", "\n", "Mechanically, we slice the embedding into `n_head` equal pieces, run a separate attention head (exactly the one from notebook 05) on each piece in parallel, glue the results back together (concatenate them), and pass them through one more projection `Wo` that lets the heads mix. The output has the same shape as the input, so it drops straight into the block." ] }, { "cell_type": "code", "execution_count": 5, "id": "27e9fe80", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.450130Z", "iopub.status.busy": "2026-07-01T09:08:58.450074Z", "iopub.status.idle": "2026-07-01T09:08:58.452795Z", "shell.execute_reply": "2026-07-01T09:08:58.452497Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "multi-head output: (4, 16, 64)\n" ] } ], "source": [ "def multi_head_attention(x, Wq, Wk, Wv, Wo, n_head):\n", " B, T, C = x.shape\n", " hs = C // n_head\n", " def proj(W): return (x @ W).reshape(B, T, n_head, hs).transpose(0, 2, 1, 3) # (B,nh,T,hs)\n", " q, k, v = proj(Wq), proj(Wk), proj(Wv)\n", " scores = (q @ k.transpose(0, 1, 3, 2)) / hs**0.5 # (B,nh,T,T)\n", " scores = np.where(np.tril(np.ones((T, T))) == 0, -np.inf, scores)\n", " attn = softmax(scores, axis=-1)\n", " out = attn @ v # (B,nh,T,hs)\n", " out = out.transpose(0, 2, 1, 3).reshape(B, T, C) # concat heads\n", " return out @ Wo # output projection\n", "\n", "Wq = np.random.randn(C, C)*0.02; Wk = np.random.randn(C, C)*0.02\n", "Wv = np.random.randn(C, C)*0.02; Wo = np.random.randn(C, C)*0.02\n", "x = np.random.randn(B, T, C)\n", "print(\"multi-head output:\", multi_head_attention(x, Wq, Wk, Wv, Wo, n_head).shape)" ] }, { "cell_type": "markdown", "id": "2223c3bd", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • def multi_head_attention(x, Wq, Wk, Wv, Wo, n_head):: run several attention heads at once and combine them.
    • \n", "
  • hs = C // n_head: whole-number division: 64 features split across 4 heads = 16 each.
    • \n", "
  • def proj(W): return (x @ W).reshape(B, T, n_head, hs).transpose(0, 2, 1, 3): one helper used three times: project, then reshape carves the 64-wide result into 4 heads of 16, and transpose reorders the axes to (batch, head, time, hs) so every head can work independently.
    • \n", "
  • q, k, v = proj(Wq), proj(Wk), proj(Wv): queries, keys, values, all carved into heads.
    • \n", "
  • scores = (q @ k.transpose(0, 1, 3, 2)) / hs**0.5: notebook 05's scaled query-key scores, now with a head axis riding along: 4 separate (T,T) grids per sequence.
    • \n", "
  • scores = np.where(np.tril(...) == 0, -np.inf, scores): the causal mask, applied to every head at once.
    • \n", "
  • out = attn @ v then .transpose(...).reshape(B, T, C): blend values per head, then undo the carving so the 4 heads' 16-number outputs sit side by side as 64 again (\"concatenate\").
    • \n", "
  • return out @ Wo: one final projection lets the heads' findings mix with each other.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "m6mhaa", "metadata": {}, "source": [ "### How `C` splits into heads\n", "\n", "The reshape inside `multi_head_attention` is the one line that trips people up. All it does is cut each token's projected vector into `n_head` equal slices, run attention on every slice in parallel, then glue the results back and mix them with `Wo`. Here are the shapes at each step, and a picture of the split." ] }, { "cell_type": "code", "execution_count": 6, "id": "m6mhab", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.453873Z", "iopub.status.busy": "2026-07-01T09:08:58.453818Z", "iopub.status.idle": "2026-07-01T09:08:58.476937Z", "shell.execute_reply": "2026-07-01T09:08:58.476669Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "C = 64 -> 4 heads of head_size = 16\n", "\n", "x @ Wq : (4, 16, 64) (B, T, C)\n", "reshape + move heads up : (4, 4, 16, 16) (B, n_head, T, head_size)\n", "per-head attention q@k.T: (4, 4, 16, 16) one T x T map per head\n", "glue heads back : (4, 16, 64) then a final @ Wo mixes them\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAACMCAYAAACAlkp0AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQjtJREFUeJztnQd0FFUXx296CCkQSkLvvfcOCigCIqBgQwFRkA6CiAXpCIjwCYrSpChVUJrSBelFeu+9hB5ICOnznf/LmWWy2U02ISG75P87ZyE7Mzvz3rvvvblz3713nDRN04QQQgghhBAHwzm9C0AIIYQQQkhKoCJLCCGEEEIcEiqyhBBCCCHEIaEiSwghhBBCHBIqsoQQQgghxCGhIksIIYQQQhwSKrKEEEIIIcQhoSJLCCGEEEIcEiqyhBBCCCHEIaEiS9Kdjh07SsGCBW06dujQoeLk5JTkcRcvXlTHfffdd2Iv/Pvvv6pM+N9RQfkhA53Zs2erbWjvtOgX3t7e8jyRlu2VFBhjaNOMVGdH44UXXlAf83kMbZhW840+p965c0cc8Z5ACBVZYneEhYWpydWRFb6MzE8//WTzjZeyTl127Nihxk5wcLA8Dxw/flzVh0owIcQaVGSJ3QHlZtiwYRYV2UGDBsnjx4/TpVwkIe+//76SR4ECBVKsyFqTdUZpr9RWZNGelhTZU6dOyfTp08WR6gxFFvWhIksIsYar1T2E2CGurq7qQ+wDFxcX9cmoREdHS2xsrLi7u9t9e3l4eKTLdTNqH9E0TcLDwyVTpkzpXRRCnmtokSWJ+k2dPn1a3nvvPfHz85McOXLI119/rSboK1euSMuWLcXX11cCAwNl/PjxNvnFJeW3heNxHQBLDI41+mXa6iNrZNq0aVKkSBF1I69WrZr8999/CY45efKktGnTRvz9/cXT01OqVq0qK1asiHfMvXv35NNPP5Vy5cop303UvWnTpnLo0KEE57t69aq0atVKMmfOLDlz5pRPPvlEIiIi5FkTFRWl2rFYsWKqXtmyZZO6devK+vXrE/iinj9/Xpo0aaLKnDt3bhk+fLiSdWKYyxl+bceOHZPNmzebZGf0+UuOrHWuXbum2hJlxPGQQUxMTLxjoEx+//33UqZMGVXPgIAA+fjjj+X+/ftJtpGt9Tf6XeNaep+C1RBs3LhR6tWrp36fJUsWNT5OnDiRaHvprF692vRbHx8fad68uWpHS/30zTffVO0ABalEiRLy1VdfqX1otwEDBqi/CxUqZGpPo2zMfWRR57Zt26p+7+XlJTVr1pS///7b4pj9/fffZdSoUZI3b17Vxo0aNZKzZ88m2b6W6oyyvPrqq7Jt2zapXr26Ol/hwoXl119/jfc7lA28+OKLpvoY5w5Y/yFzyAEy69Gjh01uFfo8orcnxjLGRp8+fZTyaWTWrFnSsGFDNY5xndKlS8vPP/+c4Jx6ndauXavmD8hn6tSpyTqHrdgyXyUXtBv6B/ou5vsPPvhArZiYM3fuXKlSpYqqH67/9ttvq/uBka1btyrZ5c+fX9U3X758ag60tJq2bNkyKVu2rKoH/l+6dKnF8i1cuFBdF+MD8sI8PHHixKeqM3k+oGmLJMpbb70lpUqVkjFjxqgb3MiRI9XkhQkaE/PYsWNl3rx5SrmAkli/fv2nalHcoDHBd+vWTVq3bi2vv/662l6+fPkUnW/+/PkSEhKilBrcuL799lt1TtzA3dzc1DFQGOrUqSN58uSRzz//XCkTuGlDefrjjz9UOQB+g0kXEzQUhZs3b6p2aNCggVJmcCMFmKxxk798+bL07t1bbf/tt9+UomOr8vngwQObjoUsnJ2dE71hjx49Wj766COlMDx8+FD27t0r+/fvl5deesl0HBTDV155RSkyaKM1a9bIkCFDlMURCp2tQMHr1auXUgx1BQtKZUpljXJBuaxRo4ZSIDds2KAemqBE4nc6kC8UH9x80eYXLlyQH3/8UQ4cOCDbt283ydoayak/lBIoO126dFE3acgA5cJDDZQxtDn6wA8//KD6Fdo6scAV9I0OHTqoemI8QXlAu+CBA+XXf3v48GGl7KIuuDa2nzt3TlauXKkUTLQfHjwXLFgg//vf/yR79uymdrYE+m/t2rXV9dBmUOTmzJkjr732mixZssTU73UwB6CvYayjf6Kd2rVrJ7t375aUACUYytiHH36o6j9z5kylSEFZgXKKuQTlmjRpknz55ZdqHgL6/2hnPAA1btxY9QW4TqDd8KBqi8wBlFi0I8bIrl271LXw8GNUqHFOlAftgtUgtHf37t3VwxMUZyMowzvvvKP6Y+fOndWDRnLPkRS2zlfJBW2BeQ1tgT47Y8YMpXijT+qgn8GYgWMxp9y+fVv1c8gKfRVKMFi8eLHqV5AL+tWePXvUcXjAxz6ddevWyRtvvKEUe1z37t27agzjYckIHrzRrphX9fLgIRFyxsMHyeBohFhgyJAhMEVpXbp0MW2Ljo7W8ubNqzk5OWljxowxbb9//76WKVMmrUOHDqZts2bNUr+/cOFCvPNu2rRJbcf/OvhdgQIFTN9v376tjkEZrJUrKXBdHJctWzbt3r17pu3Lly9X21euXGna1qhRI61cuXJaeHi4aVtsbKxWu3ZtrVixYqZt2B8TE5PgOh4eHtrw4cNN277//nt1jd9//9207dGjR1rRokUT1N0SehvZ8jFvX3MqVKigNW/ePNFj0P44V69eveLVH79zd3dX8tAxl4slOZcpU0Zr0KCBZguJyVovl7FtQaVKlbQqVaqYvm/dulUdN2/evHjHrVmzxuL2lNZf71O+vr7arVu34p2jYsWKWs6cObW7d++ath06dEhzdnbW2rdvb7W9QkJCtCxZsmidO3eOd76goCDNz88v3vb69etrPj4+2qVLl+Idi7LqjBs3zmq/wBgzjtG+ffuqY9F+OihPoUKFtIIFC5r6ut4fS5UqpUVERJiOnThxotp+5MiRRNvXUh9BWbBty5Ytpm1oU4yl/v37m7YtXrzY4pjBsZDNyy+/HG9M/vjjj+r4mTNnJlomfR557bXX4m3v3r272g7Z6YSFhSX4fZMmTbTChQvH26bXCf3OHFvPgXFjHDt6n0MbJne+sjTXJtYWnTp1ire9devWav7UuXjxoubi4qKNGjUq3nGQv6ura7ztluo7evRode8w9l+Mm1y5cmnBwcGmbevWrVPlMd4T+vTpo8Yd7kGEmEPXApIoeOrWgZ8blrCgz8CKooOncFgeYLG0R4ty1qxZTd9h0QJ6WeEuAEspLAyw3CIFDT6wDMBCdubMGbW0DWB9062fsODhGFgeUXdYMHRWrVoluXLlUtYmHSzbwopmCxUqVFAWCFs+cOtIDMgGFhzUIyl69uxp+hvWa3yPjIxU1sb0pGvXrvG+Q4bGvgYLD5ZCYWHW5YcPLHuQz6ZNm2y6jq31hwXJaOW8ceOGHDx4UFkTYZ3VgWUZZUJ/sAZkiCVdWJuMZcdYgxVaLzssX1u2bJFOnTqp5VojyXW10UG5YKWH5VcH7YV+CjcA3WVCB5Yyoy+w+VhKLrDC6ecAaFNb5xHIBLLp27dvvBUJWEGx7GzuHmENc2soVhOAUWZGH1dYoiEfrMKgnOYrJ7BoYt4wJznnSIzkzFepMc5wXqzigD///FNZkHFtY1/FHATXJeM4M9b30aNH6jhY/3HvgOXWOG5gjcf41cGYQd8wn8dwHqNLFCE6dC0giWJ+08SEA18mfdnSuB2T3rMGEztuaMYJ1DgpmpdfV2p130ksb2JyxXIZPpa4deuWWsbDJA6fLPjlYena6KeJ5TOdS5cuSdGiRRMoGPoyY1KgjFguTQ2wLA5fzeLFiyv/MyyfI4rc3FUDygCWxY3gNyA9I8bR18yXxtE+Rt9X3LyhDGAZ1Jr8kiI59YeyYgTytiZfLIPDZxI3YSwBm6M/YMBNxxJQyoCu3EGGqQXKDWXZUpn1/cbrJTWWkov5+fRz2nI+a20ORRty1PcnBRQwI3BZQV8wyhzL13Az2blzZwKfUfQ743xj3jdSco7ESM58lVwSky/6Ifoqrm3eZjpGVw64VQ0ePFj57ZrLU1fcdRlZOp+5cQBuGHCfgPsO6vbyyy8rhRrzGSFUZEmiWIo2thaBbAyMsWYlMg/SeVrgF4jAIh083RtTPyVVViinAH5/liwpAEop+Oabb9TNA1axESNGmPxTYRXSz5MaQDGHgm4LUPISiwiH7xr8KJcvX6780eD3Bv/JKVOmxLO22yu2RLuj7aHEwlfbEtZ8RFNKakah6/0GfrKWrOv2lKHDlnGfnudLDcznLYwd+GWWLFlSJkyYoIKWoCzDYotxZD7uLfWN5J4jMZIzXyUXW+ZKtA8CEy0dq7+8BHM8rKqYwwYOHKjqjYc4WIqxapGSuRLjG9ZbPBTi+vjAV719+/bKr5tkbOxnliTPFfrTvHkEsS2WkuQslSLwx/jErwdc2YpuhYM1ISkrKAJgED39yy+/xNuOOhot1MiXefToUXUDMNYFgSC25gLFdWwBluGk3oADhRvLwviEhoYq5RaBMkZFFjcXWP10KyRA4BBI7ht2kiO/lC6Lm1vRsNSMAJiUKplPU389P6ol+SK6HH3DkjVWL7t+o06s/+n9FP0qtdoT5bZWZn1/emOtPsY2N1rS8RCIMWHrigasjEYrKiye6Au6zBGUhWwjsCwaLZa2uquk1jlSMl+lNuirmNPQXsZxYs6RI0fU2IGCCUVTx9wtQJehJbcnS/0Syn+LFi3UBzKClRbBtjAupFR5J88H9JElaYJ+g4Zfnw6e1JEKKyngTwpsSaMDP0hM6PrH3LcqKaBAID0UJkT4bJkD30QdWCHMrUXwzzT3SWvWrJlcv35dKb46WE60pe6p7SNr7u4BqwkmfUupwBDlr4N64jtumLAmJQcobba+WSo5srYGlhjRt2AlNwdZB2w9d0rrD3/oihUrqhu38VpQOmEFR3+wBqxqWLaFtR/ZKqz1P1iV8QCCyH4s2xox9kldYbalzigXosmx3K0DFwj0UyhyyR1LaYG1+mCsQ7FBlgFj/fGQiaVrpC+zhcmTJ8f7jsh6gCVsoFsejdfA+WENtJXUOEdK5qvUBqtfqAsyRZjPg/iuzzWW6ou/zVNlGceN0U8Y85q5f7b5PIaVMN09Kj3SGhL7ghZZkiYg1QxSGX3xxRdqiQlWQeQBhGKRFLCq4Sa6aNEi9eSP38JXLzX9A81vZgh4QV5CBIvA6oHURLjBI12MnicWOSLhcwrLJgIXYHnAcra5byXOASUI1oh9+/apCRtLx7rS9ix9ZNGOuPFB4Uc7IvUWFGxjYJPui4qUU3DNgN8klu4QMIO0R8ldmse1kG4IqdqgNOPma80HNDVkjaAZpDtC+h4sP8J/DgooLD140MAN1Bh4Z4mnrf+4ceOU8lOrVi0VCKmn34Lvo3leXCNQYtFW8FuuXLmyysmJ60FZxfVhZdYVbCht6Kc4DgFZsIzBlxPHod562wOkPsO50A6wYFmyCCN1E1J1odxIc4W2h1IBiybSOCWW1u1ZAUUHihFSLkHZQcClno8VcwuUKvhJIq0VrHjwX0caQOS+tgXUFb/FOTDekSP13XffVQ+TAH1JtwSij2FFA29Hw/UtKZKWSI1zpGS+SgvjBMY02h39Dum+kNMVbYjcr+iTcHmAKwGOxd94yEcfR3+y5PuMMYuHDtQHLlu4V2Dc4P6BdtLB6hH2QfZIzYWVPRyH/qH7dJMMTII8BoQYUrIYUy8BpO/JnDlzgjZCyhikXTJy7tw5rXHjxiqlTkBAgPbll19q69evTzL9FtixY4dKsYQUO8b0TMlNv4V0ROZYSveEsiJNUmBgoObm5qblyZNHe/XVV7UlS5aYjkG6G6QGQroYpBurU6eOtnPnzgQpcwBSzCC1j5eXl5Y9e3aVPkZPB5VUOpzUZOTIkVr16tVViieUuWTJkipNTmRkZAKZog2QzghlhrzQRubpxmxJv4XUUUhdhVRR2JdUKi5rsrbW16z1gWnTpqnzoJ64NlIUffbZZ9r169cTvb6t9U+sT4ENGzaoPoHrI1VQixYttOPHj8c7JrG0dEjHhJRbnp6eWpEiRbSOHTtqe/fujXfc0aNHVVokyBPHlShRQvv666/jHTNixAjVf5H6y3gt8/RbAHVu06aN6XzoK3/99VeCsuE8SIVlxFJqqOSk37KUFs7SWJo+fbpKU4XUT+bjB+m20KcxZiGzbt26qXSASaH3IcgH9Ud/yZo1q9azZ0/t8ePH8Y5dsWKFVr58edU+SEs2duxYld7L1jol5xy2pN+ydb5Kbvot87neWl/9448/tLp166oxgw/av0ePHtqpU6dMx6BdMfd7e3ur+Q9p5JDSzFJdcD6kdsN9onTp0tqff/6Z4J6AemFsIsUd5on8+fNrH3/8sXbjxo1E60YyBk74J72VaUJI+oEADFhpjRaQjMSzrD+WvmFdwpuQzJO+k2eH/jIFLMWbZ2AhhDgW6b92RAghGQQsJSOAyZhvlhBCSMqhjywhhKQx8GGE1Rdpz+BHa6u/NCGEkMShRZYQQtIYvBd+wIABKvjNmOeYEELI00EfWUIIIYQQ4pDQIksIIYQQQhwSKrKEEEIIISTjBHvh9XB4cxGSIafGKyYJIYQQQggByAwbEhKiXjuf1MtZUqTIQonNly9fSn5KCCGEEEJIktiScztFiiwssfoF8Po5QgghhBBCUoOHDx8qg6mub6a6Iqu7E0CJpSJLCCGEEEJSG1vcVxnsRQghhBBCHBIqsoQQQgghxCGhIksIIYQQQhwSKrLkuSM2VkvvIhCSAE2LYasQu011RIijkqJgL51fT++WYOfo1CsNIU9J2ay5pVWhCrJqy3m59yCc7UnsgoJ5fKVu5bwiN74UiTyf3sUh5AledcUpR085cOCAyttJiD2QnHcUPJUiGxT2UG47RTzNKQhJVQIzxaWDgxJ7614YW5fYBf5+nnF/QImNOJnexSHkCe4F1X9QYpHyiBB7wMXFxeZj6VpACCGEEEIcEiqyhBBCCCHEIaEiSwghhBBCHBIqsoQQQgghxCGhIksIIYQQQhwSKrKEEEIIIcQhoSJLCCGEEEIcEiqyhBBCCCHEIaEi+xRMq99OIkIeSVoRFRaurmGN/XOWyoK3P1GfPdN/T7NyEMejf8dq8vhR2r2lJyI8TF3DEpfPH5NJIzvJ513qysyJn6ZZGYjj4VTikAQ/TLtX9YY+ilHXsMTCv+9LxZanpOyrcZ/xM2+lWTmI49GiRQsJDQ1Ns/M/fvxYXcMSO3fulF69eknv3r2le/fu8uuvv/K1wcngqd7sRdKPGwdPyLl/dkqbWaPF2cVFlvcYJoFli0n+WpUoFpKu+GbJLq3e7S/XLp2SE4d3UBrELsgX6C5rZhSWwBxu8iAkRqq8flqqlPGSF2p4p3fRSAanQoUKUqNGDXF2dpaoqCgZOHCgFCtWTGrVqpXeRXMIqMg+JceXbZCLW/dKeHCIVO7YWko0a6C2P7gSJDt++E3Cgx9KTGSUlGzRUMq+8bLat3H4ZAm+ckNio6Ilc85s0mBgZ/HKliXufMv/kcML/xY3L08pVN+yxQuc27hLir1cV9wyxb36Etc9u2EnFVliYvvGJXJ0/7/yKCRYXmr5oVSv95rafjvosiyfP0FCQ+5JdFSU1HyhtdRt/KbaN3fKILU/JjpKsvgHyJudBinFFOzc9Kf8u2auuHtkkvJVXrTa0vgdPkHXzlMaJAE/zb8jyzY8kNv3YmRwjwD54A1/tf3MxQjp+801uXU3WiIiNenyVjbp+V5c32vX/5KcuhAhkVGa5MvlJr+MyqcUUjB14V35buYt8fZyltdfiptHLVGnSmbT334+LlKysIdcvBZJCRETq1atkl27dsmDBw/knXfekcaNG6vt169fl+nTp0twcLBER0dLkyZN5NVXX1X7vvvuO7l27Zranj17dmVVzZo1q9q3evVqWbp0qWTKlClRpdTLy8v0NxRZfIjtUJF9SlzcXKX1tBESfOm6LO3ytVIuxclJ/hn+ozQc1F2yFMgt0eERsqzrEMlZuojkLFVEavV+XzJl8VW/Pzh3heyb9YfU+/RDuXf+ivr7jRmjxCt7VtkzbZHV64beuiuB5UuYvvsE5lAWWkJMg9vVTfoOmSM3r1+UicM7SJXazcTJyUkpq+92GS4BuQtKZES4TBrxgeQvXEZ9YEn19o2bhP/5a7asWzZd2nT8Qm5cPStrl06VfsPnKcV21ZLJbGiSIjzcnWTPkuJy8ly4VGtzRt5vmRVTprzT75LMHZdfShbxlLDHsVLzzTNSo7yXVCvvJd9/lUdy+MfdrsZMuylDf7gpU4bnlaOnH8uQH4LkwNLikiunm3w54YZNZTh+Nlx2HgyTKcPyUorEhJubm0yYMEGuXLki/fv3lxdfjHtgHzdunPTr10/y5csn4eHhMmDAAClevLj6dO7cWfz8/NRxixcvlvnz50uPHj3k0qVL6u+JEyeKv7+/chdIjBMnTsjkyZOV0ty0aVOpWbMmJWMjVGSfkqIv1VH/Q2F1cnGWsHvBEhX2WO5fuCr/DPshnr9r8MVrSpE9u36HnFm3TVlq8fH0i1vaur7/mOSrUUEpsaB0y8ZK0SUkJVSu9Yr6Hwqrs7OLhDy4K+GPHylL6dyfv4zn73rz+gWlyO7ftUb27Vgt0VEREhUVKZm94yxcZ0/slZLla5uss7VefEMpuoQkl3Yt4uY3KKyurk4SdCdKHobGyrGz4fJ2v0um40IexcpxKLvlvWT+yvvy2/L7Eh6pSXhErGTPGnfr2rgrVJrW91FKLOj2TjYZPTVx39erQZHSsvsFmTI0r+QNdKcAiYkXXnhB/Q+FFcv89+/fl7CwMLl8+bJSZnWwDcouFNnNmzfLpk2bJDIyUllSfX3jjFSHDh2SKlWqKCUWQDmFomuNUqVKyY8//qiswaNHj5Zjx45J2bJlKR0boCL7lLi4x02gAIqsFhMrmibi4estb8wcneD4oMOn5Ogfa6XVz0MlU1Y/ubhtn+ybucTyyZ2sX9c7ZzYJDbpj+h4SdFu8A+KUDEKAm5vHk67k7CyxMQiy0cQrs6/0HzE/QSOdP31Qtq1fJL2+nik+vv5y9MBmWfvnVMtdEyY0QlKAp8eTGGMXZ5HoaFFzpr+fixxc/mSVSWfb3lCZ9Nsd2bmoqOTM5iYr/nkggycFWemXiV/7+s0oadzxvAzqFiBtm1p3QyAZ1yKrA0U2JiZGBV15e3vLpEmTEhwPZXPlypVKyc2SJYvs3r1b5s2b91RzJqy7UIC3b99ORdZGmLUgDciSL5e4e2WSU6s2m7Y9uBok4Q9DVZYDdy9P8fD1kZioaDmxYqPpmNyVy8iV3Ycl7G6w+n5i+T9Wr1H4xRrKqhv1OFxZdXGtIo24FEESJ0dgAfHMlFn2bH1i6b9z84qEhT6Qx48eioenl2T29pPo6CjZtWmp6ZiiparKySM75WHwHZO/LCGpRYlCHuLr7SKz/rhn2nb2UoTcC46W+w9jxCezs2TL4iqRkbEyddFd0zENa3rLmq0hEnQ7zqdwyoIn+8y5cStKGnU8JwM755QOreOsZIQkRd68eZUP64YNG0zbsPwfEhIijx49Uv6vPj4+yhq7Zs2aeAFc+/fvV1Zd3V/WGrDuxsbGmqy9e/fulYIFC1I4NkKLbBrg7Ooir4z9VAV7Hfl9tWixseLp5yMNB/eQfDXKKwX09/c+VVbbPFXLStiduMnbv3A+qfLB67Ki5/Akg71yVyothRvWlCUdP1ffizSsKQVqV06L6pDnCBcXV/nwk//J8nnjZcvaBRIbGyOZfbLIe11HSslytWXfztUy5vM2SpktVrq6PLgft0ybK29RadKqi0we3SXJYK9bNy7KlG97SGRkuERFRsjwT5pLo1c7Sp1GbZ9hTYkjAReDv6YWkr7fXJf/zb4tMbGach+YP76AvFLPV+auuC8lXjmplNnGtb3l2s04xbVs8UwytGeg1Gt3NslgL1hxL9+Ikom/3lYf0Kd9DlOwGSGWcHFxkcGDB6tgr+XLlysrLdwH4CdbuXJl5VbQrVs3pcxCeb17N+5hqkCBAipgDBkIkgr22rZtm2zdulVdCwptnTp15OWX44LDSdI4abCbJ5OHDx8q8/dXm5fIbaeI5P6ckDSjWvYC8lGpOjJ35XG5dS+MLU3sgpKF/KVZ/cIil94WiTiZ3sUh5Ak+r4jkGiNbtmxR93ZC7AEo9c2aNVM+w7rfsTXoWkAIIYQQQhwSKrKEEEIIIcQhoSJLCCGEEEIcEgZ7pRLT6reTrIXySo2u70j+WhXVtv1zlsqp1VvU30Ua1ZLqnePenmTOgd+Wy+k1W1Vmg5dH9pWC9aqa9sGFed+sP+Xchh3i7OYqnll8pMXEQaY3hF3bf0yKNKwltXu/n1pVIQ5O/47VJDBvEXm1bS8pVaGOXD5/TJbNHy/XL5+W4mVqSqc+38U7/tzJfbJy0UT1cgSAt3kVLFo+wXkXzhgmVy+eVGlkXFxdpVnbnlK8dHXTvgtnDoubu4d4eGSSlu/2U3lpwcqFE+XgnvWSp0DJBNcmGQenEoekbHFPGftpLmnWwFeWrn8gQyYFibOzqDd2tX7JT0b2DVT9a+Hf92XMtFsSjYxxIvLB61mlf6eciZ7/xLlw9drZLm9mUy9QALfvRUunL6/IpWuREhWtSfXyXuolCJk8nWXRqvsy7Mebcv1WlATvLfcsmoDYIS1atFCBWR07dpSqVZ/ceyMiIuSTTz4RV1dXU+otBGLNnDlTZSOADycCvHr27Cm5c+e2eG6k0FqwYIG6jwMEjQUEBJj2401hvXr1khIlSsigQXH3dQR94Tf37t2ThQsXpnHtnw+oyKYir/04WDx84l6DeOPgCfWmrTazRouzi4ss7zFMAssWs/gK2TxVyipFd/OYaQn2HV2yVu6duyxt5oxVbxHTU3MBZEHYO/MPiQxlUBOJT88vpkumzD7qb7zEAG/sunbplJw4vCPecQ/u35YF04dJ5/4TJSB3IYmOilSZBizR8p1+pnNevXRKpnzbXYb/sF7lWyxb5QVp+8FXKivC8YNb5dfJX8ig8XEpvlq83UcC8hSWo/ufpKMjGZOt84pKFl8X9XfjWt7SslFxcXZ2Umm16r57VqqW9VIKbb5Ad1kzo7B6De2DkBiloFYp4yUv1Ih7eYw5UVGadPn6qrRuHPeGJZ1RP9+UYgXcZeWUQhITo0nzLhdUiq/u7bLLW82ySo3ymaViq1PPpO7EfhkzZozKFWtk9uzZ6iUFZ86cMW1Dnli8gQuKLRTcRYsWqTd2ff55XPYgI+fOnZPffvtNRo0aJdmyZVNptTBXGsGbvKpVqxYvyK5evXrqRQt9+vRJk7o+j9C1II04t3GXel2tWyZP9dKEEs0ayNkNll8hi1fX+ua2bG04vPAvqd71baXEAq9sTOJNkkcW/wBlHXVxfZLsW2fHxiVSuVYTpcQCVzd3k7JqjnF7eFhovH1lKzVQSizIX6ScPAi+JTEx0RQVsYqPt4tSYlV/itQkIlIzvdCgTpXMSokFfj4uUrKwh1y8Fmn1XMMnB0nbV/ykWMEnLwEBOB/eEBYbqymrb1h4rOQNTDgOCDFy8OBBlUZLf9PXk/7kpPLF4i1esLJCOc2e3fKLiJYtWyatWrVSSixALlpPT0/T/nXr1inrbOnSpdn4TwkV2TQi9NZd8Q580sF9AnNI6M24ZPK3T56X1QO+TfIckY/CJOzeQ7m0bZ8s/Xiw+sDKS0hqcfP6eWWBhXV1/Nfvyp+/jZOIiMdq35ULx2X6hPhWgb9+/0G+GdBKZv/4mXTsOTaBhQFsXbdASpWvY1JsCbHGjv2PpFyLU5Kz1jH1coOWjRKm2Tl+Nlx2HgxT+WP1t3NVbPnEirr70CO1v9f7CRWKr7sHqBcrBNY5rq5RqoiHvNYovtWWECOhoaEya9Ys6d69e4KGqV69upQrV07at2+vPngNbbt27Uz7e/fubcoji9fa3r59W1lrYV2dO3euykELgoKC1AsS3n+fLoGpARXZdCBHycLSdNxnSR4Xi9fdxsRIdESktJ46XBoP7SU7f5wrd88+eR85IU8DJtbzpw5I+x5jpO+QX+Vx2ENZuzTutbT5CpWWzv0mxjv+1Td7yZfjlkn77t/IX79PUm8AM7Jvxyo59N8GadvxSwqGJEntypnlyMoScmVzadl37LFs3fso3v6rQZHSsvsFmTI0r+QNdFfbcge4mV5lG/Y4VroPuybTR+S1+ArQhX8HS+minnJjW2m5vrW0nL4QITMWW3/7FyFTp06VN998U71y1pyzZ8/KpUuXlNvBnDlz1AsQfvrpJ9N+uBzoFlj4054/f16GDRsmY8eOVS4JUF5hycVxXbt2FQ+P+CsIJGVQkU0jvHNmk9CgOAssCAm6Ld4BlpcgrOHp661cE+CiAHxy5ZCAcsWVRZeQ1CBrtkAVEOaV2VcFcFWq0UQunTua5O+Kl6kh4Y/D5MbVs6ZtB3avk3XLZ8jHAyaLj1/cZE6ILeTwd5VmDXxk8ZonMQCwvDbueF4GdQuQtk0tu1Sduxwhl69HyosdzknBhsfl+zm3ZeYf96TDwMtq/08L7kq7FlnFxcVJuTK0eSWLbNoV3y2GECPHjx9XAV0ffvihfPvtt+r1sVA6wcaNG6V8+fLKnxarUQ0bNpTDhw9bbEC4HNSuXVspq3ApwN+nTp1S7ggXL15U58Y1cK0DBw7IV199RUGkECqyaUThF2uoV9FGPQ6XmMgoObVqsxRpVDPZ5ynSuJZc2X1I/R3+MFRunzgn/oXzp0GJSUakUs0mcvbkPhXkBU4e2SG58xVLcFxMdLTcuXnF9B2ZEEJD7ku2HHHR4chKsObPKUqJhXJMSFKcPBeufFdBSGiM/P1viJQvkUl9v3ErShp1PCcDO+eUDq2tv0K2XIlMcntXWbm4sbT69O2QQzq94S9zxsbNkYXzusuarSGmgLC1W0NU5gRCrPHLL7+YPp999pnky5dPpkyZovYFBgYqxRV+suC///5TGQ8s0aBBA6WgwjKLlS/8XahQIcmcObPMnz/fdI1OnTpJpUqVVFAYSRl0YksjclcqLYUb1pQlHeOiGYs0rCkFaldWf8OiuveXJSb3AqTpOrHiH3kcHCL3LlyR7d/Pltd/+UYyZfGV6l3ekn9HT5PjyzaoYyu820IFhxFiK7duXJQp3/aQyMhw5Q87/JPm0ujVjlKnUVspVKyClKlYXyYMeU+cnJ0lMHdhadPhC5OP7JqlU5V7AQK3FkwfKuGPQ8XZ2UXcPTJJhx5jlCUXzJv6tbLCzprY33TdrgN/kszeDE4kllm0OlgWrQoWN1cniYkVadPETz5qG6e0Dp4UJJdvRMnEX2+rD+jTPod88Ia/stQ263Le5F6QGBO/yi1dh15VfrjIWlCrYmb5pGMOioSkiObNmysLLXxhkbUA7gc9evQw7cf2IUOGKPeC+vXrq8wF2A/rbZkyZVSqL5L6OGl6grNkgFQRfn5+8tXmJXLbyXKqnoyYR7bD39NM6beeFXr6LeaRjaNa9gLyUak6Mnflcbl1LyzD5pEdOXmj1ewD6cGerStV+q2Mmke2ZCF/aVa/sMilt0UiTkpGzSN7/7+ypvRb9sDFq5Eq/VaGziPr84pIrjGyZcuWeGmgMgpQLpG31Tz9Vnpy8+ZNFSCWkfPIuri4SLNmzeTBgwfi65swCNQIXQtSiUz+frKy90i5vPOgPCvwQoSz67eLe+a45ThCgI+vv0we87GcOLTdLhoEL0TY+Pds8bIjxZo8ewKyu0qD987Kqs32oSzhhQgtul2QgGxMx5WRgVX1iy++kL1794o9gBcijBgxwmKwGbEMLbLkuYIWWWKP0CJL7JYMbpEl9gktsoQQQggh5LmHrgWEEEIIIcQhoSJLCCGEEEIcEiqyhBBCCCHEIaEiSwghhBBCHBIqsoQQQgghxCGhIksIIYQQQhwSKrKEEEIIIcQhcX2aHwd6+Yqnc3TqlYaQpyS7Z9xrBv39PNmWxG7w9XaP+8O9cHoXhZD4uOZV//n48M17xH5wcnKy/VhN07TkXgBv//Dz87PpHbiEPGtiYzVxdrZ9EBDyLNC0GHFycmFjE7sDakByFAdC0prk6Jl0LSDPHVRiiT1CJZbYK1RiiSNDRZYQQgghhDgkVGQJIYQQQohDQkWWEEIIIYRknKwFenwYnHEJIYQQQghJLXT90pZ8BClSZO/evav+z5cvX0p+TgghhBBCSKKEhISo7AWprsj6+/ur/y9fvpzkBUj6PMngIePKlStMj2aHUD72DeVj31A+9g3lY988dBD9AJZYKLG5c+dO8tgUKbLOznGutVBi7bkhMjqQDeVjv1A+9g3lY99QPvYN5WPf+DqAfmCroZTBXoQQQgghxCGhIksIIYQQQjKOIuvh4SFDhgxR/xP7g/Kxbygf+4bysW8oH/uG8rFvPJ5D/c1JsyW3ASGEEEIIIXYGXQsIIYQQQohDQkWWEEIIIYQ4JFRkCSGEEEJIxlFkJ0+eLAULFhRPT0+pUaOG7NmzJ/VLRpJky5Yt0qJFC5Uw2MnJSZYtWxZvP9yfBw8eLLly5ZJMmTJJ48aN5cyZM2zZZ8To0aOlWrVq4uPjIzlz5pRWrVrJqVOn4h0THh4uPXr0kGzZsom3t7e88cYbcvPmTcroGfDzzz9L+fLlTfkUa9WqJatXr6Zs7JAxY8aoOa5v376mbRw76cvQoUOVTIyfkiVLmvZTPunPtWvX5L333lP3F+gA5cqVk7179z53OkKyFdlFixZJv379VNTb/v37pUKFCtKkSRO5detW2pSQWOXRo0eq/fFgYYlvv/1WJk2aJFOmTJHdu3dL5syZlawwwZC0Z/PmzUpJ3bVrl6xfv16ioqLk5ZdfVnLT+eSTT2TlypWyePFidfz169fl9ddfp3ieAXnz5lUK0r59+9Tk3rBhQ2nZsqUcO3aMsrEj/vvvP5k6dap66DDCsZP+lClTRm7cuGH6bNu2zbSP8klf7t+/L3Xq1BE3Nzf1gH78+HEZP368ZM2a9fnTEbRkUr16da1Hjx6m7zExMVru3Lm10aNHJ/dUJBWBKJcuXWr6HhsbqwUGBmrjxo0zbQsODtY8PDy0BQsWsO3TgVu3bik5bd682SQPNzc3bfHixaZjTpw4oY7ZuXMnZZQOZM2aVZsxYwZlYyeEhIRoxYoV09avX681aNBA69Onj9rOsZP+DBkyRKtQoYLFfZRP+jNw4ECtbt26Vvc/TzpCsiyykZGRynoB87PxdbX4vnPnzrTQs0kKuXDhggQFBcWTFV73BlcQyip9ePDggfrf399f/Y+xBCutUUZYmsufPz9l9IyJiYmRhQsXKms5XAwoG/sAKxrNmzePN0YA5WMfYBkarm2FCxeWdu3ayeXLl9V2yif9WbFihVStWlXatm2rXNsqVaok06dPfy51hGQpsnfu3FETfkBAQLzt+I4GIfaDLg/Kyj6IjY1V/n1Y6ilbtqxJRu7u7pIlS5Z4x3I8PTuOHDmifJORHLxr166ydOlSKV26NGVjB+DBAu5r8DU3h2Mn/YHCM3v2bFmzZo3yN4diVK9ePQkJCaF87IDz588ruRQrVkzWrl0r3bp1k969e8ucOXOeOx3BNb0LQEhGsSwdPXo0ng8ZSX9KlCghBw8eVNbyJUuWSIcOHZSvMklfrly5In369FG+5QgqJvZH06ZNTX/DfxmKbYECBeT3339XgUMk/Y0nVatWlW+++UZ9h0UW9yD4w2Kee55IlkU2e/bs4uLikiCqGt8DAwNTu2zkKdDlQVmlPz179pS//vpLNm3apAKMjDKCu05wcHC84zmenh2wiBctWlSqVKmiLH8Inpw4cSJlk85gaRoBxJUrVxZXV1f1wQMGAlPwN6xGHDv2BVaWihcvLmfPnuX4sQNy5cqlVpeMlCpVyuT+8TzpCM7JnfQx4f/zzz/xtH58h18ZsR8KFSqkOqNRVg8fPlSRiZTVswExeFBisVy9ceNGJRMjGEuIKDXKCOm5MNFQRukD5rOIiAjKJp1p1KiRcvuAtVz/wLoEP0z9b44d+yI0NFTOnTunFCjObelPnTp1EqR7PH36tLKaP3c6QnKjwxYuXKii2mbPnq0dP35c69Kli5YlSxYtKCgobcLRSKIRvQcOHFAfiHLChAnq70uXLqn9Y8aMUbJZvny5dvjwYa1ly5ZaoUKFtMePH7NVnwHdunXT/Pz8tH///Ve7ceOG6RMWFmY6pmvXrlr+/Pm1jRs3anv37tVq1aqlPiTt+fzzz1UGiQsXLqjxge9OTk7aunXrKBs7xJi1AHDspC/9+/dXcxvGz/bt27XGjRtr2bNnV9lZKJ/0Z8+ePZqrq6s2atQo7cyZM9q8efM0Ly8vbe7cuaZjnhcdIdmKLPjhhx/Uzdfd3V2l49q1a1fql4wkyaZNm5QCa/7p0KGDKb3G119/rQUEBKiHj0aNGmmnTp1iyz4jLMkGn1mzZpmOwYTRvXt3lfYJk0zr1q2VskvSnk6dOmkFChRQ81iOHDnU+NCVWMrG/hVZjp305a233tJy5cqlxk+ePHnU97Nnz5r2Uz7pz8qVK7WyZcuq+3/JkiW1adOmxdv/vOgITvgnva3ChBBCCCGEPJNX1BJCCCGEEJLeUJElhBBCCCEOCRVZQgghhBDikFCRJYQQQgghDgkVWUIIIYQQ4pBQkSWEEEIIIQ4JFVlCCCGEEOKQUJElhBBCCCEOCRVZkqb8+++/4uTkJMHBwRm+pTt27CitWrV6bso2e/ZsyZIli+n70KFDpWLFimlQOsfm4sWLagwcPHhQ7JX0kGVS13C0uSM95fws2mrZsmVStGhRcXFxkb59+6bZdQhJLlRkSarxwgsvPNMJDpN2jx49JFeuXOLh4SHFixeXVatWWTx2zJgxaqJPzwl44sSJSmF4XhXjTz/9VP755x95XnEEhfR5kmXt2rXlxo0b4ufnl65j4u7du5I3b16HUqrTgo8//ljatGkjV65ckREjRlg8Ztq0aeo+4Ovrm2h7/f3331KjRg3JlCmTZM2a1a7mMeJ4uKZ3AQhJCZGRkfLSSy9Jzpw5ZcmSJZInTx65dOlSPKuSzn///SdTp06V8uXLp1lZ3N3dkzwuOTdkR8Tb21t9HB28tTsmJkZcXTPu9GgPssSYCgwMlPTmww8/VHPHtWvXJKMSGhoqt27dkiZNmkju3LmtHhcWFiavvPKK+nzxxRcWj/njjz+kc+fO8s0330jDhg0lOjpajh49moalJ887tMiSVAGWkM2bNyurI57E8YEFS2ffvn1StWpV8fLyUpaWU6dOxfv98uXLpXLlyuLp6SmFCxeWYcOGqQnOGjNnzpR79+6p5a46depIwYIFpUGDBlKhQoUEE3C7du1k+vTp6snflnrAOoDr58iRQ1kWunbtqpRVHVgcevbsqay72bNnV5M7QP2rV6+urMOwEn/++efx6mBuLYqNjZXRo0dLoUKFlGUCZYdSbuTYsWPy6quvqnL4+PhIvXr15Ny5c2pZds6cOard9PbG8iKAxeTNN99USr2/v7+0bNkyniygpPXr10/tz5Ytm3z22WdKeUsKWJPz58+vZNi6dWtlqUpsqVivL25YAQEB6nrDhw9XbTJgwABVNli6Zs2aZfoN2hlti/ZDXyhQoIBqo6eRV1LtrC/Lrl69WqpUqaLkt23btgTXwu9BpUqV1PHoB/r5US/UBb9FG6xZs8ZqmdH+nTp1kpIlS8rly5dt6v+43owZM1S7o/2LFSsmK1askORw6NAhefHFF1U/Qjuhrnv37rV52R9jrkyZMqb+DTnpwPL20UcfmWQABQXXM18VQT/A9aEchoeHJ2u5XHd/WLt2rZQqVUop2lCYYLXVy2xtTBw5ckSVCfJHn+/SpYuaG5Li559/VteHhdpWzp8/r9oZckJf27lzZ7z96FsYxyhLvnz5pHfv3vLo0SPT/t9++03NlWgnKPLvvvuuUiKNYOUJK1A4B65lHN8AD/UtWrRQc17mzJmV3KytVoH79+9L+/bt1fEod9OmTeXMmTNqH9oQZQFoQ2O7moM5EfNezZo1Le5Hn+7Tp4+MGzdOjVPUoXTp0mq+IiTFaISkAsHBwVqtWrW0zp07azdu3FCf6OhobdOmTdCQtBo1amj//vuvduzYMa1evXpa7dq1Tb/dsmWL5uvrq82ePVs7d+6ctm7dOq1gwYLa0KFDrV6vadOmWrt27dT1cubMqZUpU0YbNWqUuqaR9u3ba3379lV/N2jQQOvTp0+i9ejQoYPm7e2tvfXWW9rRo0e1v/76S8uRI4f25Zdfmo7BeXDMgAEDtJMnT6rP1atXNS8vL6179+7aiRMntKVLl2rZs2fXhgwZEu/cLVu2NH0fOXKkVrJkSW3NmjWq3rNmzdI8PDxUOwGc09/fX3v99de1//77Tzt16pQ2c+ZMdb2QkBDtzTff1F555RVTe0dERGiRkZFaqVKltE6dOmmHDx/Wjh8/rr377rtaiRIl1H4wduxYLWvWrNoff/yh9n/44Yeaj49PvLKZs2vXLs3Z2Vn9FuWYOHGiliVLFs3Pz890DOpaoUKFePXFeXv06KHK/Msvv6i+0KRJEyWr06dPayNGjNDc3Ny0K1euqN+MGzdOy5cvn+oTFy9e1LZu3arNnz//qeSVVDvrfbR8+fKq7509e1a7e/dugmvt2bNHHbdhwwbV3voxEyZMUP13wYIFqp6fffaZqhPqBy5cuKB+d+DAAS08PFxr3bq1VqlSJe3WrVs293/8Pm/evKotzpw5o/Xu3VvV21I5rYEx8t5776n+ibL9/vvv2sGDB9U+tElisvzpp580T09P7fvvv1fyR1v873//M+1v3Lix1qJFC9VPce7+/ftr2bJlM5Vv0aJFqs1nzJih2uirr75SfcN4DXN0udy/f99URrQrroXr7Nu3T/V19G9gbUyEhoZquXLlUuPoyJEj2j///KMVKlRI9Z3EwFwVGBioXbp0KUFZLKHLGX0N/RDt1KZNG61AgQJaVFSUOgZ9K3PmzKrt0E7bt29XfaFjx46m82CcrFq1SvWFnTt3qnkV853O5cuXVVv269dPteXcuXO1gICAeOVr3ry59tJLL6k5AOdZuXKltnnzZqtlf+2111Rboi+iT2CMFi1aVM0naEPUBefHnKG3a2JYa6/du3er7ZjHKlasqNoX8oJcCEkpVGRJqmFJUdQnNNz8df7++2+17fHjx+p7o0aNtG+++Sbe73777Td187EGFDNM5lDY9u7dqy1cuFApfcabPxSLsmXLmq5jqyKL8zx69Mi07eeff1ZKQ0xMjOk8uPkYgeKEMsXGxpq2TZ48Od7vjIosFBoovjt27Ih3HiiV77zzjvr7iy++UDdc3EysldVc+US7mZcDN51MmTJpa9euVd/Rrt9++61pP26yUJISU2RRpmbNmsXbBuUxKUUWN3G9/gBlw4OMDh48cGOHrECvXr20hg0bxiv/08jLlnbW++iyZcsSvZZRITWSO3dupZgbqVatmnqoMf4OSjn6et26ddWDn44t/R+/HzRokOk7lDNsW716tWYrUByhLFsiKUUWdYTyaQnUC4o42tpIkSJFtKlTp6q/oYzp7aGDh9vkKrL4DmXQOMagxCU2JqZNm6Ye3NBmxjkID2ZBQUEWr4264MEGcrBUFkvocoayblSGsQ0PD3q/69KlS4L2Q1n0ecocKO04BxR1fV4oXbp0vGMGDhwYr3zlypVL1BBgBAo1fgulWufOnTtqzsDDDsB5cQzawRastRfGObbnz59fW7JkiZq7MQ6NDz2EJBe6FpBngtE/FcuSQF8uwxIklmZ1vzx84EOFJUP4XFkCy7nwj0VwAZZI33rrLfnqq69kypQppuV1LGHNmzdPLdcmBywHYnlNp1atWmoZEufUwTWNnDhxQh2HZTcduDzgd1evXk1wjbNnz6q6wc/XWO9ff/1VuQ4ABBVhCdLNzc3msqMtcW4sBernxBI+lnFx3gcPHqh2RaCFDnxBsZSZGKif8Td6uyQFljSdnZ9MM1haLleunOk7IqCx1Kv3BbgKoN4lSpRQS67r1q17KnnZ0s46SbWBJR4+fCjXr19XsjaC72gzI++8845aQkadjP7StvZ/4xjCcjGW8M2XnBMD7iRY/m/cuLFa5jevvzVwDdSxUaNGFvej/GhvyNFYhwsXLpiukdL+Yw7kXKRIkXhzSVJtgGujj6DNjPLBHGLu4qQD/064L7z33nupPtfBRcLYTnBNQlnQXrobFtwC4MaDcQyXKaC7odjSlhg7I0eOVPUcMmSIHD58ONH2wRxgPCdkiTFo3oefFtQTYK5+44031DwK1yLMm4sXL07Va5GMQ8aNZiDPFKMypit7+qSGmyB8Al9//fUEv7OmhOIGgXNCEdLBjScoKEj5R+JmgJsH/A6NvolbtmyRH3/8USIiIuL9NrkYb4opQffPQ/QuAtWMwAcRwP8tJefFzQEKvDnwX3zWmCvhkL2lbXpfgLxwQ4e/6oYNG5TvHBQvc9/h1Gzn1JJpUjRr1kzmzp2rfCbha2gsoy39P7F2swX4kMLfEm2B9oWCs3DhQuV3mxhJ9UOUH+PRkt+kpeDLp8FSG9ji351cNm7cqPxq9X6nXwM+8VDCIK+UznWI/oeiaQ4UVzzoQLHFB2MYYxYKLL4b/b6TAg8s+A1kjQcn+IiPHz9eevXqJemJrtjDL9Y4DuEXrivqhCQXKrIkVaOMoSwmFygvsIwgR6GtwNIwf/58dYPQLX6nT59WEyXKAesRbkRGPvjgAxVgM3DgwESVWFhNHj9+bLqB79q1S1lOEJhhDSjRiMbFDU+/eW3fvl1ZVBAEZA4mckzgmLx1i4slyw6CV6KioixaZS21N9py0aJFyloNi50l0Ea7d++W+vXrmwIwoPgblX5L9cNvjKBd0gKUGxZ2fJDuBwE9COyDZTm58sJvkmpnW9EzUxjbHGVFFDdkbTw/viPwz0i3bt2kbNmy8tprrykFQz8+Jf0/pSC4Bp9PPvlEWYhhDUtKkUUfRjAl0nEhsMgclB8PkLDq4bjE+g8CitKy/1gaE7g2rKBQEvWHFcgH8wasjpbAWEafMmY+QYDe1q1b41mEkwva6vjx41ZljTkLQZSwmOvzjXlAHupjHuhnqS3xewRU4QMLMwJeLSmyOB/mAMgHgbgAZUCfNCqcqYEeTIlz161bV23D/IZgNQR2EpIS6FpAUg3cxDAZYlK6c+eOzdaiwYMHq6VeWDkQpY/lLFiKBg0aZPU3UAqg3MB9AAosFANExyOvrH7zhdJg/OAmhiUz/J0YsHwgqho3HET6wnKFCG3jErk53bt3V0vZuFGcPHlSRU7jd1jOtfQ7lA+R0FAooKxiCXb//v3yww8/qO8A18TS9dtvv61uZogiRkSzvhyK9saSIb6jvXFDQIYGWI2QqQA3XVg3YSmDBUh3cUCb4UaJjA8oK8qeVH5M/B6R+N99950qB6zaiUXmp5QJEybIggULVLkgVyw3InI7McteYvKypZ1tBQ8HUJZR75s3byo3DYAMDGPHjlUPEJAForbhHoF2Ngf9A0u+yEShZ0ZISf9PLlDK0CboC4hohyIH5QxKjK3WXFj0Jk2apOSvtyGAxRxL28geAesfxv+OHTuU5VJXwtAWyHoAxRlyhYxQ19TG2piAZbtDhw4qzdOmTZuUHN5//33l6mIJKKvGuUPPWIH2Qj9IKXiIRttAFugjaEvMFXoGCFhloYyjbZH9AAqrec5WKKb4Hfod6okHevP81MgegOwOGP+QFepsTdbIgIH5Au4s6JN4MIRLBVYwsD054IEG9YJLj66Y4zvmaqBnFYH80VdQfszloG3btsm6FiEmku1VS4gVENlas2ZNFSSAroXgB0tO/wiW0ffrIKIcmQzwWwSOVK9eXQVpJAYCeBAwgqCvwoULW8xaYMTWYC8EiwwePFgFICBoCJkRjIEs1s6DKHgE+bi7u6toXARg6NHKxnPrIKAJUeAIgEI0NqLtES1sjC4+dOiQ9vLLL6uAJQTrIFAKUcgAUe+ITEYZjYEYiCpGtgZkTdDbBnV48OCB2o8yofxoZ2QeQPQzjk8s2EuPpkZQGGSECPXvvvsuyWAv83NaajsEhOkR8JA5opkRAIbyIRBq//79TyWvpNrZlkAenenTp6usCgjOQV0AgsoQWJMnTx51frSBMQjLUpDY+PHjlTz1AJuk+j9+j0wYRtD2CIAytqMxS4YRBPy9/fbbquzonwje6tmzpynAKKlgLzBlyhRTGyIQDYF5Og8fPlTfcV7sx3WQVQQR9joYn+iTkBHkhuwOyQ32MpYRoE2MtzFrYwLR+y+++KLKvIDgQPQRPXjKFpIT7GWUs6UgKWR80MuIfo6gMmOwIDJTIGsFxi6C5FasWJHgvMhCgKwCOAZzArIAGMsH2SLYDvvR399//30VwGWNe/fuqWPQvuiDGB961g1r9bAE+g2OM/8Y+ymCV5HVAtlmMAaQhQIZRwhJKU7454laS0jGBsFGsE7CWpnaYCkXLg3wkyT2Ly9HAkFhWG2A76ue35YQQjICdC0gJI2B/xmWvRHkgyh+QlIbLB0jgIxKLCEko0FFlpA0Bn55SO0EJRb+YYSkNs2bN1d+4oQQktGgawEhhBBCCHFIaJElhBBCCCEOCRVZQgghhBDikFCRJYQQQgghDgkVWUIIIYQQ4pBQkSWEEEIIIQ4JFVlCCCGEEOKQUJElhBBCCCEOCRVZQgghhBDikFCRJYQQQggh4oj8H0eDWBN1TkcNAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Trace the shapes through one multi-head pass, then draw the head split.\n", "hs = C // n_head\n", "print(f\"C = {C} -> {n_head} heads of head_size = {hs}\\n\")\n", "print(\"x @ Wq :\", (x @ Wq).shape, \" (B, T, C)\")\n", "qh = (x @ Wq).reshape(B, T, n_head, hs).transpose(0, 2, 1, 3)\n", "print(\"reshape + move heads up :\", qh.shape, \" (B, n_head, T, head_size)\")\n", "print(\"per-head attention q@k.T:\", (B, n_head, T, T), \" one T x T map per head\")\n", "print(\"glue heads back :\", (B, T, C), \" then a final @ Wo mixes them\")\n", "\n", "fig, ax = plt.subplots(figsize=(7, 1.5))\n", "colors = plt.cm.Set2(np.linspace(0, 1, n_head))\n", "for hi in range(n_head):\n", " ax.barh(0, hs, left=hi * hs, color=colors[hi], edgecolor=\"white\")\n", " ax.text(hi * hs + hs / 2, 0, f\"head {hi}\\n[{hi*hs}:{hi*hs+hs}]\", ha=\"center\", va=\"center\", fontsize=8)\n", "ax.set_xlim(0, C); ax.set_ylim(-0.5, 0.5); ax.set_yticks([])\n", "ax.set_xlabel(f\"the {C} projected dims per token, sliced into {n_head} heads of {hs}\")\n", "ax.set_title(\"multi-head = split the projection into parallel heads\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "m6mhac", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • x @ Wq: project every token, still (B, T, C) -- nothing split yet.
    • \n", "
  • .reshape(B, T, n_head, hs).transpose(...): cut the C numbers into n_head slices of head_size, and move the head axis up so each head is an independent (T, head_size) block.
    • \n", "
  • Each head then runs the exact attention from notebook 05 on its own slice, giving one T x T map per head.
    • \n", "
  • The outputs are glued back into (B, T, C) and passed through Wo, which lets the heads' findings mix.
    • \n", "
  • The coloured bar is the split: dims 0-15 are head 0, 16-31 head 1, and so on -- parallel heads, each watching for a different kind of relationship.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "70e6e3ec", "metadata": {}, "source": [ "## Feed-forward network\n", "\n", "Attention let the tokens talk to each other; now each token thinks on its own. The feed-forward network is a small two-layer MLP (multi-layer perceptron, from notebook 04) applied to every token independently. It **expands** the vector to width `4*C`, four times wider, giving it room to compute richer features; passes it through a **nonlinearity** (here GELU, a smooth on/off gate that roughly keeps positive values and suppresses negative ones); and then **projects it back** down to width `C`.\n", "\n", "That nonlinearity is essential. Without a bend in it, stacking layers would collapse into a single straight-line layer that cannot learn curves or rules. This sub-layer holds most of the model's numbers, and much of what we would loosely call its \"knowledge\" lives here." ] }, { "cell_type": "code", "execution_count": 7, "id": "c0e0c8cc", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.477985Z", "iopub.status.busy": "2026-07-01T09:08:58.477928Z", "iopub.status.idle": "2026-07-01T09:08:58.480332Z", "shell.execute_reply": "2026-07-01T09:08:58.480027Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "feed-forward output: (4, 16, 64)\n" ] } ], "source": [ "def gelu(x):\n", " return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))\n", "\n", "def feed_forward(x, W1, b1, W2, b2):\n", " return gelu(x @ W1 + b1) @ W2 + b2\n", "\n", "W1 = np.random.randn(C, 4*C)*0.02; b1 = np.zeros(4*C)\n", "W2 = np.random.randn(4*C, C)*0.02; b2 = np.zeros(C)\n", "print(\"feed-forward output:\", feed_forward(x, W1, b1, W2, b2).shape)" ] }, { "cell_type": "markdown", "id": "9b9838aa", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • def gelu(x):: the gentle on/off gate. The exact formula is a standard fast approximation; what matters is the shape: very negative inputs → near 0 (off), positive inputs → near x (passed through), with a smooth bend instead of a hard corner.
    • \n", "
  • def feed_forward(x, W1, b1, W2, b2):: the per-token \"think about it\" network.
    • \n", "
  • gelu(x @ W1 + b1) @ W2 + b2: the whole thing in one line: widen (64 → 256), bend with GELU, narrow back (256 → 64). b1/b2 are bias vectors added to every token by broadcasting.
    • \n", "
  • W1 = np.random.randn(C, 4*C)*0.02; b1 = np.zeros(4*C): weights start small-random, biases start at zero.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "m6gea", "metadata": {}, "source": [ "### See the GELU gate, and the width change\n", "\n", "The feed-forward network does two things worth seeing. Every token passes through GELU, the \"smooth on/off gate\" the text mentions -- so here it is, plotted against the blunt ReLU. And it briefly widens each token from `C` to `4*C` and back, which is where its extra room to compute lives." ] }, { "cell_type": "code", "execution_count": 8, "id": "m6geb", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.481216Z", "iopub.status.busy": "2026-07-01T09:08:58.481165Z", "iopub.status.idle": "2026-07-01T09:08:58.508811Z", "shell.execute_reply": "2026-07-01T09:08:58.508483Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgsAAAE8CAYAAABOyPJGAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAXMNJREFUeJztnQd4VMX6xr/0npCEhJBKIPTem4iIXRHshStYrzRFUUH432u7KhZULIgNQUGuXAvgVUAQkN57C81AEkgoSQjpdf/PO7ln2d1kl01IsiXv73lOsnvO7Dlz5pR55/u+mXHR6XQ6IYQQQggxg6u5DYQQQgghFAuEEEIIuSy0LBBCCCHEIhQLhBBCCLEIxQIhhBBCLEKxQAghhBCLUCwQQgghxCIUC4QQQgixCMUCIYQQQixCsdAAmDNnjri4uMj27dttnRWn4JprrlGLNTz88MPSrFkzaajMnTtX2rRpIx4eHtKoUSOLaV955RV1nxJiT5w4cULdl3iPNmQoFqwkKSlJxo0bJ61atRJfX1+1tGvXTsaOHSt79+6t8qVnbklPTze6CadNm2b2uKhobrvttiq3ofLnTWx7Tp8+ra757t27bZ0VuyIxMVGJpRYtWsiXX34pX3zxheTn56uy+vPPP22dPUKMmD9/vkyfPp2lYgZ3cxvIJX799Ve57777xN3dXYYPHy6dO3cWV1dX9TL8+eefZebMmUpMxMXFGRUb1vv7+1cqysu1sIh9s3z58kpi4dVXX1XCrkuXLkbbUEmWl5dLQwSCAOf+4YcfSkJCglp3/vx5VVbAWusMIfUlFvbv3y/PPPOM0Xq81wsKCpR1rCFDsXAZjh8/Lvfff7+6YVauXClNmzY12v7222/Lp59+qsSDKXfffbc0bty4dq8YsTmenp5Wp23IL5izZ8+q/xTH1pOXlyd+fn51dk1I9YH11tvbu8EXHd0Ql+Gdd95RD/Ds2bMrCQUAa8PTTz8tMTExDnUzZWVlSa9evSQ6OloOHz6s1hUVFcnLL7+sWoFeXl7qnCZOnKjWmzJv3jzp3r27+Pj4SEhIiBJUKSkpRmnQcuzQoYPs2LFD+vXrp9LGx8fLZ599Vml/H3/8sbRv3165d4KDg6VHjx5K6V+u5YoHecGCBTJlyhSJiIhQL9rbb7+9Ul7ADz/8oM8zRNzf/vY3OXXqlFEauIgeeeQRVS4oA1zzoUOHKpeR4XlprWLkoWfPnuozfqe5mjT/pmHMQklJiSorpDPl4sWL6oX0/PPP69dZez1WrFghV111laqUYclq3bq1Ko+aYM35AwhkXC+kiYyMVO64Cxcu6LfjnJF3EBYWpsoEZYHPANYFrazglqiKgQMHKiteVeAcb7zxRovnAjcd0uBaa/feo48+Wun+MXWJVOWjRt5Rtn/99ZfaJ+4znPdrr70mhhP3GroWP/jgA9XIwLFxLmi1GqLtEw2SW265RQICApTlEsAiA5M4yhj3RZMmTeTJJ59Uz211zhF8//336r7H/gMDA6Vjx47K2mMIrh1a1LjHcE1xz6EhZGoVs2ZfphiWCVxRcEvhGHhutm3bVik9LLZoaOFZwbnjXfDLL79USgf3L8oV54379fXXX1fvaRzL8H5dvHix3Hrrrep64bg4/r/+9S8pKyvTp8Hz/Ntvv8nJkyf196X23JreD9OmTVPfkdaUyZMnq8aE4XXasmWL3HTTTRIUFKTeb8jzhg0bjH6Xk5Ojyh/HRB7Dw8Pl+uuvl507d4q9QMuCFS4IPDi9e/euduFmZmZWLnB3d5u3tGAKxo2I/K1Zs0Y9PHgpoJJdv369/P3vf5e2bdvKvn371AvvyJEjsmjRIv3v33jjDfnnP/8p9957rzz++ONy7tw5VdlfffXVsmvXLqPzw0ODFyHSPvDAA/Kf//xHRo8erR4o7aUGUz0EF14Q48ePl8LCQvUiwEP24IMPXvZ8kB88vJMmTVKtWbxkr7vuOhVDgBcJwIOOShAvqKlTp8qZM2fUSw4PrWGe77rrLjlw4IA89dRT6sHF/lAZJycnVxmoiHJChfHSSy+pchswYIBaD3FUlZXhjjvuUK6rzz//3MhCgfKFCIDoAtZeD+QVMS2dOnVS+cCL5tixY5VeRtZizfmjckdljzLGtYTYhMsNL34cF+eJa/Dtt9/KwoUL9e44VCx9+vRRv0E53HnnnWp/yHtVPPTQQ/LEE0+oShaiUwPHQRn84x//MHseyPcNN9ygxMmLL76ori9e+ij7moLKBS99nAMaEcuWLVOCqLS0VJW9ITh3VAAQUbifca9de+216hqi4tfAb1HZQ+yhEkJlAiAMtHsWzwbcnJ988om6V7UytuYcce3w3A0ePFhV/uDQoUNqH3jWAOJIUIFBOOO4sbGxsnHjRlXxpaWl6f341uzLEhD/KBMcA88ryhD3AASYZoHDvde/f3+JiopS5wRRhnfGsGHD5KefflL3DUBeBw0apPaDfCLdV199pe5/U1COuP8mTJig/q9atUo9rxDo7777rkrzf//3f5KdnS2pqanqGQNVuZAB3mUQ7cjXCy+8IIZgHa4JGjwAx7r55puVwMK9Ags0BA3uhXXr1qkGGxg1apT8+OOPKi4OsXAZGRnq2Uf5duvWTewCHTFLdnY2mgy6YcOGVdqWlZWlO3funH7Jz8/Xb3v55ZfV76paWrdurU+XlJSk1r377rtm8xAXF6e79dZbq9y2bds29fvZs2dbvIrYjnRIn5aWpmvfvr2uefPmuhMnTujTzJ07V+fq6qpbt26d0W8/++wz9dsNGzao7/iNm5ub7o033jBKt2/fPp27u7vR+oEDB6rfvvfee/p1RUVFui5duujCw8N1xcXFat3QoUNVnqrL6tWr1f6joqJ0Fy9e1K//z3/+o9Z/+OGH6juOg+N16NBBV1BQoE/366+/qnQvvfSS/ppe7npo54XFmuswcuRIdQ01fv/9d5X2v//9r1G6W265RV2T6l6PDz74QH3HPXilWHP+Z8+e1Xl6eupuuOEGXVlZmX79J598on779ddfV3oODPOGz1iHbaZo6TUuXLig8/b21k2aNMko3dNPP63z8/PT5ebmms3nwoUL9ff85e4f/DdEey4NryeuI9Y99dRT+nXl5eXq2UR5aOeo/dbHx0eXmpqqT7tlyxa1/tlnn620zxdffNHo+LjmWP/dd98ZrV+2bJnRemvOcfz48brAwEBdaWmp2TT/+te/VHkeOXLEaD3yhWc9OTnZ6n1VhVYmoaGhuszMTP36xYsXV3oWBg8erOvYsaOusLDQqJz79euna9mypX4droOLi4tu165d+nUZGRm6kJAQtU8cU8Pw3azx5JNP6nx9fY2Og2tp+Kxauh/69u2r6969u1G6rVu3qnTffvutPt/I84033qg+G+YnPj5ed/311+vXBQUF6caOHauzZ+iGsACUpzmFCbMVFL22zJgxo1IaKGGoccMFqtJWQDWjBQFz+Nq1a40CMmGiR+sV3dxgedAWKGCwevVq9R+tFrR6oa4N08EF0LJlS306Q0sKWhIaaE3jO1pFcE8AtIiQt6pMktYwYsQIZRbVgIUC5vMlS5boTbU43pgxY4x8jzBN4nxhfgSwQiB/MEubmntrC5QnTMZwnWjgWLg3EERb3euhWURgar3SQEprzv+PP/6Q4uJiZTI1jNOBBQBmaa0sawOYbeEC+fe//6039aN1j7JDS9OSb18rF1gGcb/XFmj5aaBVi+8oD5SLIcgfWscaaEHCOqndk4bA0mIIrj3OHdY/w2uP1ineRabX3tI5Ig3cqLi/zIHjwSKG1rDh8WA5QnnjXWHtviyB+1trcQPNCgfLAoClEy1xvFtggdDygVY2rC9Hjx7Vuw1h1enbt69RQDHcFpobxxDNugi0/eLYsKjA5VHTc9mxY4dyIWngvoRlA/csgGUTeYZ1FOegnQ/KENYZlKv2zKJsYUlFsLS9QrFgAa0Cys3NrbQNZmQ8NPDdmwNmeTxwhgtu8NrG2r7pMOui0oTrwfBFBnBTwwRoKICwoKuoYbAa0uHFDWFgmhYmMy2dBvyEpi91bZ+aXxHuA7wE8ULFfmG6rY4ZHb8xLQ+4jrT9a75F+LlNQWWsbceDDvPq0qVLlakY1w+mUq2ra20A8QRTPyp3LfYAAgwve0OxYO31wG9gtoU7CHmGGwOm0JoIB2vO31xZQmQ0b968Sj/ulQAhCBcITLYAlTJcSLiXLQFRjHKGuwTiDC9wCPWq4m+sBeII52jpXjZ3T2ppTdPhfoC/3RBce5jE4bc2vf54F2nX3ppzhEDGcWEKx3Hg+kNFa3o8rDM9Ft5XQDueNfuyBNwbhmjCQROmcJ/h3QIXp2letPgXLS+4z7QeNoZUtQ7PEdwXEGAQtNgf4pUAyrkm3HPPPep+0EQ/8g3RhbLBMbRyBSNHjqx0PnCZ4Dppx8dzBncbYkbwHoSrTxNR9gJjFiyAmwstVNPAJKDFMJg+/LUNWsLotlMVUMZaGmuAfxC+VPhP4bc3BJULfMrvv/9+lb/VAjiRDpUxKhQ3N7dK6cz5+SyBFjT83mgh4eUDiwwC6OBX1LrZ1RdoMQ8ZMkTFBPz+++/qxYWyQouna9eutXIMVOgQmyhDtEBRuUO0GAbzWXs90GpCCwWtTbTqUX54gcECgS6eVV0jW59/dUCLEsIFohziBf9hxdIqMnPgHoUPePPmzfLf//5XnQsqt/fee0+tw31qTmQbBr7VNRBopj2pcO0hFL777rsqf6MFiVpzjtgPWrjYhvsNCwQFRNg333yjPx6sGPDDV4UmiKzZlyXM3Yua1UgTuAjyNRe8WpUYsAQCNyGqUIEjrgTxWXhfInAQjZSaWuMiIyOVdQLPLoKJUd4QtVosh+H5IC7CtEu16fsS1hTsDzE+eG7xG+wLDQkIELvA1n4Qe+fxxx9Xfij4HKuiqriDqny11v7WFPiGo6Ojq9z2xRdfqN9v2rTJ6piF1157TX2eOnVqJZ85fP+GvrWqeOedd9TvDx8+rLsc8OsjjsHUtzxz5kyL+UZcA/yH8JcaxhiY8zlPnjzZaD3OoWnTpspXCDZu3KjSffrpp5X20bZt20q+R0Pgx4Vvc/jw4WZjFrZv3251zAKArx/5u//++9U9gjIy9eFbez2qAnEjyM+KFSt0V4rp+c+fP1/te8mSJZWuGfyud911l8Xn4Pz581bHLGjAzx8cHKz83f7+/kZ+/+oAXz/2/+WXX6rve/bsUd/h+zdk5cqVZmMWTO/7pUuXqvX//ve/jZ7pBx54oNLxe/fubRSzhH0iVsCUMWPGqHu/Kl97dc/RFNx78NcjzdGjR9W6du3aKR98dalqX9V9zxneC2fOnKnyea4KxAIgjsEUxDIYxixocR1r1qyp8t1pGK9y2223WR2zAPA+wfrExEQVz4HnxPBdp8UwfP7557rqgrLA89+/f3+dvUA3xGWA2kaEMhQ7zJ9ViC2pS9CTAP58w94IACYsmLKg9qsTLYuWIpQ7IogRpa4BZQt/IHommALLBvxsmnUCLQS0+E3PHd/hmzME0d5oRWvAv4vvaB3BBwtMfwOTNiKCsT9r/M1a5LkGWluI4tYUObpeoZzQZdPQRIuWEVwniF3QLDWIXDcELRG4oyyZrzU3i2HXQUugJYm4CrQGMRwyysjQBVGd61FVjxutFVNdk7s1548WPa7PRx99ZHT9Z82apUyqWlmaQ4v2t7asAFwOMFUj1gVmeM2EbAmkN70/TcsFMTu4lzWfvAasWuZAjwQN7B/fEckPH7QheF4Nu+Vu3bpV+aStaSXi2sO6ge59puBe0crOmnM0fbZw72m9T7Q0ON6mTZuUxcAUHAvHtHZfVwKeUcSC4f2A59cU9LrSgOUBeTYcNRXPgqk1RrNmGJYT3kFVXWM8x9VxS9x1111q/4ipgQsCvZIMXa54v+H5QS+XqlzZ2vngWpseF2UB60VtlGttQTfEZYDvEV1+0GUIflptBEfcfOjOhG14aEz9jlqlVZVZHiY/w+5TGOzJ9CUNYKJGt7mvv/5a+cggWGAKxkMLUzPcI6goqzNIEICJCzcnYgNQEeDlixcyTGrowgOTNvzguIkRAIT1eJGg0sXNj/7MEBtwwSCP2AfKAiY05NdwrADc8DCnIS3Mmcg3HnD0t9a6S6GrEUzLOCbKBRU4XsKoeAwDF82BwCZ0PUM3Mwg6dPWCuRJBdwDHQR6wHSZJXEut6yS6Az777LMqHbrj4aWPlyfECvzJOCek1bo0VgXKBAFKECPIL14YcFOhz7s5IA7Q3RS+WLgb4IoxxNrrAdMqKjuUFSo/+HTxIsT9iDLRwEsYsSqWxK015w+Rh2sPsYhuhOjeCRcSjoluqZeryOE2wb5xH+B+wLVDt0jDrpGm4J7Hdi3o0xpxDLM48gRfNa4PxCSEF8zREOCamxHPFa4DTPpIB1eYadyNBszXcPPAB43rC7EJ1w/M0JprQAP3H8ofwYt44eOeDA0NNWvqNwT3KIQR3D94VvB84B6GDxxlgPsWYtOac0QsCypRuKVwT8DXj/OFqNDuOXT/wzgGqOww9gMqOYhRdPPEOwzPLmIirNnXlYJAcZQbngk8v4gRwf0HYYBG0549e1Q6lCNcUniXopuv1nUScRHIo+ZiQhdmxEbgmqELKtZDoFf1HOC8cV+iiyXuZby74ZIzR3h4uOq+CVchyt5U8KNeQJ4gEDFeBt4/iBWDiMQzjeuEBgN+i/LENUXdguMiNgcB33Ap2Q22Nm04CseOHdONHj1al5CQoLpzoWtUmzZtdKNGjdLt3r3bKK2lrpOGpi/NvGVuQfc5rUsbTK/obuPh4aG6Lw0aNEiZQK3B0A1haEKEqRQm8EWLFum7GL799tuqG6OXl5cy/cJE/+qrr6pupIb89NNPuquuukqZUbGgLND1x9BMC1M99gUzPcycKDeY+dDNzhCY6a6++mrVtQrHbdGihe6FF16odExzbgiYgGG6RPdIXBe4ME6ePFkp/YIFC3Rdu3ZVx0AXK5jWDbu3wUSOc8C54JxgVofpGF0xDTF1Q2jdwGDORXkamiyrckMAuBdiYmJU2tdff73K87PmesBkjq6nkZGRqgsf/uO6mnaDw+8iIiIslqe15w9wDZEO92OTJk3Us4H71BBz7ji4hZAf5NfQDG3ODWHo/nrzzTd11rBz505VDrGxsarscG/AzIx70RDkDa4TmJBRvjCr79+/v0o3BMrk+PHjyjWI9Dhv5NmwC6mhyR1dhnGNcfwBAwYot4ch5twQhqZylBPu6YCAANWlcOLEibrTp09bfY4//vijyi+2obyRFueILtSG5OTkqGcI7zeka9y4sTLzT5s2Td/F2dp91dQNoYEyHjFihLpfcX/BHI/zwvENQbdJlCvOHa5auFY/+ugjtc/09HR9OnQz7tOnjypHPB8oQ60Ls6EbAi6EBx98UNeoUSO1TXtuzbkhANw92IbrY85linzeeeed+vcb9nvvvfeqZ1dz4eF917lzZ7Uf3BP4XJXb1JZQLJA6QxMLdYUmFn744Yc6O4YzgDEoIGJMRZojMX36dNWvvioRWB9crmKvThwSqTsQO4BGSXXHgiCXhzELhDg5cFPA/Km5ZRwNNGoQEwHzvGn3O9JwMe0lBvcsXAxwY1S3FxC5PIxZIMTJQTzD5QIP7RH4zeFLh38X/nOMTUGIBsasQSwO4iUQ1wBBiYH0EMRNah+KBUKIXYJocYx+h+BRBBEimJIQDQRxIgATwdIIXETgKwQDxuMgtY8LfBF1sF9CCCGEOAmMWSCEEEKIRSgWCCGEEOK8MQsYexuzdGEgHGsnUyKEEEKIqJ5GGBQKg+eZzlHiVGIBQkGbUIcQQggh1SclJaXKUYidRixoQwHjRLVpQQkh9gWGwdWm8iWE2A/oaooGtzXD6ju0WNBcDxAKFAuE2CeY14DPJyH2izVufAY4EkIIIcQiFAuEEEIIsQjFAiGEEEIcI2bhrbfeksmTJ8v48ePV3O+12TWktLRUysrKam2fhDgbmHjH3d2dXZAJIfYrFrZt2yaff/65dOrUqVb3W1xcLGlpaZKfn1+r+yXEGfH19ZWmTZuKp6enrbNCCLEzbC4WcnNzZfjw4fLll1/K66+/XqsDNiUlJakWEwacwAuQAzcRUrX1DcIaEzfhmWnZsuVlB2ghhNQvhw7lSX5+uXTvfvlujk4pFsaOHaumz73uuusuKxaKiorUYthH1Bx4+UEwoA8pWkyEEPP4+PioLo4nT55Uz463tzeLixA7YdeubJk376hkZPjLc89FS8eO/vWeB5s2H77//nvZuXOnTJ061ar0SBcUFKRfrBm9kS0kQqyDzwoh9seWLZmyefMG6dUrSUJDs2X16gs2yYfNxAJGXUQw43fffWd1KwYBkNnZ2foF+yCEEEKckXXrzsvu3RulceM8KStzlYSEABkzJqphuSF27NghZ8+elW7duunXocfC2rVr5ZNPPlHuBsQbGOLl5aUWQgghxJlZufKMJCVtlZCQQikudpP8/Pby97+3Ejc3l4ZlWRg8eLDs27dPdu/erV969Oihgh3x2VQoEPsjIyNDwsPD5cSJEzbNR7NmzSx2t3344Ydl2LBhYguWLVsmXbp0UfEzhBBiDUuXnpGTJ7dIUFChFBa6SWlpJ3nsMdsJBZuKBUxc0aFDB6PFz89PQkND1eeGTnp6unLTJCQkKDdNkyZNpH///jJz5kyjrqCoKNHLw3TBuBUAFTm+Q4CZ8ueff6ptFy5cqHYFDN544w0ZOnSoSkuq5qabblKBg3C3EULI5XomLVp0Xr77LkMyMnyloMBd3Ny6yogRLcTV1XZCwS56Q5DK/PXXX0oYNGrUSN58803p2LGjcr/AEvPFF19IVFSU3H777fr0r732mjzxxBNG+7BmFrErAYJl1qxZ8vvvvzfISwiXGYSWNUGBsGx89NFH8tBDD9VL3gghjikUFiw4J7/+moGpnWTPnki5445AufPOKLvo9m9XnanR0q3N0RsdlTFjxqjR9LZv3y733nuvtG3bVpo3b65a8b/99psMGTKkkjCIiIgwWmClqUuWLFmiBEyfPn306+bMmaMEjiGLFi0yutH37NkjgwYNUnnGTITdu3dX56mxfv16GTBggOrKh94uTz/9tOTl5em3I84F54/t8fHx1WqxT5s2TQ06BOsVuuyWlJTotyFG5vnnn1dCDGXXu3dvdT+antsvv/wi7dq1U+eenJx82d8B5BfnePz4cavzSghpOJSX62Tu3L8kJWWvuLjo1LoHHmgid90VbRdCoUFaFv75zyS5cKG0Xo/ZqJG7/Otf8VbHASxfvlxZFMxV+PZw86xbt05V9NUFMSldu3ZV7hTEpcA9AjM9QGUKsz3G2/j666/VIEHjxo1Ty+zZs/Wt9NOnT8vq1avV7yAmICAuB9JDKOD/sWPH5L777lOxBJpFBsc4ePCg6s6LQbwWLlyo8gJrDgYp0qwpb7/9tnz11VdKcCBew5rfxcbGKjcSyqxFixbVLjNCiHMLhdmzj4i39wFp1qxMiorcpU+fznLddcFiTzQ4sQChkJVVv2KhOqAigzmqdevWRusbN24shYWF6jNaxai0NCZNmiT/+Mc/jNIvXbpUtdDrCgzeg8qxuqA1/sILL0ibNm3Ud61C1cbRgJh45pln9Ntgvh84cKASF/gtzmvr1q3Ss2dPlQauEFheLkdwcLDqZQOBgmNjILCVK1cqsYD9Qozgv3ZOsBYgOBHrIdwALBGffvqpdO7cWX8u1vwOYDvKjBBCNEpLdTJrVqIEBBwUD49yycz0kQED2svAgfYlFBqkWEAr3xGPiQoSEfWoTA1HsQSofNHiNgRm8bqkoKCgRqP8TZgwQR5//HGZO3euGrXznnvu0be24aLYu3evkWsBwkkbuvvIkSPKPWNo0UDFb+r6qIr27dsb9bCBlQGtf4D/iEFo1aqV0W9QzrAgaGDIcMP5S6z9HYDbhHOUEEI0iovL5csvD0hIyGFxd9fJ+fN+0rVrH+nTJ0TskQYnFqx1B9gK9H6Am+Hw4cNG6xGzoFU6psDqgN9VF8QMAAxwZVrhoocERsk0B46ZlZVltA7BfqjcDTGMCwCvvPKKPPjggyr2AlaCl19+WZnw77jjDjVPyJNPPqlcC6bAlA+xUFM0V4cGyljrzojjQkhg7A/TLrv+/peGVUXZG7qArP0dyMzMlLCwsBrnnxDiPBQWlssXX+yV8PBj4uamk7NnA6Rv377SrZv5d66taXBiwd5Bi/T6669XJvOnnnqqTgMVtQmDUNnFxcUZ9caAgDBtMRuCuIN58+YZrUNlmJOTowIStXxX1WUT+8Xy7LPPygMPPKBM9hALGKAL/n9zwgdWBEw3jvxqbgiIqqq6flYHnAssBIh9qI7rxtrfwX2EeAykJ4Q0bPLzy+T995OkefPjSiikpwfKoEH9pEMH20wQ5ZC9IUgF8IujUsQgVQsWLJBDhw6pShGVc2JiYqVWLCpojMtguJhOsoXfGw6AhQVuBLgEnnvuORXlD1M/RtCEqwO9HPr162f2ktx4441y4MABI+sCegJg0q4pU6aoynH+/PmqF4Gh6wIBgegtAP/9hg0b1PTkWswBYi82btyo0iB/R48elcWLF6vvAHEcCB6E9WHLli1KNCD/VVlbqgOEC855xIgR8vPPP6tygNsHMRSwgFzp7zZv3qx6T6DlQAhpuOTklMrUqcmSmFgiW7fGSmpqiNx4Y3+7FwoKnQOTnZ0Nm7f6b0pBQYHu4MGD6r8jcvr0ad24ceN08fHxOg8PD52/v7+uV69eunfffVeXl5enTxcXF6fKwHR58skn1fakpKQqt2NJSUlR5fPyyy/r2rRpo/Px8VHH+/vf/647d+7cZfOI/Hz22WdG6xYuXKhLSEhQ+7rtttt0X3zxhToWKCoq0t1///26mJgYnaenpy4yMlKdo+E12rp1q+76669X5+vn56fr1KmT7o033tBvT0tL09166606Ly8vXWxsrO7bb79VZfDBBx+YzefIkSN1Q4cONVo3fvx43cCBA/Xfi4uLdS+99JKuWbNmqrybNm2qu+OOO3R79+5V22fPnq0LCgqqtO/L/Q6gPLXrYc/U1TMzZMiQWt0fIY5IVlaJbsqUw7rhww+qZdSow7qkpAK7rUNNccEfcVDQeoZfHSZzzf9uaPpFSw998Tndbt2A1jOCK/fv388ZC81w/vx5ZRHBOAu4F+2ZunpmMIAYLFeENFTOnSuWb7/dKk2anJX16+PF3d1fJk+OlagoL7utQ01hzAKpMeh+CFfBqVOnrJouvCGC4bbhVrJ3oUAIqRvS0ork+++3SGzsGfW9efMCGTmyvTRp4ulQRU6xQK4IbUwEUjWIO8FCCGl4pKYWyo8/bpLo6PPq+8mTUfLYYz0lNNS4d5YjQLFACCGE1DJJSQXyyy8bJCoqS+DsT06Olccf726TsX5qA8fMNSGEEGKnHD2aJ8uWbZDIyGwlFFJS4uXJJ7tIQIDjVrnsOkkIIYTUEocO5cm0acni4VEi5eUukpraQkaP7urQQgE4du4JIYQQO2Hv3lz54INUwcC1mzfHSefOOhkzpr14ezt+u5xigRBCCLlCtm7NlF9+OSwlJRVD53foECRjx0aJp6fjCwVAsUAIIYRcAevWnZPExM3SuXOhuLiUSVhYMxkzJkrc3S/NJePoUCwQQgghNWTVqjOSlLRFGjUqkqIiN4mNDZdHH40SNzfnEQrAOewjpF7JyMiQ8PBwNeCQBuZ56Nixo5rdcdiwYbwil+Gaa66p1zEqPvvsMxkyZEi9HY+QhsCyZafk5MnNEhhYJAUF7lJa2lkee6yl0wkFQLFghzz88MNqKmQsqHwx+t/EiRPVcLzWgoocv69q1kdM5IRtVc3W2KxZM5k+fbrFfb/xxhsydOhQlVZjwoQJ0qVLFzVcsOHkUaRqMPHUv/71r3ornkcffVR27twp69at4yUhpBZYvDhZzpzZJv7+xZKX5yGurl1l5Mjm4urqfEIBUCzYKZhdMS0tTU0X/cEHH8jnn38uL7/8sq2zJfn5+TJr1ix57LHHjNZjlslrr71WoqOjpVGjigCf6lJcXCzOjnaOISEhEhBQfzPNeXp6yoMPPigfffRRvR2TEGdEp9PJDz+kycWLO8TXt0RycjzFz6+7DB8epxphzkqDFAuY/tncUlZWVqtpawqmNI6IiFBzLsCsf91118mKFSv028vLy9VUyLA6YIrmzp07y48//ih1zZIlS1TeMIW1oQUDrgm0XvFZsyysWbNGevXqpdI3bdpUXnzxRaMygSke00/DHN+4cWM17bU5vv76a2nfvr1+X9q01SA5OVlZOvz9/dVkKPfee6+cOVMxDjt45ZVXlNUD+4iNjVXpxowZo67fO++8o8oZbhVYTAzBucycOVNuvvlmVcbNmzevVMaYVhtTVWNqbmz/5z//KSXoN2Vy7K+++spogiZTNwTmj2jZsqXa3qRJE7n77rv124qKiuTpp59WecT2q666Sk3tbWopWrlypRpaGnnB9OKYltwQuCEwoROmCieE1EwozJ9/VhYvviBHjzaW7GxvCQ3tKffcE+PUQqHBBjguXLjQ7DZUHAMGDNB/x8vVVBRohIWFqZe+4SyMpq3je+6554rzi1kdN27cKHFxcfp1EArz5s1TvmhUMmvXrpW//e1vKk8DBw6UugJm7O7du+u/Q8zAAoKZFV977TW577771CxmmFzqlltuUS6Vb7/9VhITE+WJJ55QlR0qUI1vvvlGRo8erWIezIEKG26Ot956S1XcmCFNSw/RpAkFiBOIkbFjx6p8oBI1tHwsXbpUli1bpj6jMobVBhU9fofyhdiBKOvdu7f+d6j8cdwPP/xQ5s6dK/fff7/s27dP2rZtq7bDOgBxFBkZqdbjHLEObiONY8eOyU8//aRcD25ubpXODzNSQgxg/6jkMzMzjdwF2Bd+j7LCPQCBA2GF/cJCofF///d/8t5776l7YNSoUep8DMsVQgLls2XLFqP7lhByecrLdfLNN2mycmW2+v7XX42lf/82cuONYQ2i+BqkWHAEfv31V1UB4uWOlqWrq6t88sknahu+v/nmm/LHH39I37591Tq0atevX6/cFXUpFk6ePKkqRg1UfhBYUNUQCfistZQhJJBnbGvTpo2cPn1atcRfeukl/ZTWEDqo/Czx+uuvy3PPPSfjx4/Xr+vZs6f6j9Y0KmnESmgzX0KcwAqB1reWDqIClgVU5O3atZNBgwapljcsJcgLxM7bb78tq1evNhILEHuPP/64+owYA1h3Pv74Y3V+4B//+Ic+LWI4nn/+efn++++NxAIEJPKESrwqYBnx8/OT2267TeUPgqBr165qW15enhJLECQQSuDLL79U+YA7CFOEa8Ayol17WHEwKyjiXDRrBiwOuEa4hoQQ6ykr08mcOYekpCRJ3N3jpKzMTR57rKlcc03NXK6OSIMUC3fccYfZbaampNtvv93qtHg51xaozFBJoLJAzIK7u7vcddddahtalIgduP76641+g0pJq2TqCpiwtcrHEocOHVJCxrCM+vfvL7m5uZKamqrcAcDQSlEVZ8+eVSJj8ODBZo8DkWA4RTbEAOImsE0TC6jIDWMEYOqH0NFEi7YOxzNEE2OG3w2DRhcsWKDiAGCtwLlB3JnOC4/K35xQALiOSAPBh1gVLLhHUbljv3BroOw0EPQK9w7Oz5BOnTrpP8NVo5WfVtYA7hTcO4QQ6ygt1cmsWfskKOiIuLnppHXrczJwYDfp1y+oQRVhgxQLqHhtnfZyoKWZkJCgPqNFjJgELbAQlZLm9oiKijL6HXz6l0OrzGDONw1GRA8JtD7NgdiCrKwsqc3ztAQqt9oAFawhWk8T03WwQFjLpk2bZPjw4fLqq68qtwDKDVYFuAKqc44QMeipALfJ8uXLleUFrhrDuARrMDwfTaSZng9cHJaECyHkEsXF5fLVV3skNPSYoF2Rnh4o113XTXr2bFhCocEGODoaaP1OmTJFmbzRskfLGaIA5msICsPFsIVtDpj+sc8dO3YYrYcPHwICfnxzwHJx8ODByx4DPn1UpggI0oD/HBUjekxYC9LDKgB3g7njpKSkqEUD+YPoQTldKZs3b670XYtX0OJIECuAeACUa01N/BCaiJeAS2bv3r0qcHTVqlXSokUL1ZPBMPYAlgYIieqeH6wUcEvUtfWJEGegsLBcvvhipzRuXCEUTp1qJFdf3bdBCoUGa1lwROA7h396xowZyi+O5dlnn1UtR0THa0F/sBqMHDlS/zvTiHgAfz788IgDQCWFwZRQ2SKeAL0cEGRnDrSgJ0+erKwLwcHBZtOhtwHGa3jqqadUzwXkA10/EahoaPq3BrSyEbCH3gDw2+fk5Khzxb5RwSL/aOHjeHAD4Njw3aMCv1J++OEHtR+U8XfffSdbt25VFh4AcQDBBmsC3B2w9FgKnrUUnwKhdvXVV6syRRwFriviKGCVQAAorj2CGeFSgKCAK8G0++rlQNAkXB0QIIQQ8+Tnl8msWdslIiJZfU9JCZGbbuot7dr5N9hio1hwEFCpo9JFRYHKA8F2MCejVwQqGrgTunXrpiwQhiB63xQIA0T3I8ofAgGtYQQmwneOIDlLXYBQMeM4//nPf+TJJ580mw7uEVR6qOTgQkFFh8rNMCDQWiB+0CJG7AZEElwhWtdC5HXx4sVKOKCyhRCBzx9BiLUBXAwQAxAgiAP497//rW/RI54Fgg3XBUGniFlB7wnD3h7WgGuHnhL4Hc4TIgTHgagDuE4QDw899JASShAvv//+u0WxVhXYJ3prEELMk5NTKtOmnZAWLU6r7ydPNpahQ3tLy5a+DbrYXHSGdmIH4+LFi8pPjFa1aVAZXrqIkDfs205qB7SgIQLQpbO6VgJHAkIElgJnGL76wIEDatCsI0eOmI1JqatnBqIKXZAJsXcuXCiVt95KltTUIvH3L5K4uFy5//7uEh9fO7FTjlSHmkLLAqk2aEEfPXpUjaVgTYwEsT0YCwPdNy0FrxLSkDl/vlg+/PCIpKZWjIXi7u4nI0a0laioyweNNwQoFkiNqM9JkMiVg9gOQkjVnDlTJD/8sEHat8+U3Nw4KS9vJJMnx0lEhCeL7H9QLBBiBgf20BFCrCQ1tUAWL94gTZpkCR758PByeeKJZtK4sXHX6oYOxQIhhJAGyYkTebJkyXoJD78oGJIkKSleRo3qLMHBFAoNTiywdUgInxVCTDl6NFdWrlwnYWG5UlbmIidPNpcxYzpJYKDTV4s1wmlD2bXR7Di0LSHWoT0rpiNbEuJsHDx4UVatWiuhoRii3VVSUlrJuHGdKRQs4LQSCuP+o/+6NtY/xtl39ilECamp9Q1CAc8KnpmqZsYkxFnYty9XPvjglHTo4C4BAa6SltZaxo1rKz4+vO8bpFgA2gyIppMDEUIqA6GgPTOEOCM7d+bIRx+dktJSkV27oqVbNzd56qmW4unptEb2WsOpxQIsCRh1D8MEYzx9QkjVwPVAiwJxZtavPyPr1h2U0tImqB2ke/dAGTs2Ujw8KBSkoYsFDbwE+SIkhJCGyapVpyUlZaskJJRIebmLBAe3lCefjBQ3N7qmraVBiAVCCCENk+XLU+TMme3i61squbmeEhHRTB55JFJcXSkUqoNN7S8zZ86UTp3QVSVQLX379pWlS5faMkuEEEKchN9+Oylnz24Tb+9SuXjRS9zcusqjj8ZRKDiaWIiOjlYz6u3YsUO2b9+uJroZOnSomvSGEEIIqSmLFh2X7Ozt4uVVJhcueIu/f3d56KEY9opzRDfEkCFDjL5jemRYGzZv3qyfnpcQQgipTlfgn35Kl8LCPeLlVS4ZGb4SFtZN7rwzgkLBGWIWysrK5IcffpC8vDzljqiKoqIitRhOr0kIIYRoQmHBgnPy668XpEmTKGnWLFNiY7vJ0KHoAUGuBJv3Gdm3b5/4+/uLl5eXjBo1ShYuXCjt2rWrMu3UqVPVFLvawumRCSGEaEJh3rzT8uuvGer7mTOBkpDQi0LBWcRC69atZffu3bJlyxYZPXq0jBw5Ug4ePFhl2smTJ0t2drZ+SUlJqff8EkIIsS/KyyEU9kh5+Tbx9S1W6x55JEJuvjnU1llzGmzuhvD09JSEhAT1uXv37rJt2zb58MMP5fPPP6+UFtYHLIQQQogmFL79dqf4+f2lvsfFZclVV3WRgQMbsYCcSSyYUl5ebhSXQAghhFRFWZlOvvlmmwQEnFTf//orVK65pov070+h4FRiAW6Fm2++WWJjYyUnJ0fmz58vf/75p/z++++2zBYhhBA7p6SkXObO3SIBAanq+9GjjWXw4G7Su3eQrbPmlNhULGCCpxEjRkhaWpoKWMQATRAK119/vS2zRQghxI4pKiqTuXM3SVBQmvqemNhEbrutq3TtGmDrrDktNhULs2bNsuXhCSGEOBhFReUyfXqyhIRkqe8HDzaVO+/sIh07+ts6a06N3cUsEEIIIVVRWFgu06alSGJioXh6NpOmTfNk+PCO0q6dHwusjqFYIIQQYvfk5pbI558fkMTEih5xbm6eMnJkC2nVytfWWWsQUCwQQgixa7Kzi+XHH1dLbOxFycyMlPPnG8vEibHSooWPrbPWYKBYIIQQYrdkZhbIokV/SmBgrpSWuohO5y1TpsRJXJy3rbPWoKBYIIQQYpecPZsnv/22RgIC8qSkxFX274+XMWPaS3Q0B+erbygWCCGE2B1pabny++9/ir9/gRQXu8n+/S3k6afbSNOmFAq2gGKBEEKIXZGeni/Ll68WP79CKSx0l0OHEuTZZ1tLeLinrbPWYLH5RFKEEEKIxtmzxTJ16ik5eTJQ8vM9JDGxpUyY0IZCwcbQskAIIcQuSE8vljffPCmZmaWSkREmBQURMnFicwkO9rB11ho8FAuEEEJszrFjZ2XFit2SnR2JURQkKspLJk2Kk0aNWE3ZA7wKhBBCbMrhw+myY8dGady4TNq1c5OsrHiZPDlWgoJYRdkLvBKEEEJsxsGDp2XPnk3i4VEumZk+kpsbI//3f7ESEMDqyZ7g1SCEEGIT9u1LlgMHtoq7u07OnfOTtLSWMnFiPIWCHUKxQAghpN7ZtStJjhzZIW5uOklPD5Bz51rKiy82Ez8/N14NO4RdJwkhhNQrx47lyoEDu8XVVSenTgVKRkZLmTSJQsGeoWWBEEJIvfHXXwXyzjunxN0d8ztkSWFhczUplK8vLQr2DMUCIYSQeuHgwUyZPv2c5OeXi4iPFBWFyMSJMeLjQ6Fg71AsEEIIqVN0Op2sX79XUlOPird3M8nP95M2bXzl+edjxNub3nBHgGKBEEJInQqFNWt2yblzx8XdXSQkJF8iIsJkwgQKBUeCYoEQQkidCYVVq7ZJZuZJ9X3//gjx8YmTZ5+NFi8vWhQcCYoFQgghtU55ebn88cdmyc4+JTqdyJ49kRIYGKOEgqcnhYKjQbFACCGk1oXCihUb5OLFdCkvx5gK0RIaGiXjx1MoOCqUd4QQQmqVgwfz5MiRfCkvd5Ht22MlLCxannmGQsGRoWWBEEJIrQqF998/JaWl0RIUVCjx8eHy9NPR4u7uwlJ2YCgWCCGEXDHFxcWyefNhmTPHXYqLscZVmjWjUHAW6IYghBByRRQVFcnvv6+WM2cSpXnzM2pd167+8vTTUbQoOAm0LBBCCKkxBQUFsmLFn1JUlCtFRW5y+nSgdOlSIRQ8PNgedRZqdCVfe+01yc/Pr/KmwTZCCCHOT15enixfvkoJhYICd1m/vrk0bx4m48dTKDgbNRILr776quTm5lZaDwGBbYQQQpybnJwcWbFilRQX50tenocSCi1ahFAoOCnuNR2Vy8WlcmTrnj17JCQkpDbyRQghxE4pKyuTlSv/lJKSQsnJ8ZSNG+OldetGahwFuh6ck2qJheDgYCUSsLRq1cpIMODmgbVh1KhRdZFPQgghdsLx40WyfXu4xMWdlU2bmkmbNkEccMnJqZZYmD59urIqPProo8rdEBQUpN/m6ekpzZo1k759+9ZFPgkhhNgYvP+PHi2Qd95JkcLCQDl5MkA6dPDngEsNgGqJhZEjR6r/8fHx0q9fP/Hw8KirfBFCCLEjzpw5I1u37pRly6KlsLCi6oBQ4FwPDYMaxSxALKSlpZndHhsbeyV5IoQQYkecPn1aNm7cJDpduXI97N0bKR06+FEoNCBqJBbgbqgqwNEwfoEQQojjk5ycLFu2bIUTQo2hgGmmIRQmTOBcDw2JGomFXbt2GX0vKSlR695//3154403aitvhBBCbEhSUpJs375dfU5JCVKzR7ZrR6HQEKmRWOjcuXOldT169JDIyEh599135c4776yNvBFCCLERR48eld27d6vPSUnByvXQti2EQox4enJkxoZGrQ733Lp1a9m2bVtt7pIQQkg9A1fy4cPH1edjx0LlwIEIadXKV557Lka8vCgUGiI1EgsXL16s1J0GAY+vvPKKtGzZsrbyRgghxAacO1cmq1bFir9/hvz1V6i0aOEjzz8fI97eFAoNlRqJhUaNGlUKcIRgiImJke+//97q/UydOlV+/vlnSUxMFB8fH9Ud8+2331YWCkIIIfUH3uEZGRlSXh4ob755UjIzXeTs2cYSF+clEyfGiq+vGy9HA6ZGYmH16tVG311dXSUsLEwSEhLE3d36Xa5Zs0bGjh0rPXv2lNLSUpkyZYrccMMNcvDgQfHz86tJ1gghhNRAKOzYsUMFNB45EieZmQFqfUyMl7z4Yqz4+VEoNHRqJBYGDhxYKwdftmyZ0fc5c+ZIeHi4ummvvvrqWjkGIYQQ85SXl8vWrVslJSVFdDqR3NxStT4y0lMJhYCAWg1tIw5Kje+Cw4cPy8cffyyHDh1S39u2bSvjxo2TNm3a1Dgz2dnZ6r+5yaiKiorUYi52ghBCSPUCGTdv3qwGXSovF9m+PUbS0oIkIsJTJk+OlaAgCgVSQY2iVX766Sfp0KGDsgCgGyWWnTt3SseOHdW2mqrbZ555Rvr376/2bS7GAfNRaAtiJAghhFQfuH43bNjwP6HgIlu3ximhEBbmoYRCcDCH8yeXcNHBWVVNWrRoIcOHD5fXXnvNaP3LL78s8+bNk+PHK7rcVIfRo0fL0qVLZf369RIdHW21ZQGCARaJwMDAah+TEFL33H777fLLL7+wqO3MorB27Vo5f/68lJW5yubNsXL+vL+EhrrLP/4RJ2FhnrbOIqkHUIei4W1NHVojywK6SY4YMaLS+r/97W8W54wwB9wXv/76qwqcNCcUgJeXlzohw4UQQkj1QFB6YGCwlJW5yYYNzZRQCA52lylTKBRILYqFa665RtatW1dpPawCAwYMsHo/MGpAKCxcuFBWrVqlJqgihBBStxQWlsuvvwbJypUtJCvLV4KC3JTroUkTWhRI1bjX1Kw4adIkFbPQp08ftQ5BMj/88IO8+uqrRiZHpDUHuk3Onz9fFi9eLAEBAZKenq7WwyyCcRcIIYTUDvn5+apbevv2neWDD07L0aOFIuIpAQEQCnESGenFoia1G7MAE5Y1YOAmSzNQmpu5cvbs2fLwww/Xqr+FEGIbGLNge3Jzc9W4NhAMOTnhsmpVuFrv5+eqXA9xcd62ziKxAdWpQ91r2nOhNqiBTiGEEFLNCgFCobCwUEpLvWXjxkZqPYZuxsiMFAqkzmIWvv32W6NeCRrFxcVqGyGEENuTlZWlAschFMrKfOWPP+KksNBTPDxcZMKEaDXnAyF1JhYeeeQR/QBKhuTk5KhthBBCbAu6RcKigEacThcgv/8eK0VFHgIv8lNPRUm7dhxSn1iPe03dB1XFG6Smpir/ByGEENuPzFhSUiIuLkHy22+RUlrqJnhtjxoVKd26Vcz9QEidiIWuXbsqkYBl8ODBRpNG4ebEJCQ33XRTdXZJCCGklnFzc5O+ffvKunX7ZOHCMDXwEnj44Qjp148NOlLHYmHYsGHq/+7du+XGG28Uf39//TZPT09p1qyZ3HXXXTXIBiGEkCsFLge8i0Fiorv89FMTNTkUuO++MBk8OJiFTOpeLGA4ZwBRcN9994m3N7vbEEKIPXDixAnZs2ePmrE3KcldPv/8tF4oDBkSKkOGNLZ1FklDi1kYOXJk7eeEEEJIjTh27Jjs2rVLfd6587jMm+evZpEEgwc3knvvDWPJkvoXCxiUydyASsDSQEyEEEJqj8OHD8vevXvV57CweJk7109KSipMCv36BcrIkREW39eE1JlY+Pnnn41uPkTcQtV+8803arhnQgghdQt6pWH4ZiwgKqqlfPONtxQVVQiFrl395e9/jxRXVwoFYiOxoAU6GnL33XdL+/btZcGCBfLYY4/VQtYIIYSYEwqITzh69Kj63qxZW5kzx1Py8iqsuu3a+aqxFNzdKRSIDQdlMgcmlVq5cmVt7pIQQkgVYgHDOIOWLTvKd995S3Z2hVBo0cJbnn02Wjw9a/X1Tho4NbIsVEVBQYF89NFHEhUVVVu7JIQQYiZurF+/fnLiRJp88UWxnD9frNZHR3vJCy/Eio+PG8uN2F4sBAcHG8UsQOViqGdfX1+ZN29ebeaPEELI/wLHU1JSJC4uTr1/i4tdZM6cUjl9ukIohId7yKRJMeLvT6FA7EQsfPDBB0ZiASo3LCxMevfurYQEIYSQ2qO0tFQ2bdok6enpaprpFi3ayHvvpciJE4Vqe3Cwu0yeHCvBwR4sdmI/YuHhhx+WCxcuyKxZs+TQoUNqXbt27dTwooQQQmoP9DbbsGGDnDt3Tg3j3KhRiHz0UaocPlygtsOS8OKLsRIWVjFyIyF1QY0iYLZv3y4JCQnKwpCZmakWfG7RooXs3Lmz9nNJCCENdPjmtWvXKqGAuXiuumqA/PRTqezZk6e2e3u7KtdDVJSXrbNKnJwaWRaeffZZGTJkiHz55Zf6yaRgJnv88cflmWeeUTc3IYSQmlNYWKjepdnZ2Wq+hwEDBsjPPxfJli05aruHh4s8/3yMxMf7sJiJfYoFWBYMhYLakbu7TJw4UXr06FGb+SOEkAZHeXm5rFmzRnWP9PLyUvM9/PZbkfz55wW13c1N5JlnoqVNG19bZ5U0EGrkhggMDJTk5ORK6xGpGxDAedIJIeRKQNB4q1atVA+zQYMGyZ9/lsrSpZlqG2LLR4+Oks6dL836S4hdWhYw4yRGaZw2bZrq6wsQgPPCCy/IAw88UNt5JISQBgG6oWs9zeLj4yUmJkZWrrwoP/54Tp/m0UcjpE+fQBvmkjREaiQWIBJwQ48YMULFKgAPDw8ZPXq0vPXWW7WdR0IIcXrQwwxz7KBXmbe3t1q3cWOuzJ17Rp/mwQfDZdAgdk8nDiIWEGzz4YcfytSpU+X48eNqHXpCwGRGCCGkemRkZMi6detUN0nMINmrVy/Ztu2ifPllmj7NsGGN5ZZbQlm0xPGGe4Y46NixY+3lhhBCGhjoFrl+/XplpQ0NDZWuXbvKvn25MmPGadFVTCApN9wQLHfd1djWWSUNmFqbG4IQQkj1wIiMiPdC74fw8HDp37+//PVXsUyfniqlpRVKYcCAIPnb35oYjZpLSH1DsUAIITYgNTVVNm/erIIamzZtqmIVUlNLZNq0FCkqqhAKPXoEyOOPNxVXVwoFYlsoFgghpJ6BJWH//v1KKERHR6t5dc6cKZG3306W/PxylaZDBz8ZOzZS3NwoFIjtoVgghBAbjKOAERkRII64r4yMUnnrrWS5eLFMbW/Z0kcNuuThUaOhcAipdXgnEkJIPYERGTX8/PykU6dOSiBAKEAwgNhYLzWMM+Z9IMRe4N1ICCF1DNwNBw8elOXLl8upU6f06/PyypTrIT29WH2PiPCUSZNixc/PjdeE2BUUC4QQUsdCYd++fXLgwAH1WbMuFBaWq2DG5OQi9T001F1NNR0URO8wsT94VxJCSB0BcYBRGbXB6zp37qzmfCgpKVfdI48eLVDrAwPdlEWhcWMPXgtil1AsEEJIHfV4wAy9J0+eVN+7d+8uzZs3l7IynXz66WnZvz9Prff1dVVCITLSi9eB2C0UC4QQUgdCYcuWLWosBQymhOGbY2NjpbxcJ199lSbbtuWodF5eLiqYMS6uYi4IQuwVigVCCKllIBC8vLxUF8k+ffpIVFSUcknMm3dG1q3Lrnj5uruo7pGtWnFOHWL/UCwQQkgdiAXM8QC3Q6NGjdS6n38+L8uXZ/1vu6gBlzp29GfZE4eAvSEIIaQWKC4uVr0e4ILQBIMmFJYsyZCFC8/r0z7xRFPp2TOQ5U4cBloWCCHkCikqKpK1a9fKhQsX1OcePXrot/355wWZP/+s/jsmhbr66goRQYijQLFACCFXQEFBgaxZs0ZycnJUnEJCQoJ+25YtF2XWrDT9d0wzfdNNISxv4nBQLBBCSA3Jy8tTQgH/fXx8ZODAgRIQEKC27d6dK59+ekp0FRNIys03h8iwYY1Z1sQhsWnMAsx2Q4YMkcjISOXfW7RokS2zQwghVgNLwurVq5VQwDwPgwYN0guFgwfz5MMPU6WsYl4oGTgwSB58MFy95whxRGwqFvCQYUSzGTNm2DIbhBBSLRDEuG7dOuWCgECAUIBgAMeOFcj776dKSUmFSaFPn0B57LGmFArEobGpG+Lmm29WCyGEOBIYPwFBjPv375d+/fqJt3fFoErJyYXy7rvJat4H0LWrv4waFSmurrQoEMfGoWIWEGWMparpXgkhpK4pKysTN7eKGSHDw8OVRUFzLZw+XaSmms7LqxAK7dr5ylNPRanBlwhxdBxqnIWpU6dKUFCQfomJibF1lgghDYT09HRZtmyZZGdXjMAINKFw7lyxEgoXL1YEKSQk+MiECTHi6elQr1hCzOJQd/LkyZPVg6otKSkpts4SIaQBcOrUKdmwYYPk5+fLkSNHjLZlZZXI1KnJkplZqr7HxXnJCy/EiLe3Q71eCXEeNwT6MGMhhJD6Ijk5WbZu3armdsAcD5g9UiMnp1TefjtFzp4tUd+bNvWUiRNjxc+vwlVBiLPgUGKBEELqk6SkJDXNNIiLi1NBjQhuBPn5ZfLOOymSmloRRxUW5iEvvhgrQUF8rRLnw6Z3dW5urhw7dszowdy9e7eEhISo6VwJIcRWHD16VL2PACaE6tatmz5GoaioXKZNS5GkpEL1vVEjdyUUQkM9eMGIU2JTsQDFjmhijQkTJqj/I0eOlDlz5tgwZ4SQhj6OQmpqqvrcqlUr6dSpk14olJSUy/TpqXLkSIH6HhDgJpMnx0qTJp42zTMhTisWrrnmGuUHJIQQewKuhquuukoFUcfHx+uFQmmpTj7++JTs25envvv4uMrEiTESFcVYKuLcMFyXEEJEVMMlLe3SpE8eHh7K/WAoFD755JTs3Jmrvnt5uaheD/HxPiw/4vRQLBBCGjwQCnCLrl+/XhITEyuVR1mZTk0KtX17jvru4eGixlFo1cq3wZcdaRgwbJcQIg09PgFdI7VxWzB7pKlQmDnztGzdaigUoqV9+4q5IAhpCFAsEEIa9PDNmzZtUu4HuBv69Okj0dHR+u3l5Tr54ovTsnlzxdDyGLr5mWeipWNHfxvmmpD6h2KBENIgKS0tVaMynj17VgU0YkKopk2bGgmFL79Mkw0bKoQCpoQYPz5KOnemUCAND4oFQkiDnWL6/PnzamIo9HzAxFCXtuvk66/TZd26bL1QePrpaOnaNcCGuSbEdjDAkRDS4IAlAe4G9HgYOHCgkVBAsOM336TLn39e+F9akbFjMcwzhQJpuNCyQAhpkLRs2VLNXOvt7W0kFL799oysXFkhFNBrcsyYKOnVK9CGOSXE9tCyQAhpEGDGyI0bN0pxcbF+naFQgOth9ux0WbEiSy8URo+OlD59KBQIoWWBEOL05OTkyNq1a5VgQK+Hvn37Gm3Xghm1GAUIhb//van06xdkoxwTYl9QLBBCnJrs7GwlFAoLC8Xf3186d+5caRyFzz8/LRs3XtTHKIwaFUmhQIgBFAuEEKclKytLCQW4HoKCguTqq682cj1gCGeMzKgNuIReDwhmZIwCIcZQLBBCnBJ0i0T3SIynEBwcrISCp+elmSExeyTmetixI1c/4NJTT7HXAyFVQbFACHHauR4gFBo3bqzGUUA3SY3i4nL58MNU2bMnTz+EM0Zm5IBLhFQNxQIhxOlAEGP//v3l4MGD0r17d3F3v/Sqy88vk/ffT5XExHz13dOzYlKoDh041wMh5qBYIIQ4DQhi1GISAgICpHfv3kbbs7NL5Z13kuXkySL13dvbVZ5/PkbatOHskYRYguMsEEKcghMnTsiSJUskPT29yu3nzhXLa6+d0AuFgAA3mTIllkKBECugZYEQ4vAcO3ZMdu3apT5jBsmIiAij7SkphfLOOymSlVWqvoeGusukSbESGellk/wS4mhQLBBCHJrExETZt2+ffghn03EUjh7Nl2nTUiQvr1x9j4z0VEIhNPRSwCMhxDIUC4QQh+3xcODAATl06JD63rZtW2nfvr0KbtTYsydXPvooVYqKdOp7fLy3TJwYIwEBfPURUh34xBBCHFIo7NmzR44ePaq+d+zYUdq0aWOUZuXKLDV7ZHmFQUHatfOVZ5+NFh8fN1tkmRCHhmKBEOKQaBNCde3aVRISEozmeViw4Kz89lumfl3PngEyZkykeHgwppuQmkCxQAhxOOBq6NGjh8TFxUmTJk2MBluaOfO0bNtWMXwzuOWWELn//nBxdb3kniCEVA+KBUKIQ1BWViZ//fWXsiJALLi6uhoJBYyh8P77KXL8eKH6jtCFkSMj5Lrrgm2Ya0KcA4oFQojdg2GbN27cKGfOnFHTTXfr1s1o++nTRfLuuyly7lyJfrAlzPPA4ZsJqR0oFgghdk1JSYmsX79eTQzl5uYmUVFRRtvR42HGjFOSn18RyRgc7K5GZYyLuzS7JCHkyqBYIITYdRAjppjGVNOY32HAgAFqYiitR8R//5shP/xwTnQVPSMlLs5LnnsuRkJCOIYCIbUJxQIhxG7neYBQyM7OVlNLY4ppTDVdsa1cvvzytGzZcimQsUePAHnyyabsGklIHUCxQAixO2A10IQCJoaCUAgKClLb0tOL1UBLyckVczyAu+8Ok9tvD2WPB0LqCIoFQojdgd4OGGhp9+7dyvXg7++v1m/efFG++ipNWRa0QEaMn9CtW4CNc0yIc0OxQAixK4uCNlxz06ZNVddIdJHE+Anz55+VP/7I0qfFHA/PPBPNyaAIqQcoFgghdsGFCxdk69at0q9fP70lAUIBbodPPjklJ05UjJ8A+vULlEcfbaosC4SQuodigRBiczIyMmTdunWqm+TevXuVYICVYdWqCzJ//hn9RFAeHi4yYkQTueaaRkYTRhFC6haKBUKITTl79qwaRwEjNIaGhkrPnj3VaIyITdi1K1efLiLCU55+OkpiYzl+AiH1DcUCIcRmpKWlqZEZy8vLJTw8XPr37y+7duXL7NnpcvFimT7dtdc2kgcfbEK3AyE2gmKBEGITUlNTZfPmzcrdgGDGNm16yIwZ6UaTQAUGuskTTzSVrl3Z24EQW0KxQAipdyAQDh8+rP5HR0dLYWFrmTLlpH7IZtCtm7889lhTCQria4oQW8OnkBBS7yA4EeMn7NhxSFauDJLExDP6bQEBbvLQQ02kb99ABjESYidQLBBC6o3MzEwJCQmR3Nwy+emnTPnjD0/R6Qr026+6KkiGDw+XgAC+mgixJ/hEEkLqHLgbDh48qBYfn9by66/eSjBohId7yMMPR0inThXjKxBC7Au7GNFkxowZ0qxZMzUGfO/evdXALIQQ52HPnj1KKIAdO7L0QsHLy0XuvTdM3nqrOYUCIXaMzcXCggULZMKECfLyyy/Lzp07pXPnznLjjTeqvteEEMemtLRUzp7NkqNHj6rv+/ZFyNGj4fpRGN99t4Xcfntj8fS0+auIEGLPboj3339fnnjiCXnkkUfU988++0x+++03+frrr+XFF1+s17zk5JTKjBmnxcfngri7F0tVA8RhXW5uuD5Iy9s7W9zdK2a/M02P7fn5YfogLU/Pi+LufmnIWtP0RUWNxcXFTX12d0faCl9uVSPVFRdXpMUmN7cccXPL1+8Ti6uri/pdxffG4ubm8b916JaW9780WjrtN5iYJ1zc3DzFzc1Fyspypazsori54RguJgtMxxHi7+8jHh6ukp+fIxcuZJgNSMMY/76+vv8r5xw5f/682euA/vZ+fn7qc15enkXh2LhxYwkIqOhWl5+fL2fOXAqUMwUD/gQGBqrPBQUFkp6ebjYt/OraLIdFRUVy+vRps2kbNWqknzq5uLhYTp06ZTYt9ol9A4xWiO6D5sB54fwABixKTk42mxbDI4eFhanPGLPg5MmTZtOibFHGmnvgxIkTZtP6+PhIRESE/jvS4jdVAcsgukCC0lKdrFixW7KyTkhpaYHgJ7t3R0pycoi0b+8r99wTLgkJPmaPSwixL2wqFvBi3bFjh0yePFm/DmPBX3fddbJp06ZK6fHSxqJx8eLFWs0PXnD79+dJnz6nJDz80shxpmzYUFGRgR49TkmTJubzsWGDl5SVVbSaunY9JbGxF8ymXbrUQ4qLKy5Jp06nJT4+02za5ctdpaDAU31u1y5dWrY0X/muWlUuOTkVo961bn1G2rQ5Zzbtb78VyYULFZV6QsI5ad/efOX79dfxkpFRURbx8RnSqVOa2bRnzrRGda3Mzn5+58TD44jZtO3b95T4eG81AA8C4rZv3242LUb708QCpjO2lLZr1656sQDBYiltp06d9GIBgsVS2nbt2unFAkSIpbStW7fWiwXc/5bStmjRQi8WICwspYUbz1AsWEqLroqaWACW0kIoGIoFWP8gXKoCxw8KCpc1ay7I0qWZ0qtXknh5lan7f9u2GPHzi5DJk8OkfftLzw8hxDGwqVhA6xIvHrQ6DcH3xMTESumnTp0qr776aqX19913n3h4eFxxfkpKymXfvnzZvr1IPDyqfiGCrKyKyhRs21Yknp7m01644CM6nUu1027dWixeXqVm02Zne0t5eYUI2bKlWLy9zae9eNFbL1i2bCkRb+8SC2khbiqsG5s3l4iPj/m0OTleUlpakXbTphJZtMh82txcTykpqbjdcF6+vsVm0+bleSrRBCOFt3eZ+PoWGVhJKiwtmiXE09NXvLy81JwBOl2p5OWZF3mwbKD1q1W+EAyWWtRYNFO6JWFqmBb3M0SLOXB8zcKCSh2TJ5kD56VZWC6X1tPTUz/5Elr+WVlZVqUFEGTmwHOliTFzaXE86IfiYlc5f95Dyv83VMLKlQVSXOwmaWlJEhj4mhovYedOs4cihNQzeA86jBuiOsACgfgGDbzAY2JiVNyD1mK8EsrLdVJcrFMvP0NLa1WfDU2x+GiaxnS7uf1o+6r8e+Ptln9/KQ1e1DgPw/9lZVV9rzimtg3/Db9jOywtWEpKKpaKz+Um36tehymFCwsrFq3yqA9iYlwlKMhNVUxYGjVyl+BgdwkJ8ZCQkEv/6SO/ci5cKJWtWy8qS8LJk5csfhpdu/rLbbeFygsv3C+//PJLLRyREFKboA7VLKh2LRZgYnVzc6vkZ8Z3Q9OnYUsLS10B/723N2eyq00gYiAeiorKpaCgXP3XRITpgtH7ECWfl1exVHwuV5/xW2vQ9nXmjGXF7O/vZiQeTP+HhnqIlxeD7qqK68FwzJs3X5RDh/KNRCuA66h//0C5/voQiY6uu2eVEFK/2FQswBzavXt3WblypQwbNkxvbsX3cePG2TJrpJaAy8DTE4urGFizqw0sHvn5FQIiJ6dMTTKEmQkvXiyV7Oyy//2/9Nlw2OCqwH6wJCdXbhEbjiQI4dC4MQSEhxIQFZ8rxAQsFhCYzgwsTH/9VSh79+bKnj256nNV8Y3Nm3vLtdcGS58+gZzsiRAnxOZuCLgVRo4cKT169JBevXrJ9OnTVUCZ1juCEIAeGBjVD8v/Au4tAjcIBAVM5ZmZJZKZafw/K6viv5lYPQVECZaqTOwAMRMQDJdEBARFxXdtna+vq0MNWQzLT1JSoRw5ki9HjxaoxXDwJEOaNPFQ4gBLTAynjSbEmbG5WEBw4rlz5+Sll15SXdm6dOkiy5YtqxT0SEh1gCWjcWMsCHz1MdtqhhgwFBEZGdr/S5/NxVxgPdJgOXLk0pDFhqD3hyYeKpZLYgKLreInIKbOny+RtLRiSU0tkpSUIvU/La3IooCCa6FLF3/p3TtAmjXzdighRAipOS46c52mHSg4A9HntRHgSEhVggLWCU08VPy/JCbwH4LjSoCfHy4PTMeMWApYT/DZx8dVbcOC+AntM3p+aD1CgNZLBCBAF9aBikWn4j3gloF1AJYWiB+IBLhrrAH5adPGVzp39lMjLELgVJfbb7+dAY6EOHgdanPLAiH2DGISKoIePaRly6rToGLWrBJViQksqMQvF5R57pz13ZjqAgy0FRnpJS1a+EjLlhVL06aetB4QQigWCLlS0Opv2tRLLVUB4x1a9lWJCYgMLTYCVoD6sPMhzgLumbCwigWuBcQcRER4irs73QqEkMrQskBIHQOXgRacCT+/JZeH1tsDS0FBmXIlmHYxxbgWpmN3aP/R8wTiRfvv61vh3oCbQ3NvcIwJQkh1oVggxI5cHoGBqND5WBJC7AuOOkMIIYQQi1AsEEIIIcQiFAuEEEIIsQjFAiGEEEIsQrFACCGEEItQLBBCCCHEIhQLhBBCCLGIQ3fo1qa1wPjWhBD7pKSkhM8oIXaIVndaM0WUQ4uFnJwc9T8mJsbWWSGEWACT1RBC7Lcuvdwz6tCzTpaXl8vp06clICCg1ia7gdKC+EhJSXGamSx5To4Br5P9w2vkGPA6WQeqfwiFyMhIcXV1dV7LAk4uOjq6TvYNoeAsYkGD5+QY8DrZP7xGjgGvU+1Z/RjgSAghhBCLUCwQQgghxCIUCyZ4eXnJyy+/rP47Czwnx4DXyf7hNXIMeJ1qH4cOcCSEEEJI3UPLAiGEEEIsQrFACCGEEItQLBBCCCHEIhQLhBBCCLEIxYKVFBUVSZcuXdRIkbt37xZH5fbbb5fY2Fjx9vaWpk2bykMPPaRGwXRUTpw4IY899pjEx8eLj4+PtGjRQvVmKS4uFkfmjTfekH79+omvr680atRIHJEZM2ZIs2bN1L3Wu3dv2bp1qzgya9eulSFDhqjR7vAeWLRokTgyU6dOlZ49e6oRcMPDw2XYsGFy+PBhcWRmzpwpnTp10g/G1LdvX1m6dKk4C2+99Za695555pl6PzbFgpVMnDhRvSQcnUGDBsl//vMf9VL46aef5Pjx43L33XeLo5KYmKiG/f7888/lwIED8sEHH8hnn30mU6ZMEUcGYueee+6R0aNHiyOyYMECmTBhghJuO3fulM6dO8uNN94oZ8+eFUclLy9PnQdEkDOwZs0aGTt2rGzevFlWrFihJvy64YYb1Hk6KhjRFxXqjh07ZPv27XLttdfK0KFD1bvB0dm2bZt6z0EM2QR0nSSWWbJkia5Nmza6AwcOoJupbteuXU5TZIsXL9a5uLjoiouLdc7CO++8o4uPj9c5A7Nnz9YFBQXpHI1evXrpxo4dq/9eVlami4yM1E2dOlXnDOA9sHDhQp0zcfbsWXVea9as0TkTwcHBuq+++krnyOTk5OhatmypW7FihW7gwIG68ePH13seaFm4DGfOnJEnnnhC5s6dq0zCzkRmZqZ89913ytzt4eEhzkJ2draEhITYOhsNFlhF0LK77rrrjOZxwfdNmzbZNG/E8nMDnOXZKSsrk++//15ZSuCOcGTGjh0rt956q9EzVd9QLFgADYiHH35YRo0aJT169BBnYdKkSeLn5yehoaGSnJwsixcvFmfh2LFj8vHHH8uTTz5p66w0WM6fP69e1E2aNDFaj+/p6ek2yxcxD1x58IP3799fOnTo4NBFtW/fPvH391ejOOLdvXDhQmnXrp04Kt9//71y5SHGxJY0SLHw4osvqiARSwt84ah0MH3n5MmTxRnOR+OFF16QXbt2yfLly8XNzU1GjBihhJEjnxM4deqU3HTTTcrXD2uQvVGTcyKkvlqu+/fvVxWTo9O6dWsVhL5lyxYV8zNy5Eg5ePCgOCIpKSkyfvx4ZQFGoLAtaZDDPZ87d04yMjIspmnevLnce++98t///le9xDXQYkIFO3z4cPnmm2/Ekc7H09Oz0vrU1FSJiYmRjRs32pWprrrnhB4d11xzjfTp00fmzJlz2bnZHeU64VzQ4rtw4YI4khsCLrsff/xRRdhr4KWN83AGSxbeCWixGp6fozJu3Dh1TdDbA72KnA2Y7tFLCsGBjsaiRYvkjjvuUHWOYR2E+w/vOPTSM9xWl7hLAyQsLEwtl+Ojjz6S119/Xf8dFRIiuhHpja5gjnY+5syPADedPVGdc4JFAb08unfvLrNnz7ZLoXCl18mRgNjBtVi5cqW+MsV9hu+omIh9gHbiU089pUTPn3/+6ZRCQbv37O39Zi2DBw9WbhVDHnnkEWnTpo1yJ9eXUGiwYsFaMB6BIfCDAahUdNFxNGCWQ/ebq666SoKDg1W3yX/+85/qfOzJqlAdIBRgUYiLi5Np06ap1rtGRESEOCqIJUEAKv6jJaGN7ZGQkKC/D+0ZdJuEJQGxPr169ZLp06erQDO86ByV3NxcFROjkZSUpK4LAgJN3xWO4nqYP3++sipgrAUtniQoKEiNWeKIwGV88803q+sBFzLOD0Lo999/F0ckICCgUgyJFm9W77El9d7/woFJSkpy6K6Te/fu1Q0aNEgXEhKi8/Ly0jVr1kw3atQoXWpqqs6RuxbimlS1ODIjR46s8pxWr16tcxQ+/vhjXWxsrM7T01N1pdy8ebPOkUHZV3VNcK0cEXPPDZ4pR+XRRx/VxcXFqXsuLCxMN3jwYN3y5ct1zsRAG3WdbJAxC4QQQgixHvt07hJCCCHEbqBYIIQQQohFKBYIIYQQYhGKBUIIIYRYhGKBEEIIIRahWCCEEEKIRSgWCCGEEGIRigVCCCGEWIRigRBiFgyljYmsCCENG47gSAgxC+an8PDwUGPU1xevvPKKmm1Pmw+DEGJ7OJEUIcQsmCSJEELohiCEWOWGaNasmbz55pvy6KOPKksDZvb74osv9GlPnDghLi4u8v3330u/fv3E29tbzYy3Zs0afZo5c+ZIo0aNjI4BKwJ+p21/9dVXZc+ePWodFqwjhNgWigVCiNW89957atrpXbt2yZgxY2T06NFy+PBhozQvvPCCPPfccyoNpj4fMmSIZGRkWLX/++67T/22ffv2kpaWphasI4TYFooFQojV3HLLLUokJCQkyKRJk6Rx48ayevVqozTjxo2Tu+66S9q2bSszZ86UoKAgmTVrllX79/HxEX9/f3F3d5eIiAi1YB0hxLZQLBBCrKZTp076z3ARoDI/e/asURpYEzRQ6cMScejQIZYyIQ4MxQIhxGrQM8IQCIby8nLrXziurqLT6YzWlZSU8AoQYudQLBBCapXNmzfrP5eWlsqOHTuUSwKEhYVJTk6O5OXl6dOYdpH09PSUsrIyXhVC7AiKBUJIrTJjxgxZuHChJCYmytixYyUrK0v1oAC9e/cWX19fmTJlihw/flzmz59fqbcDel0kJSUpEXH+/HkpKiriFSLExlAsEEJqlbfeekstnTt3lvXr18svv/yiAiG1cRvmzZsnS5YskY4dO8q///1vNQiTIQiOvOmmm2TQoEHKEoE0hBDbwhEcCSG1AsZZiI+PV10mu3TpwlIlxImgZYEQQgghFqFYIIQQQohF6IYghBBCiEVoWSCEEEKIRSgWCCGEEGIRigVCCCGEWIRigRBCCCEWoVgghBBCiEUoFgghhBBiEYoFQgghhFiEYoEQQgghYon/ByQyX1ulnNXHAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "feed-forward width per token: 64 -> 256 (expand + GELU) -> 64 (contract)\n", "W1: (64, 256) W2: (256, 64)\n" ] } ], "source": [ "# GELU vs ReLU, then the expand-then-contract widths of the feed-forward net.\n", "xs = np.linspace(-4, 4, 200)\n", "plt.figure(figsize=(6, 3))\n", "plt.plot(xs, gelu(xs), label=\"GELU (used here)\", color=\"#5b5bd6\", lw=2)\n", "plt.plot(xs, np.maximum(xs, 0), label=\"ReLU (for comparison)\", color=\"#aaa\", ls=\"--\")\n", "plt.axhline(0, color=\"k\", lw=.5); plt.axvline(0, color=\"k\", lw=.5)\n", "plt.legend(); plt.title(\"GELU keeps positives, softly suppresses negatives\")\n", "plt.xlabel(\"input\"); plt.ylabel(\"output\"); plt.show()\n", "\n", "print(f\"feed-forward width per token: {C} -> {4*C} (expand + GELU) -> {C} (contract)\")\n", "print(f\"W1: {W1.shape} W2: {W2.shape}\")" ] }, { "cell_type": "markdown", "id": "m6gec", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • xs = np.linspace(-4, 4, 200): a range of inputs to feed the activation.
    • \n", "
  • plt.plot(xs, gelu(xs)): GELU's smooth S-curve -- it passes large positives through almost unchanged and eases negatives toward zero, with a soft dip near -1 rather than ReLU's hard corner.
    • \n", "
  • The dashed ReLU (max(x, 0)) is drawn for contrast: same idea, blunt instead of smooth.
    • \n", "
  • The prints show the width change: each token goes 64 -> 256 -> 64. The wide middle (four times C) is the room the network computes richer features in before shrinking back.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "0f93f646", "metadata": {}, "source": [ "## Assemble the block (pre-norm, with residuals)\n", "\n", "Now we wire the two sub-layers together. Modern GPTs use the **pre-norm** arrangement: apply LayerNorm first, then the sub-layer, then add the result back to the input. That \"add back\" is the **residual** (or skip) connection, and it is worth pausing on:\n", "\n", "```\n", "x = x + MultiHeadAttention(LayerNorm(x))\n", "x = x + FeedForward(LayerNorm(x))\n", "```\n", "\n", "Read each line as \"keep `x`, and add an improvement on top.\" Each sub-layer only has to learn a small edit to the running representation, rather than rebuilding it from scratch. Just as important, the `x +` gives the gradients a clean route straight back through the whole stack during the backward pass; without residuals, deep Transformers simply do not train, because the gradient fades to nothing before it reaches the early layers.\n", "\n", "Stacking the block three times below returns the same shape it took in, which is the whole point: identical blocks can be stacked, and a real GPT simply stacks more of them." ] }, { "cell_type": "code", "execution_count": 9, "id": "c91bc971", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.509810Z", "iopub.status.busy": "2026-07-01T09:08:58.509755Z", "iopub.status.idle": "2026-07-01T09:08:58.513045Z", "shell.execute_reply": "2026-07-01T09:08:58.512786Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "after 3 blocks: (4, 16, 64) (shape preserved -> blocks are stackable)\n", "finite: True\n" ] } ], "source": [ "def transformer_block(x, params):\n", " p = params\n", " # sub-layer 1: attention, with a residual connection\n", " a = multi_head_attention(layer_norm(x, p['g1'], p['b1']), p['Wq'], p['Wk'], p['Wv'], p['Wo'], n_head)\n", " x = x + a\n", " # sub-layer 2: feed-forward, with a residual connection\n", " f = feed_forward(layer_norm(x, p['g2'], p['b2']), p['W1'], p['ff_b1'], p['W2'], p['ff_b2'])\n", " x = x + f\n", " return x\n", "\n", "params = dict(\n", " g1=np.ones(C), b1=np.zeros(C), g2=np.ones(C), b2=np.zeros(C),\n", " Wq=Wq, Wk=Wk, Wv=Wv, Wo=Wo,\n", " W1=W1, ff_b1=b1, W2=W2, ff_b2=b2,\n", ")\n", "# token embeddings + positional encoding go in; same shape comes out\n", "h = x + positional_encoding(T, C)\n", "for _ in range(3): # stack 3 blocks\n", " h = transformer_block(h, params)\n", "print(\"after 3 blocks:\", h.shape, \" (shape preserved -> blocks are stackable)\")\n", "print(\"finite:\", np.isfinite(h).all())" ] }, { "cell_type": "markdown", "id": "0a498581", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • def transformer_block(x, params):: wire the two sub-layers together; params is a dictionary holding every weight, so p['Wq'] fetches the query projection, etc.
    • \n", "
  • a = multi_head_attention(layer_norm(x, ...), ...): read inside-out: normalize, then attend.
    • \n", "
  • x = x + a: add the attention result back onto x. That x + is the residual: keep the original, add an edit on top.
    • \n", "
  • f = feed_forward(layer_norm(x, ...), ...) then x = x + f: the same normalize → think → add-back pattern for the second sub-layer.
    • \n", "
  • params = dict(g1=np.ones(C), b1=np.zeros(C), ...): bundle all the pieces: LayerNorm dials (set to \"no change\"), the four attention grids, the two feed-forward grids and biases.
    • \n", "
  • h = x + positional_encoding(T, C): stamp positions onto the tokens before the first block.
    • \n", "
  • for _ in range(3): h = transformer_block(h, params): stack three blocks. Same shape out as in, every time -- which is exactly what lets you stack them.
    • \n", "
  • np.isfinite(h).all(): a health check: no infinities or NaNs anywhere after 3 blocks.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "m6resa", "metadata": {}, "source": [ "### Residuals: keep `x`, add a small correction\n", "\n", "Both block lines read `x = x + something`. That \"+ x\" is the residual connection. Measure the size (root-mean-square) of each piece and you can see what it buys: each sub-layer adds only a small correction on top of `x`, and the block's output stays on the same scale as its input." ] }, { "cell_type": "code", "execution_count": 10, "id": "m6resb", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T09:08:58.514076Z", "iopub.status.busy": "2026-07-01T09:08:58.514019Z", "iopub.status.idle": "2026-07-01T09:08:58.516794Z", "shell.execute_reply": "2026-07-01T09:08:58.516453Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "block input x : rms = 1.213\n", "attention correction a : rms = 0.014 <- added: x = x + a\n", "feed-forward corr. f : rms = 0.027 <- added: x = x + f\n", "block output x+a+f : rms = 1.214 (same scale as the input)\n", "\n", "The corrections are small on purpose: small initial weights make each block\n", "start as ALMOST the identity (output ~ input). The residual is what preserves\n", "the scale, so you can stack many blocks and still train them.\n" ] } ], "source": [ "# Measure the size of each quantity a block produces (rms = typical magnitude).\n", "def rms(t): return float(np.sqrt((t ** 2).mean()))\n", "\n", "x0 = x + positional_encoding(T, C) # block input\n", "a = multi_head_attention(layer_norm(x0, params[\"g1\"], params[\"b1\"]), Wq, Wk, Wv, Wo, n_head)\n", "x1 = x0 + a # first residual add\n", "f = feed_forward(layer_norm(x1, params[\"g2\"], params[\"b2\"]), W1, b1, W2, b2)\n", "x2 = x1 + f # second residual add\n", "\n", "print(f\"block input x : rms = {rms(x0):.3f}\")\n", "print(f\"attention correction a : rms = {rms(a):.3f} <- added: x = x + a\")\n", "print(f\"feed-forward corr. f : rms = {rms(f):.3f} <- added: x = x + f\")\n", "print(f\"block output x+a+f : rms = {rms(x2):.3f} (same scale as the input)\")\n", "print(\"\\nThe corrections are small on purpose: small initial weights make each block\")\n", "print(\"start as ALMOST the identity (output ~ input). The residual is what preserves\")\n", "print(\"the scale, so you can stack many blocks and still train them.\")" ] }, { "cell_type": "markdown", "id": "m6resc", "metadata": {}, "source": [ "
Line by line: what each line does\n", "
    \n", "
  • rms(t): the root-mean-square, a single number for \"how big are these values typically\".
    • \n", "
  • x0: the block's input (token embeddings plus positions), rms about 1.2.
    • \n", "
  • a and f: the attention and feed-forward outputs -- both much smaller than x0, so x = x + a and x = x + f nudge the signal rather than overwrite it.
    • \n", "
  • The output stays on the same scale as the input, which is exactly why cell above could stack three blocks and stay finite.
    • \n", "
  • Small corrections at the start (from small weights) mean the block begins near the identity and training gently grows it -- the residual is what makes that possible.
    • \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "a0657c2d", "metadata": {}, "source": [ "## The wall we have just hit, on purpose\n", "\n", "We have built the entire forward pass of a Transformer by hand. Notice what we did not do: the backward pass. Differentiating multi-head attention, LayerNorm, the residuals, and the feed-forward stack by hand, the way we did for the bigram in notebook 03, would be hundreds of careful and error-prone lines, and a single wrong sign would break the whole thing silently.\n", "\n", "This is exactly the moment autograd (notebook 04) earns its place. In Part 2 we rebuild all of this in **PyTorch**, where every operation we just wrote already knows how to compute its own gradient. The concepts are the same, the code is a fraction of the length, and it actually trains.\n", "\n", "Next: notebook 07, PyTorch and autograd." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (.venv)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }