SaShiMi-MNIST-Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "be9c5d7a",
   "metadata": {},
   "source": [
    "<center><h1>SaShiMi Demo on MNIST</h1></center>\n",
    "\n",
    "This notebook is intended to test whether our SaShiMi implementation works correctly by running it on MNIST. SaShiMi introduces new layers for down-pooling and up-pooling, which require special handling in recurrent mode.\n",
    "\n",
    "There are unit tests available in this repository. Run all unit tests by executing the following command in the root directory:\n",
    "```bash\n",
    "python3 -m unittest\n",
    "```\n",
    "\n",
    "For a more preliminary introduction to S4, see [S4-MNIST-Demo](https://github.com/necrashter/SaShiMi-796/blob/master/S4-MNIST-Demo.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec6e85f1",
   "metadata": {},
   "source": [
    "# Prelude\n",
    "\n",
    "Setup, import modules, select device, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "132b48c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "# Built-in IPython extension to reload modules when updated.\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c1eee2fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using PyKeOps Cauchy kernel.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.auto import tqdm\n",
    "import copy\n",
    "import IPython.display as ipd\n",
    "\n",
    "# Custom modules\n",
    "from S4 import *\n",
    "from sashimi import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cd3b4d5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "#device = torch.device(\"cpu\")\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f56c5212",
   "metadata": {},
   "source": [
    "# Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2f898191",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load datasets\n",
    "MNIST_TRAIN = datasets.MNIST(root='./datasets', train=True, download=True, transform=transforms.ToTensor())\n",
    "MNIST_TEST = datasets.MNIST(root='./datasets', train=False, download=True, transform=transforms.ToTensor())\n",
    "\n",
    "dataloader = DataLoader(MNIST_TRAIN, shuffle=True, batch_size=8)\n",
    "testloader = DataLoader(MNIST_TEST, shuffle=False, batch_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3d13cbd",
   "metadata": {},
   "source": [
    "# Model\n",
    "\n",
    "Here we define the model. We won't go into the implementation details in this notebook. See the documentation and comments in the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7c2c24c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "model = SaShiMi(\n",
    "    input_dim=1,\n",
    "    hidden_dim=64,\n",
    "    output_dim=256,\n",
    "    state_dim=64,\n",
    "    sequence_length=784,\n",
    "    block_count=2,\n",
    ").to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52b170ff",
   "metadata": {},
   "source": [
    "# Overfitting Test\n",
    "\n",
    "In accordance with the well-respected 796 tradition, we first test whether the model works by trying to overfit into a single example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6226d367",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fadd3f59d80>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMkAAADICAYAAABCmsWgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAO00lEQVR4nO3df1CTd54H8HdQ8oD8eCJ6JKTCyt5213a9xTkKNGOvQ9ucjLvDSOV27M1cx/64um2Ds8jMdcSpeON1Nj29ra5K239aaHfO0qF74NT2mOlEhbELuFJbz9Iydo+t7EBC7S1JRAk/8r0/XOPlnge/BBKS6Ps18/zBJ988fL7omy/Pk+dJDEIIASKaVUq8GyBKdAwJkQRDQiTBkBBJMCREEgwJkQRDQiTBkBBJMCREEgwJkcTSWO24sbER+/fvh9vtRlFREQ4fPozS0lLp84LBIIaHh5GVlQWDwRCr9ugOJ4SA3++H1WpFSopkrRAx0NLSIoxGo3jzzTfF559/Lp555hlhMpmEx+ORPndoaEgA4MZtUbahoSHp/0mDENG/wLGsrAwlJSU4cuQIgOurQ35+PrZv346dO3fe8rlerxcmkwkP4MdYitRot0YEAJjGFE7jQ4yNjUFV1VuOjfqfW5OTk+jr60N9fX2olpKSArvdju7ubs34QCCAQCAQ+trv9/+5sVQsNTAkFCN/Xhrm8id91A/cL1++jJmZGZjN5rC62WyG2+3WjHc6nVBVNbTl5+dHuyWiBYn72a36+np4vd7QNjQ0FO+WiMJE/c+tlStXYsmSJfB4PGF1j8cDi8WiGa8oChRFiXYbRFET9ZXEaDSiuLgYLpcrVAsGg3C5XLDZbNH+dkQxF5PXSerq6rB161bcd999KC0txcGDBzE+Po4nn3wyFt+OKKZiEpItW7bgm2++QUNDA9xuN9atW4eOjg7NwTxRMojJ6yQL4fP5oKoqyrGJp4ApZqbFFE7hGLxeL7Kzs285Nu5nt4gSHUNCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSMfuI6mT37TPaz1IpePwr3bFfjmrfLX8yoP9m33e9o60v++MV3bHBT/tv1SItEq4kRBIMCZEEQ0IkwZAQSTAkRBI8uzWLF/7pqKZWnfEn/cF/GcGOy7WlP0xf1R36q28eimDH8Xdm9DuaWsYvVd2xS119sW4nariSEEkwJEQSDAmRBENCJMED91kc2vWYptbwI/3fKcu/EJran+4x6I41/mhMU9u39j90xx7I69XUPriaqTv2J8v0L22Zq2tiUrfeG8jQ1MrTpvR3otPv97b8THfo911z7y3euJIQSTAkRBIMCZEEQ0IkEXFIurq6UFlZCavVCoPBgPb29rDHhRBoaGhAXl4e0tPTYbfbcfHixWj1S7ToIj67NT4+jqKiIjz11FPYvHmz5vF9+/bh0KFDeOutt1BYWIjdu3ejoqIC/f39SEtLi0rTiyHjPe2Zmoz35v787Ai+12FLuW79pfWrtfvt1L/xa1/59yL4jlpLrwV16xnnRzS1FV2/0R37V0adG8r+oH/zWTKJOCQbN27Exo0bdR8TQuDgwYN48cUXsWnTJgDA22+/DbPZjPb2djz2mPa0KlGii+oxyeDgINxuN+x2e6imqirKysrQ3d2t+5xAIACfzxe2ESWSqIbE7XYDAMzm8Hu+zWZz6LH/z+l0QlXV0Jafnx/NlogWLO5nt+rr6+H1ekPb0NBQvFsiChPVy1IsFgsAwOPxIC8vL1T3eDxYt26d7nMURYGiKNFsI+lMuz269YzfaOszs+wj471vo9jRTZ5/1L5rzA+N+v9t/u1/fqCprW76b92x0wtra1FFdSUpLCyExWKBy3Xzwhyfz4fe3l7YbNofNlEyiHgluXLlCr766uZpyMHBQXz66afIyclBQUEBamtr8dJLL+Huu+8OnQK2Wq2oqqqKZt9EiybikJw9exYPPXTzttK6ujoAwNatW9Hc3IwXXngB4+Pj2LZtG8bGxvDAAw+go6MjqV4jIfq/Ig5JeXk5hNBeGn6DwWDA3r17sXfv3gU1RpQo4n52iyjR8aYrAgAs/Y7+61NHdh3R1FINS3THtv7KrqmtGNF/ETmZcCUhkmBIiCQYEiIJhoRIggfuBAD4csdduvUSRfuuL59PXtMdm9Ov/3atyY4rCZEEQ0IkwZAQSTAkRBIMCZEEz27dgQI/KdHUPvm7A7OM1t4Q99zPf647Mv23ZxbSVsLiSkIkwZAQSTAkRBIMCZEED9zvQJc2an83Zhr037Hm7wf/VlNb1vGZ7tjZ71dNblxJiCQYEiIJhoRIgiEhkmBIiCR4dus2lpKVpVt//G9Oa2q+4ITu2NFffFdTUwK/W1hjSYYrCZEEQ0IkwZAQSTAkRBI8cL+NXfznH+rWj698VVPbdLFad6zy4Z11kK6HKwmRBENCJMGQEEkwJEQSDAmRBM9u3Sa8/3C/pnZ+yyHdsb+fntLUrvzrKt2xCkYW1thtgCsJkQRDQiTBkBBJMCREEjxwTzJL77Lq1mt3v6upKQb9f97HPntcU/uL/+TlJ7PhSkIkwZAQSTAkRBIMCZFERCFxOp0oKSlBVlYWcnNzUVVVhYGBgbAxExMTcDgcWLFiBTIzM1FdXQ2PxxPVpokWU0Rntzo7O+FwOFBSUoLp6Wns2rULGzZsQH9/PzIyMgAAO3bswAcffIDW1laoqoqamhps3rwZH3/8cUwmcDszLNX+8xQd/6Pu2J9mfqup/bs/V3esebf2d2Mwwt7uJBGFpKOjI+zr5uZm5Obmoq+vDw8++CC8Xi/eeOMNHD16FA8//DAAoKmpCffccw96enpw//3a64uIEt2Cjkm8Xi8AICcnBwDQ19eHqakp2O320Jg1a9agoKAA3d3duvsIBALw+XxhG1EimXdIgsEgamtrsX79eqxduxYA4Ha7YTQaYTKZwsaazWa43W7d/TidTqiqGtry8/Pn2xJRTMw7JA6HAxcuXEBLS8uCGqivr4fX6w1tQ0NDC9ofUbTN67KUmpoaHD9+HF1dXVi16uZ9CBaLBZOTkxgbGwtbTTweDywWi+6+FEWBouh/gMwdr+gHmtK/5P56zk9v/MVPdeumz/T/9CV9Ea0kQgjU1NSgra0NJ06cQGFhYdjjxcXFSE1NhcvlCtUGBgZw6dIl2Gy26HRMtMgiWkkcDgeOHj2KY8eOISsrK3Scoaoq0tPToaoqnn76adTV1SEnJwfZ2dnYvn07bDYbz2xR0oooJK+99hoAoLy8PKze1NSEJ554AgBw4MABpKSkoLq6GoFAABUVFXj1Ve2boREli4hCIoT8oyPT0tLQ2NiIxsbGeTdFlEh47RaRBG+6SgBL7v2+bn1by7E57+PeNx2a2upf98y7J7qJKwmRBENCJMGQEEkwJEQSPHBPAF8+v1y3Xrls7ldErzo1qS3O4ZQ9yXElIZJgSIgkGBIiCYaESIIhIZLg2a1FNlFZqqm5Kn85y+hlsW2G5oQrCZEEQ0IkwZAQSTAkRBI8cF9kw+uXaGoFS+d+gD7bW5em+rSXpfCilOjgSkIkwZAQSTAkRBIMCZEEQ0IkwbNbCcz57b2aWnfFat2xYuS/YtzNnYsrCZEEQ0IkwZAQSTAkRBI8cF9k392p/QCdH+/86wj2oP+xehQ7XEmIJBgSIgmGhEiCISGSSLgD9xufpjWNKd4QQTEzjSkAc/v0toQLid/vBwCcxodx7oTuBH6/H6qq3nKMQcwlSosoGAxieHgYWVlZ8Pv9yM/Px9DQELKzs+PdWlT5fD7OLY6EEPD7/bBarUhJufVRR8KtJCkpKVi1ahUAwGAwAACys7MT9oe9UJxb/MhWkBt44E4kwZAQSSR0SBRFwZ49e6AoSrxbiTrOLXkk3IE7UaJJ6JWEKBEwJEQSDAmRBENCJJHQIWlsbMTq1auRlpaGsrIynDlzJt4tRayrqwuVlZWwWq0wGAxob28Pe1wIgYaGBuTl5SE9PR12ux0XL16MT7MRcDqdKCkpQVZWFnJzc1FVVYWBgYGwMRMTE3A4HFixYgUyMzNRXV0Nj8cTp47nL2FD8u6776Kurg579uzBJ598gqKiIlRUVGB0dDTerUVkfHwcRUVFaGxs1H183759OHToEF5//XX09vYiIyMDFRUVmJiYWOROI9PZ2QmHw4Genh589NFHmJqawoYNGzA+Ph4as2PHDrz//vtobW1FZ2cnhoeHsXnz5jh2PU8iQZWWlgqHwxH6emZmRlitVuF0OuPY1cIAEG1tbaGvg8GgsFgsYv/+/aHa2NiYUBRFvPPOO3HocP5GR0cFANHZ2SmEuD6P1NRU0draGhrzxRdfCACiu7s7Xm3OS0KuJJOTk+jr64Pdbg/VUlJSYLfb0d2tvUc8WQ0ODsLtdofNU1VVlJWVJd08vV4vACAnJwcA0NfXh6mpqbC5rVmzBgUFBUk3t4QMyeXLlzEzMwOz2RxWN5vNcLtvnzdCuDGXZJ9nMBhEbW0t1q9fj7Vr1wK4Pjej0QiTyRQ2NtnmBiTgVcCUfBwOBy5cuIDTp0/Hu5WYSMiVZOXKlViyZInmTIjH44HFYolTV9F3Yy7JPM+amhocP34cJ0+eDN3iAFyf2+TkJMbGxsLGJ9PcbkjIkBiNRhQXF8PlcoVqwWAQLpcLNpstjp1FV2FhISwWS9g8fT4fent7E36eQgjU1NSgra0NJ06cQGFhYdjjxcXFSE1NDZvbwMAALl26lPBz04j3mYPZtLS0CEVRRHNzs+jv7xfbtm0TJpNJuN3ueLcWEb/fL86dOyfOnTsnAIhXXnlFnDt3Tnz99ddCCCFefvllYTKZxLFjx8T58+fFpk2bRGFhobh27VqcO7+15557TqiqKk6dOiVGRkZC29WrV0Njnn32WVFQUCBOnDghzp49K2w2m7DZbHHsen4SNiRCCHH48GFRUFAgjEajKC0tFT09PfFuKWInT54UuP6WFmHb1q1bhRDXTwPv3r1bmM1moSiKeOSRR8TAwEB8m54DvTkBEE1NTaEx165dE88//7xYvny5WLZsmXj00UfFyMhI/JqeJ14qTySRkMckRImEISGSYEiIJBgSIgmGhEiCISGSYEiIJBgSIgmGhEiCISGSYEiIJBgSIon/BemDrL4B7kYBAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "overfit_image = MNIST_TEST[0][0]\n",
    "plt.figure(figsize=(2,2))\n",
    "plt.imshow(overfit_image.reshape(28, 28, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5bdd8e76",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = torch.nn.CrossEntropyLoss()\n",
    "losses = []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d21a9bf4",
   "metadata": {},
   "source": [
    "You can run the following block multiple times to train the same model without restarting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6e353cfc",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/200 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGdCAYAAABO2DpVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAtI0lEQVR4nO3de3xU9Z3/8fdcMpMQcgFCbhDuCioXASFLvQuirOsF3YqUrmitFotWq/XHpr+tlj62G1Z/a23VotvV4q5VkW3FrfWyIAIqFyGQKiiRIJdgEu65kJDbzPf3R5KBCCSZZOac5OT1fDzmQXLmTOZzPJB5+z2f8/26jDFGAAAAEeC2uwAAAOAcBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxXqvfMBgMqri4WAkJCXK5XFa/PQAA6ABjjCorK5WZmSm3++zjEpYHi+LiYmVlZVn9tgAAIAKKioo0cODAsz5vebBISEiQ1FhYYmKi1W8PAAA6oKKiQllZWaHP8bOxPFg0X/5ITEwkWAAA0M201cZA8yYAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYggUAAIgYyxchi5Yn/7dA5SfqNf/KEUpNjLW7HAAAeiTHjFi8uqlIL63fq8PH6+wuBQCAHssxwcLnaTyUukDQ5koAAOi5HBMs/N6mYNFAsAAAwC6OCRY+ggUAALZzXrAIBGyuBACAnss5wcLDiAUAAHZzTrBoGrGoJVgAAGAbxwSLGEYsAACwnWOCxckeC4IFAAB2cV6wYMQCAADbOCZY+JsuhdQzYgEAgG0cEywYsQAAwH4ECwAAEDHOCRZNl0JquRQCAIBtnBMsGLEAAMB2BAsAABAxBAsAABAxzgkWHibIAgDAbs4JFoxYAABgO+cECybIAgDAds4JFqxuCgCA7RwXLLgUAgCAfZwTLGjeBADAds4JFoxYAABgO4IFAACIGMcEC7+XSyEAANgtrGDx85//XC6Xq8Vj1KhR0aotLDEeRiwAALCbN9wXXHDBBVq5cuXJH+AN+0dEBZdCAACwX9ipwOv1Kj09PRq1dIqPEQsAAGwXdo/Fzp07lZmZqWHDhmnOnDnat29fNOoKm48eCwAAbBfWiEV2draWLFmikSNHqqSkRAsXLtSll16qbdu2KSEh4Yyvqa2tVW1tbej7ioqKzlV8FqcGC2OMXC5XVN4HAACcXVjBYsaMGaGvx44dq+zsbA0ePFivv/667rrrrjO+Jjc3VwsXLuxcle3g93gkScZIDUGjGA/BAgAAq3XqdtPk5GSde+65KiwsPOs+OTk5Ki8vDz2Kioo685Zn1TxiIdFnAQCAXToVLI4fP65du3YpIyPjrPv4/X4lJia2eEQDwQIAAPuFFSx+8pOfaM2aNdqzZ4/WrVunmTNnyuPxaPbs2dGqr908bpc87sbLHzRwAgBgj7B6LPbv36/Zs2fryJEj6t+/vy655BJt2LBB/fv3j1Z9YfF53DoRDDBiAQCATcIKFq+99lq06oiIGI9LJ+qlWoIFAAC2cMxaIZLk8zbeGcKIBQAA9nBUsGAhMgAA7OWoYNF8Z0g9wQIAAFs4K1iwXggAALZyVrBghVMAAGzlyGDBXSEAANjDWcHCQ/MmAAB2claw4FIIAAC2IlgAAICIcVawCN0VErC5EgAAeiZnBQsmyAIAwFbOChae5gmyjM2VAADQMzkrWHC7KQAAtnJksKB5EwAAexAsAABAxDgrWIQmyOKuEAAA7OCsYMGIBQAAtnJUsPATLAAAsJWjgkUMa4UAAGArRwULLoUAAGAvZwULD/NYAABgJ2cFC2/zzJsECwAA7ODIYMGlEAAA7OHMYMGIBQAAtnBUsPB7GLEAAMBOjgoWXAoBAMBeBAsAABAxzgwW9FgAAGALRwWLGOaxAADAVo4KFj6aNwEAsJWjgoWfCbIAALCVo4JFc49F0EgNhAsAACznyGAh0cAJAIAdnBUsPKcEC/osAACwnKOChdfjltvV+DXBAgAA6zkqWEgnL4dwyykAANZzXrDwMEkWAAB2cV6wYFpvAABs47xgwSRZAADYxnnBgvVCAACwjWODRT0jFgAAWM6xwaKWEQsAACznvGBBjwUAALZxXrDgrhAAAGzjwGDhkUSwAADADs4LFkyQBQCAbRwXLPxcCgEAwDaOCxYxnsZVyAgWAABYz3HBggmyAACwj+OCRUxTj0U9wQIAAMs5LliEZt4kWAAAYLlOBYtFixbJ5XLpwQcfjFA5necLjVgYmysBAKDn6XCw2LRpk55//nmNHTs2kvV0WgwzbwIAYJsOBYvjx49rzpw5+t3vfqc+ffpEuqZOiWEeCwAAbNOhYDF//nxdd911mjZtWpv71tbWqqKiosUjmmK8jbebsropAADW84b7gtdee01btmzRpk2b2rV/bm6uFi5cGHZhHeXjrhAAAGwT1ohFUVGRHnjgAf3hD39QbGxsu16Tk5Oj8vLy0KOoqKhDhbbXybtCaN4EAMBqYY1Y5OXl6eDBg5owYUJoWyAQ0Nq1a/XMM8+otrZWHo+nxWv8fr/8fn9kqm2H5h6LWi6FAABgubCCxdSpU/XZZ5+12HbnnXdq1KhRWrBgwWmhwg5MkAUAgH3CChYJCQkaPXp0i23x8fHq16/fadvtwgRZAADYx3kzbzYtQkawAADAemHfFfJNq1evjkAZkcMEWQAA2MdxIxYnJ8jirhAAAKzmuGBBjwUAAPZxXLDgrhAAAOzjuGDho8cCAADbOC5YhNYKYcQCAADLOS9YMGIBAIBtHBcsTi5Cxl0hAABYzXnBwtt8uykjFgAAWM1xwaL5UkggaBQIMmoBAICVHBgsXKGvaeAEAMBajgsWzZdCJIIFAABWc1ywiHGfGiy4FAIAgJUcFyzcbpe87sbLIdxyCgCAtRwXLCSm9QYAwC6ODBbccgoAgD0cGSwYsQAAwB6ODBY+Dz0WAADYwZHBIsbLiAUAAHZwZrAILUTG7aYAAFjJkcHCR48FAAC2cGSwaL4UQo8FAADWcmSwaG7eZMQCAABrOTJYhHosCBYAAFjKkcHCF7orhOZNAACs5MhgwQRZAADYw5HBwueheRMAADs4MljE0LwJAIAtHBksWIQMAAB7ODJYhHosmHkTAABLOTpY1AUCNlcCAEDP4shgwe2mAADYw5nBgrtCAACwhSODBfNYAABgD2cGC2/j7aaMWAAAYC1HBguWTQcAwB6ODBYnL4XQvAkAgJUcGSyYIAsAAHs4MljQvAkAgD0cGixo3gQAwA6ODBY0bwIAYA9nBotQjwXNmwAAWMmRweLkImSMWAAAYCVHBwvuCgEAwFqODBa+ppk36bEAAMBazgwWHo8kLoUAAGA1RwaL0FohNG8CAGApZwaL0LLpAZsrAQCgZ3FksPCxVggAALZwZLBgSm8AAOzhyGDRPEFWQ9AoGGTUAgAAq4QVLBYvXqyxY8cqMTFRiYmJmjJlit55551o1dZhzWuFSMxlAQCAlcIKFgMHDtSiRYuUl5enzZs366qrrtKNN96o7du3R6u+Dmm+FCJxOQQAACt5w9n5+uuvb/H9L3/5Sy1evFgbNmzQBRdcENHCOsPXIlhwKQQAAKuEFSxOFQgEtGzZMlVVVWnKlCln3a+2tla1tbWh7ysqKjr6lu3mdrvkdbvUEDSMWAAAYKGwmzc/++wz9e7dW36/X/PmzdMbb7yh888//6z75+bmKikpKfTIysrqVMHtdXIuC4IFAABWCTtYjBw5Uvn5+dq4caPuvfdezZ07V59//vlZ98/JyVF5eXnoUVRU1KmC26u5gZPmTQAArBP2pRCfz6cRI0ZIkiZOnKhNmzbp17/+tZ5//vkz7u/3++X3+ztXZQc033LKpRAAAKzT6XksgsFgix6KriI0+2YDzZsAAFglrBGLnJwczZgxQ4MGDVJlZaVeeeUVrV69Wu+991606uuwmKYRCy6FAABgnbCCxcGDB3X77berpKRESUlJGjt2rN577z1dffXV0aqvw2jeBADAemEFixdeeCFadUQc64UAAGA9R64VIkm+prtCCBYAAFjHucGCu0IAALCcY4NF86WQWnosAACwjOODBWuFAABgnR4QLBixAADAKo4NFn56LAAAsJxjg0VorRB6LAAAsIyDgwUzbwIAYDXnBgsva4UAAGA1xwYLH82bAABYzrnBguZNAAAs59hg0dy8yQRZAABYx8HBghELAACsRrAAAAAR49hgcXKCLO4KAQDAKo4NFqF5LOixAADAMs4PFlwKAQDAMg4OFkzpDQCA1RwbLGJjPJKk2oaAzZUAANBzODZYxDUFixP1jFgAAGAVxwaL0IhFPSMWAABYxcHBovHQThAsAACwjIODReOIRQ3BAgAAyzg+WJyoI1gAAGAVxwaLOF/TiAW3mwIAYBnHBotY78mZNwNBpvUGAMAKjg0WzSMWEnNZAABgFccGi1jvyWBRw1wWAABYwrHBwu12yeflllMAAKzk2GAhneyz4JZTAACs4ehg0dxnwS2nAABYw9HBgoXIAACwlrODhbd5xILmTQAArODsYOFjWm8AAKzk7GDBXSEAAFjK0cEijhELAAAs5ehg0dxjQbAAAMAajg4WJ0csaN4EAMAKjg4WsTH0WAAAYCWHBwsuhQAAYKUeEiy4FAIAgBWcHSyaJ8hixAIAAEs4OljE+RoPr5ZgAQCAJRwdLJovhTBiAQCANXpEsKB5EwAAa/SIYMGIBQAA1nB0sIjjrhAAACzl6GDRPEEWl0IAALCGo4NFHD0WAABYytHBgh4LAACsFVawyM3N1aRJk5SQkKDU1FTddNNNKigoiFZtnXbyUgg9FgAAWCGsYLFmzRrNnz9fGzZs0IoVK1RfX6/p06erqqoqWvV1CrebAgBgLW84O7/77rstvl+yZIlSU1OVl5enyy67LKKFRUJzsKhtCCoYNHK7XTZXBACAs4UVLL6pvLxcktS3b9+z7lNbW6va2trQ9xUVFZ15y7A0N29KjeEizudpZW8AANBZHW7eDAaDevDBB3XxxRdr9OjRZ90vNzdXSUlJoUdWVlZH3zJssacECxo4AQCIvg4Hi/nz52vbtm167bXXWt0vJydH5eXloUdRUVFH3zJsHrdLPg9zWQAAYJUOXQq577779NZbb2nt2rUaOHBgq/v6/X75/f4OFRcJ/hi36gJBRiwAALBAWCMWxhjdd999euONN7Rq1SoNHTo0WnVFDJNkAQBgnbBGLObPn69XXnlFb775phISElRaWipJSkpKUlxcXFQK7CxuOQUAwDphjVgsXrxY5eXluuKKK5SRkRF6LF26NFr1dRqTZAEAYJ2wRiyMMdGqI2qaL4WcqGPEAgCAaHP0WiGS5G++FNJAsAAAINocHywYsQAAwDqODxahHosGeiwAAIg2xweL5hGLWu4KAQAg6hwfLGK5FAIAgGV6TLCgeRMAgOjrMcHiRB09FgAARJvjg0UcIxYAAFjG8cEidFcIPRYAAERdDwgWjFgAAGAVxwcLJsgCAMA6jg8WfhYhAwDAMo4PFqERCybIAgAg6hwfLEI9FqcEi4qaei3f+rWO1zbYVRYAAI7k+GAR5zs9WPxm5U49uDRfs55fryPHa+0qDQAAx3F8sIj1NgeLkz0WH+48LEnaXlyhW59fr9LyGltqAwDAaRwfLOJ8zaubNo5YHK2qU8GBSklSWqJfuw5V6Z+Wf2ZbfQAAOInjg4W/acSiui4gY4w+2X1UknRuWm/9ds4ESdKWfWV2lQcAgKM4Plj0T/Ar3udRXUNQG746qg1fHZEkZQ/tp/MzkuRyNY5i0GsBAEDnOT5YxMZ4dMOFmZKkpZv2aWPTiEX2sL6K83k0sE+cJOnLA8dtqxEAAKdwfLCQpNsmDZIkvb2tVDtKKyRJk4f2lSSdm5ogSSo8WGlPcQAAOEiPCBZjByZpVHqC6hqCMkYa1j9eqQmxkqQRab0lMWIBAEAk9Ihg4XK5NHvyoND32UP7hb5uHrHYyYgFAACd1iOChSTddOEA+b2Nh/s3w/qGtp/TNGJReJARCwAAOqvHBIukXjH6p+vO04zR6br6/LTQ9uH9G4PF4eN1OlpVZ1d5AAA4Qo8JFpL0D1OGaPF3J6qXzxvaFu/3hu4M2XmAyyEAAHRGjwoWZ3NOauOoxU4uhwAA0CkEC0nnpjU1cDJiAQBApxAsJI1gxAIAgIggWEg6p3nEgmABAECnECzU2GPhckmHKmt1oIIl1AEA6CiChRrvDBkzIEmS9NHOwzZXAwBA90WwaHLxiBRJ0seFBAsAADqKYNHk0qZg8VHhYRljbK4GAIDuiWDRZMLgPvJ73TpYWUsTJwAAHUSwaBIb4wktpf4hfRYAAHQIweIUl9BnAQBApxAsTnHJOY3BYsNXR1RQWqndh6votwAAIAwEi1Ocl56ovvE+VdcFdM1Ta3Xl/1utFz7abXdZAAB0GwSLU7jdLt196TAlxnoV7/NIklZ8fsDmqgAA6D4IFt9w7xXD9enPr9Hy+RdLkv66v0z1gaDNVQEA0D0QLM5ieP/eSoqLUU19UF+UVNhdDgAA3QLB4izcbpcmDEqWJG3ec8zeYgAA6CYIFq2YOLiPJClvH8ECAID2IFi0YuLgxgmztuwlWAAA0B4Ei1aMy0qSx+1SSXmNvi47YXc5AAB0eQSLVvTyeXV+RqIkKY9RCwAA2kSwaENzn8XmPUdtrgQAgK6PYNGG5mDxn+v36tqn1uqXf/lca788pJr6gM2VAQDQ9XjtLqCru3JUqiYP6atP9hzVjtJK7Sit1O8+3K2EWK9evGOSJg3pa3eJAAB0GWGPWKxdu1bXX3+9MjMz5XK5tHz58iiU1XX09nv1+rwpyvunafrN7PH69sSBSk+MVWVNg3706laVVdfZXSIAAF1G2MGiqqpK48aN07PPPhuNerqsfr39umFcpp749ji9//DlGpoSr5LyGi3446esgAoAQJOwg8WMGTP0z//8z5o5c2Y06ukW4v1ePT17vGI8Lr23/YCWbiqyuyQAALqEqDdv1tbWqqKiosXDCUYPSNIj14yUJD3+XoEqauptrggAAPtFPVjk5uYqKSkp9MjKyor2W1rmexcP1fD+8TpaVadnPyi0uxwAAGwX9WCRk5Oj8vLy0KOoyDmXDbwet376t+dJkn7/0R4VHa22uSIAAOwV9WDh9/uVmJjY4uEkV41K1beG91NdIKj/u3ybyk9wSQQA0HMxQVYnuVwu/fRvz5PX7dLaLw9p2pNr9Oe/FisY5E4RAEDPE3awOH78uPLz85Wfny9J2r17t/Lz87Vv375I19ZtjB6QpFfu/hsN6x+vQ5W1uv/Vrbrq31brv9bvUYCAAQDoQVwmzEkYVq9erSuvvPK07XPnztWSJUvafH1FRYWSkpJUXl7uuMsiNfUBLV69Sy9+vFuVNQ2SpB9cNkw5TX0YAAB0V+39/A47WHSWk4NFs6raBv3Xhr1a9M4OuVzSf8+boomDmfobANB9tffzmx6LKIj3ezXv8uG6ZcJAGSM9/PpfVV3XYHdZAABEHcEiih69/nylJ8Zqz5FqzX3xE2386ojdJQEAEFUEiyhKiovRv906Tj6PW5v2HNOsf9+gm3/7sV77ZJ8qmakTAOBA9FhYYP+xav129S4t21yk+kDjf+6EWK/+/R8u0pTh/WyuDgCAttG82QUdqKjRG1u/1uubivTV4Sr5vW49992JunJUqt2lAQDQKpo3u6C0xFjNu3y43n7gUk07L1W1DUHd81+b9d72UrtLAwAgIggWNoiN8Wjxdyfq78ZmqD5gdN8rW/TBjoN2lwUAQKcRLGwS43HrqVkX6roxjeHiBy/nafnWr5mpEwDQrREsbOT1uPXUbRfq6vPTVNcQ1INL83X5Ex/opXV7VB8I2l0eAABhI1jYLMbj1jPfGa8fTT1HfXrFaP+xE3rsf7br2qfWas2Xh+wuDwCAsHBXSBdyoi6g/84r0lMrd+pIVZ0kad7lw/V/rhkpt9tlc3UAgJ6Mu0K6oTifR/8wZYg+eOQK3fGtIZKk59bs0ryX81RWXWdvcQAAtAMjFl3Y8q1f6//88VPVNQQVG+PWTRcO0G2TB2ncwCS5XIxgAACswwRZDpG395j+afk2fVFSEdqW1TdOfz8hSz+4fJhiYzw2VgcA6CkIFg5ijNGmPcf08oa9WvH5AZ2oD0iShqXE61//fqwmDWFJdgBAdBEsHKq6rkHvbS9V7ts7dLCyVi6XdM+lw/Tw9JHyeWmZAQBEB82bDtXL59XM8QO14qHLdetFA2WM9Pzar3Tz4o+14asjsjgnAgDQAiMW3dx720u14I+fqqy6cRn2YSnxmjUpS7dMHKiU3n6bqwMAOAWXQnqQ0vIa/fr9L/VmfrGq6xr7L2I8Ls0YnaGHp5+rwf3iba4QANDdESx6oOO1DfrzX4v12if79Nf95ZIkn8etOy4eou9dPFTpSbE2VwgA6K4IFj3ctq/L9a/v7tCHOw9Lktwu6apRqfr+pcP0N8P62VwdAKC7IVhAxhitLjik364u1KY9x0LbvzW8nx6efq4mDuY2VQBA+xAs0ELhwUotWbdHSzcVqT7QeMovO7e/Hpg6QhMG9WEmTwBAqwgWOKP9x6r1zKpCLcvbr0Cw8dRn9Y3TjNEZuuuSoUpLpA8DAHA6ggVatfdIlZ5eVai3Pi1WTX1QktTb79WPrz5Xc6cMltfDFCcAgJMIFmiXE3UBrfnyoJ5b85Xyi8okSYP79dK8y4fr5gkD5PeyFgkAgGCBMAWDRks3F+nxd3foWNNkW33jfZoxOl0zxw/QxMH0YQBAT0awQIdU1Tbo1U/26T8+3K3SiprQ9gmDknX/VefoipH9CRgA0AMRLNApDYGg1u06ojfzi/XnT4tV19DYhzGkXy99+6Is/f3EgTR6AkAPQrBAxBysqNF/fLRbf9iwV1VNU4a7XdIVI1N160VZmnpeqmJo9gQARyNYIOKqahv09mclen1zUYsJt1J6+zRz/ADNmpSlEakJNlYIAIgWggWiateh41q2eb/+uGW/DlXWhraPH5SsO741RNeNyeCWVQBwEIIFLFEfCGp1wSG9vrlIq3YcDE26NahvL33v4iGaOWGgkuJibK4SANBZBAtY7mBljV7dWKQl63aHblmNjXFr5vgBmn/lCA3s08vmCgEAHUWwgG2q6xr033n79fKGvfrywHFJUozHpe9MHqR7Lh+uAclxNlcIAAgXwQK2M8Zo055j+vX7X+rjwiOSJI/bpb8dk6HvXzJU47KS7S0QANBuBAt0KR8XHtZvVxeGAoYkTRrSR3ddMkxXn58mj5tJtwCgKyNYoEv6vLhC//HRV/rzX4tDy7c3N3p++6Isxfu9NlcIADgTggW6tAMVNfrP9Xv08oZ9Kj/R2OiZGOvV7OxB+m72YGX1pdETALoSggW6heq6Bv1xy9d68aPd2n24KrT9khEpunVSlqafn6bYGFZYBQC7ESzQrQSDRqt2HNRL6/foo8LDav5bmdwrRjPHD9BtkwZpZDqzegKAXQgW6LaKjlZrWd5+LdtcpJLykyusXpiVrNsmZenvxmWqN70YAGApggW6vUDQaO3OQ1r6SZFWfnFADU2zevbyeXT92EzNmpyl8VnJLOMOABYgWMBRDlXW6k9b9mvppiJ9dUovxrlpvTVr0iDNHD9AfeN9NlYIAM5GsIAjGWO0ee8xvfZJkf7yWbFq6oOSJK/bpW+NSNHfjcnQ9AvSlNyLkAEAkUSwgONV1NTrf/KLtXRTkT77ujy03et26eIRKbp2dLqmnZem/gl+G6sEAGcgWKBH+erQcb39WYne+rREO0orQ9tdLmnCoD66+vw0TTsvTcP7x9OTAQAdQLBAj1V48Lje+axEK744oE/3l7d4LiMpVhePSNElI1L0rRH9lJoQa1OVANC9ECwASSXlJ7Tyi4P63+2l2rj7qOoagi2eH5oSr/FZyRo/KFnjB/XRyPQExXjcNlULAF0XwQL4hpr6gDbvOaaPCg/r48LD2lZcrm/+7Y+NcWvMgCRdmJWs8zMTNSo9UcP795bPS9gA0LNFNVg8++yzeuKJJ1RaWqpx48bp6aef1uTJkyNaGBBt5dX1yt9fpq37jmnrvsY/K2oaTtvP63ZpeP/eGpHWW8NS4jWkX7yGpMRraEq8+vSKoWcDQI8QtWCxdOlS3X777XruueeUnZ2tp556SsuWLVNBQYFSU1MjVhhgtWDQaPeRKm3dV6a/FpWpoLRSX5RWqPIMYaNZXIxHGcmxykyKU2ZyrDKa/kxPilO/eJ/6Nj1Y7wRAdxe1YJGdna1JkybpmWeekSQFg0FlZWXp/vvv1z/+4z9GrDCgKzDGqLi8RjtKKvTVoSp9dbhKew5Xac+RqhbTjbell8+jvvE+9Yv3KamXT739HvX2exXv96p30yPe71VCrFfxPq96+Tzyx7jl83jk87rl87rlb/rT53XL52n8ntESAFZp7+d3WAsu1NXVKS8vTzk5OaFtbrdb06ZN0/r168/4mtraWtXW1rYoDOguXC6XBiTHaUBynKae1/K5mvqASstrVFx2QsXlNSopO6Hi8hMqLqvRgYoaHa2q07HqOtUHjKrrAqquO6H9x05EtD6fpzFouF2Sx+2Sx+2Wxy15XC653a7GbU1fe90uuV2N29xulzxNr3G7XHK5JJea/jzl6+b/BpLkUvNzjduav1fz60Lfn/6zdIbXub7xs8+qlSddrb9SreWutiJZ66/txPu2mQXPvkNXPJ5IszoqOzWcPzz9XCXExtjy3mEFi8OHDysQCCgtLa3F9rS0NO3YseOMr8nNzdXChQs7XiHQRcXGeDQkpbHf4myMMaqsbdDR43U6UlWnY1V1KjtRr6raBh1veoS+rmlQVV3jn9V1AdUFgqpraHzUNv1ZF2h5V0td4PRtAPDDK4d3j2DRETk5OXrooYdC31dUVCgrKyvabwt0CS6XS4mxMUqMjWk1gLSXMUZ1gVOCRtMjYIyCQaOGoFEgaBQ0p/7ZuKBbIGhC+7X42hgZI5mmn9/4PpJR0/ZTngtdNz31+W/u3/QDQttP/fqU9zj1fc5+vK38t2jzv1Vrr2391Z25V661q8tt/dzWno7W8bR5qNbeONimrlVNoy72n0iS1Mtn3wrQYb1zSkqKPB6PDhw40GL7gQMHlJ6efsbX+P1++f1MqQxEgsvlkt/rkd9LMyiArimsm/N9Pp8mTpyo999/P7QtGAzq/fff15QpUyJeHAAA6F7CHit56KGHNHfuXF100UWaPHmynnrqKVVVVenOO++MRn0AAKAbCTtYzJo1S4cOHdKjjz6q0tJSXXjhhXr33XdPa+gEAAA9D1N6AwCANrX385sFEAAAQMQQLAAAQMQQLAAAQMQQLAAAQMQQLAAAQMQQLAAAQMQQLAAAQMQQLAAAQMQQLAAAQMRYvq5q80SfFRUVVr81AADooObP7bYm7LY8WFRWVkqSsrKyrH5rAADQSZWVlUpKSjrr85avFRIMBlVcXKyEhAS5XK6I/dyKigplZWWpqKjIsWuQOP0YnX58EsfoBE4/PoljdIJoHJ8xRpWVlcrMzJTbffZOCstHLNxutwYOHBi1n5+YmOjIvySncvoxOv34JI7RCZx+fBLH6ASRPr7WRiqa0bwJAAAihmABAAAixjHBwu/367HHHpPf77e7lKhx+jE6/fgkjtEJnH58EsfoBHYen+XNmwAAwLkcM2IBAADsR7AAAAARQ7AAAAARQ7AAAAAR45hg8eyzz2rIkCGKjY1Vdna2PvnkE7tL6pDc3FxNmjRJCQkJSk1N1U033aSCgoIW+1xxxRVyuVwtHvPmzbOp4vD9/Oc/P63+UaNGhZ6vqanR/Pnz1a9fP/Xu3Vu33HKLDhw4YGPF4RkyZMhpx+dyuTR//nxJ3fP8rV27Vtdff70yMzPlcrm0fPnyFs8bY/Too48qIyNDcXFxmjZtmnbu3Nlin6NHj2rOnDlKTExUcnKy7rrrLh0/ftzCo2hda8dYX1+vBQsWaMyYMYqPj1dmZqZuv/12FRcXt/gZZzr3ixYtsvhIzqytc3jHHXecVvu1117bYp/ufA4lnfHfpcvl0hNPPBHapyufw/Z8PrTn9+e+fft03XXXqVevXkpNTdUjjzyihoaGiNXpiGCxdOlSPfTQQ3rssce0ZcsWjRs3Ttdcc40OHjxod2lhW7NmjebPn68NGzZoxYoVqq+v1/Tp01VVVdViv7vvvlslJSWhx+OPP25TxR1zwQUXtKj/o48+Cj334x//WH/+85+1bNkyrVmzRsXFxbr55pttrDY8mzZtanFsK1askCR9+9vfDu3T3c5fVVWVxo0bp2efffaMzz/++OP6zW9+o+eee04bN25UfHy8rrnmGtXU1IT2mTNnjrZv364VK1borbfe0tq1a3XPPfdYdQhtau0Yq6urtWXLFv3sZz/Tli1b9Kc//UkFBQW64YYbTtv3F7/4RYtze//991tRfpvaOoeSdO2117ao/dVXX23xfHc+h5JaHFtJSYlefPFFuVwu3XLLLS3266rnsD2fD239/gwEArruuutUV1endevW6aWXXtKSJUv06KOPRq5Q4wCTJ0828+fPD30fCARMZmamyc3NtbGqyDh48KCRZNasWRPadvnll5sHHnjAvqI66bHHHjPjxo0743NlZWUmJibGLFu2LLTtiy++MJLM+vXrLaowsh544AEzfPhwEwwGjTHd//xJMm+88Ubo+2AwaNLT080TTzwR2lZWVmb8fr959dVXjTHGfP7550aS2bRpU2ifd955x7hcLvP1119bVnt7ffMYz+STTz4xkszevXtD2wYPHmx+9atfRbe4CDjT8c2dO9fceOONZ32NE8/hjTfeaK666qoW27rLOTTm9M+H9vz+fPvtt43b7TalpaWhfRYvXmwSExNNbW1tROrq9iMWdXV1ysvL07Rp00Lb3G63pk2bpvXr19tYWWSUl5dLkvr27dti+x/+8AelpKRo9OjRysnJUXV1tR3lddjOnTuVmZmpYcOGac6cOdq3b58kKS8vT/X19S3O56hRozRo0KBueT7r6ur08ssv63vf+16LRfe6+/k71e7du1VaWtrinCUlJSk7Ozt0ztavX6/k5GRddNFFoX2mTZsmt9utjRs3Wl5zJJSXl8vlcik5ObnF9kWLFqlfv34aP368nnjiiYgOMUfb6tWrlZqaqpEjR+ree+/VkSNHQs857RweOHBAf/nLX3TXXXed9lx3OYff/Hxoz+/P9evXa8yYMUpLSwvtc80116iiokLbt2+PSF2WL0IWaYcPH1YgEGjxH0mS0tLStGPHDpuqioxgMKgHH3xQF198sUaPHh3a/p3vfEeDBw9WZmamPv30Uy1YsEAFBQX605/+ZGO17Zedna0lS5Zo5MiRKikp0cKFC3XppZdq27ZtKi0tlc/nO+2XdVpamkpLS+0puBOWL1+usrIy3XHHHaFt3f38fVPzeTnTv8Hm50pLS5Wamtriea/Xq759+3bL81pTU6MFCxZo9uzZLRZ4+tGPfqQJEyaob9++WrdunXJyclRSUqInn3zSxmrb59prr9XNN9+soUOHateuXfrpT3+qGTNmaP369fJ4PI47hy+99JISEhJOu8zaXc7hmT4f2vP7s7S09Iz/Vpufi4RuHyycbP78+dq2bVuL/gNJLa5pjhkzRhkZGZo6dap27dql4cOHW11m2GbMmBH6euzYscrOztbgwYP1+uuvKy4uzsbKIu+FF17QjBkzlJmZGdrW3c9fT1dfX69bb71VxhgtXry4xXMPPfRQ6OuxY8fK5/PpBz/4gXJzc7v81NG33XZb6OsxY8Zo7NixGj58uFavXq2pU6faWFl0vPjii5ozZ45iY2NbbO8u5/Bsnw9dQbe/FJKSkiKPx3Na1+uBAweUnp5uU1Wdd9999+mtt97SBx980OYy89nZ2ZKkwsJCK0qLuOTkZJ177rkqLCxUenq66urqVFZW1mKf7ng+9+7dq5UrV+r73/9+q/t19/PXfF5a+zeYnp5+WjN1Q0ODjh492q3Oa3Oo2Lt3r1asWNHmctTZ2dlqaGjQnj17rCkwgoYNG6aUlJTQ30unnENJ+vDDD1VQUNDmv02pa57Ds30+tOf3Z3p6+hn/rTY/FwndPlj4fD5NnDhR77//fmhbMBjU+++/rylTpthYWccYY3TffffpjTfe0KpVqzR06NA2X5Ofny9JysjIiHJ10XH8+HHt2rVLGRkZmjhxomJiYlqcz4KCAu3bt6/bnc/f//73Sk1N1XXXXdfqft39/A0dOlTp6ektzllFRYU2btwYOmdTpkxRWVmZ8vLyQvusWrVKwWAwFKy6uuZQsXPnTq1cuVL9+vVr8zX5+flyu92nXULoDvbv368jR46E/l464Rw2e+GFFzRx4kSNGzeuzX270jls6/OhPb8/p0yZos8++6xFSGwOyeeff37ECu32XnvtNeP3+82SJUvM559/bu655x6TnJzcouu1u7j33ntNUlKSWb16tSkpKQk9qqurjTHGFBYWml/84hdm8+bNZvfu3ebNN980w4YNM5dddpnNlbffww8/bFavXm12795tPv74YzNt2jSTkpJiDh48aIwxZt68eWbQoEFm1apVZvPmzWbKlClmypQpNlcdnkAgYAYNGmQWLFjQYnt3PX+VlZVm69atZuvWrUaSefLJJ83WrVtDd0QsWrTIJCcnmzfffNN8+umn5sYbbzRDhw41J06cCP2Ma6+91owfP95s3LjRfPTRR+acc84xs2fPtuuQTtPaMdbV1ZkbbrjBDBw40OTn57f4t9ncSb9u3Trzq1/9yuTn55tdu3aZl19+2fTv39/cfvvtNh9Zo9aOr7Ky0vzkJz8x69evN7t37zYrV640EyZMMOecc46pqakJ/YzufA6blZeXm169epnFixef9vqufg7b+nwwpu3fnw0NDWb06NFm+vTpJj8/37z77rumf//+JicnJ2J1OiJYGGPM008/bQYNGmR8Pp+ZPHmy2bBhg90ldYikMz5+//vfG2OM2bdvn7nssstM3759jd/vNyNGjDCPPPKIKS8vt7fwMMyaNctkZGQYn89nBgwYYGbNmmUKCwtDz584ccL88Ic/NH369DG9evUyM2fONCUlJTZWHL733nvPSDIFBQUttnfX8/fBBx+c8e/l3LlzjTGNt5z+7Gc/M2lpacbv95upU6eeduxHjhwxs2fPNr179zaJiYnmzjvvNJWVlTYczZm1doy7d+8+67/NDz74wBhjTF5ensnOzjZJSUkmNjbWnHfeeeZf/uVfWnww26m146uurjbTp083/fv3NzExMWbw4MHm7rvvPu1/zrrzOWz2/PPPm7i4OFNWVnba67v6OWzr88GY9v3+3LNnj5kxY4aJi4szKSkp5uGHHzb19fURq5Nl0wEAQMR0+x4LAADQdRAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxBAsAABAxPx/I2+kWhb8NzsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.train()\n",
    "for epoch in tqdm(range(200), leave=False):\n",
    "    train_loss = 0.0\n",
    "    for img in [overfit_image]:\n",
    "        # Convert image to a signal with L = 784 and dimension 1\n",
    "        img = img.to(device).reshape(-1, 784, 1)\n",
    "        # Shift and pad with zero to generate input\n",
    "        x = torch.cat([torch.zeros(img.size(0), 1, 1, device=device), img[:, :-1, :]], dim=1)\n",
    "        # For each sample i, predict the next sample (i+1) from the slice between 0 to i\n",
    "        y_hat = model(x)\n",
    "        \n",
    "        y = torch.mul(img, 255.0).to(torch.int64).to(device)[:,:,0]\n",
    "        loss = criterion(y_hat.reshape(-1, 256), y.reshape(-1))\n",
    "\n",
    "        loss_val = loss.detach().cpu().item()\n",
    "        train_loss += loss_val\n",
    "        losses.append(loss_val)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "plt.plot(losses)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dd5d145",
   "metadata": {},
   "source": [
    "## Convolutional Sampling\n",
    "\n",
    "This one generates each pixel assuming that all previous pixels are predicted correctly, i.e., sample $i$ is predicted from ground truth values of samples $0$ to $i-1$ instead of generated samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "7f21780f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    img = overfit_image.to(device).reshape(-1, 784, 1)\n",
    "    x = torch.cat([torch.zeros(img.size(0), 1, 1, device=device), img[:, :-1, :]], dim=1)\n",
    "    y_hat = model(x)\n",
    "    sample = torch.argmax(y_hat, dim=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "1d1d19f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fadb4555390>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMkAAADICAYAAABCmsWgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAO00lEQVR4nO3df1CTd54H8HdQ8oD8eCJ6JKTCyt5213a9xTkKNGOvQ9ucjLvDSOV27M1cx/64um2Ds8jMdcSpeON1Nj29ra5K239aaHfO0qF74NT2mOlEhbELuFJbz9Iydo+t7EBC7S1JRAk/8r0/XOPlnge/BBKS6Ps18/zBJ988fL7omy/Pk+dJDEIIASKaVUq8GyBKdAwJkQRDQiTBkBBJMCREEgwJkQRDQiTBkBBJMCREEgwJkcTSWO24sbER+/fvh9vtRlFREQ4fPozS0lLp84LBIIaHh5GVlQWDwRCr9ugOJ4SA3++H1WpFSopkrRAx0NLSIoxGo3jzzTfF559/Lp555hlhMpmEx+ORPndoaEgA4MZtUbahoSHp/0mDENG/wLGsrAwlJSU4cuQIgOurQ35+PrZv346dO3fe8rlerxcmkwkP4MdYitRot0YEAJjGFE7jQ4yNjUFV1VuOjfqfW5OTk+jr60N9fX2olpKSArvdju7ubs34QCCAQCAQ+trv9/+5sVQsNTAkFCN/Xhrm8id91A/cL1++jJmZGZjN5rC62WyG2+3WjHc6nVBVNbTl5+dHuyWiBYn72a36+np4vd7QNjQ0FO+WiMJE/c+tlStXYsmSJfB4PGF1j8cDi8WiGa8oChRFiXYbRFET9ZXEaDSiuLgYLpcrVAsGg3C5XLDZbNH+dkQxF5PXSerq6rB161bcd999KC0txcGDBzE+Po4nn3wyFt+OKKZiEpItW7bgm2++QUNDA9xuN9atW4eOjg7NwTxRMojJ6yQL4fP5oKoqyrGJp4ApZqbFFE7hGLxeL7Kzs285Nu5nt4gSHUNCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSDAmRBENCJMGQEEkwJEQSMfuI6mT37TPaz1IpePwr3bFfjmrfLX8yoP9m33e9o60v++MV3bHBT/tv1SItEq4kRBIMCZEEQ0IkwZAQSTAkRBI8uzWLF/7pqKZWnfEn/cF/GcGOy7WlP0xf1R36q28eimDH8Xdm9DuaWsYvVd2xS119sW4nariSEEkwJEQSDAmRBENCJMED91kc2vWYptbwI/3fKcu/EJran+4x6I41/mhMU9u39j90xx7I69XUPriaqTv2J8v0L22Zq2tiUrfeG8jQ1MrTpvR3otPv97b8THfo911z7y3euJIQSTAkRBIMCZEEQ0IkEXFIurq6UFlZCavVCoPBgPb29rDHhRBoaGhAXl4e0tPTYbfbcfHixWj1S7ToIj67NT4+jqKiIjz11FPYvHmz5vF9+/bh0KFDeOutt1BYWIjdu3ejoqIC/f39SEtLi0rTiyHjPe2Zmoz35v787Ai+12FLuW79pfWrtfvt1L/xa1/59yL4jlpLrwV16xnnRzS1FV2/0R37V0adG8r+oH/zWTKJOCQbN27Exo0bdR8TQuDgwYN48cUXsWnTJgDA22+/DbPZjPb2djz2mPa0KlGii+oxyeDgINxuN+x2e6imqirKysrQ3d2t+5xAIACfzxe2ESWSqIbE7XYDAMzm8Hu+zWZz6LH/z+l0QlXV0Jafnx/NlogWLO5nt+rr6+H1ekPb0NBQvFsiChPVy1IsFgsAwOPxIC8vL1T3eDxYt26d7nMURYGiKNFsI+lMuz269YzfaOszs+wj471vo9jRTZ5/1L5rzA+N+v9t/u1/fqCprW76b92x0wtra1FFdSUpLCyExWKBy3Xzwhyfz4fe3l7YbNofNlEyiHgluXLlCr766uZpyMHBQXz66afIyclBQUEBamtr8dJLL+Huu+8OnQK2Wq2oqqqKZt9EiybikJw9exYPPXTzttK6ujoAwNatW9Hc3IwXXngB4+Pj2LZtG8bGxvDAAw+go6MjqV4jIfq/Ig5JeXk5hNBeGn6DwWDA3r17sXfv3gU1RpQo4n52iyjR8aYrAgAs/Y7+61NHdh3R1FINS3THtv7KrqmtGNF/ETmZcCUhkmBIiCQYEiIJhoRIggfuBAD4csdduvUSRfuuL59PXtMdm9Ov/3atyY4rCZEEQ0IkwZAQSTAkRBIMCZEEz27dgQI/KdHUPvm7A7OM1t4Q99zPf647Mv23ZxbSVsLiSkIkwZAQSTAkRBIMCZEED9zvQJc2an83Zhr037Hm7wf/VlNb1vGZ7tjZ71dNblxJiCQYEiIJhoRIgiEhkmBIiCR4dus2lpKVpVt//G9Oa2q+4ITu2NFffFdTUwK/W1hjSYYrCZEEQ0IkwZAQSTAkRBI8cL+NXfznH+rWj698VVPbdLFad6zy4Z11kK6HKwmRBENCJMGQEEkwJEQSDAmRBM9u3Sa8/3C/pnZ+yyHdsb+fntLUrvzrKt2xCkYW1thtgCsJkQRDQiTBkBBJMCREEjxwTzJL77Lq1mt3v6upKQb9f97HPntcU/uL/+TlJ7PhSkIkwZAQSTAkRBIMCZFERCFxOp0oKSlBVlYWcnNzUVVVhYGBgbAxExMTcDgcWLFiBTIzM1FdXQ2PxxPVpokWU0Rntzo7O+FwOFBSUoLp6Wns2rULGzZsQH9/PzIyMgAAO3bswAcffIDW1laoqoqamhps3rwZH3/8cUwmcDszLNX+8xQd/6Pu2J9mfqup/bs/V3esebf2d2Mwwt7uJBGFpKOjI+zr5uZm5Obmoq+vDw8++CC8Xi/eeOMNHD16FA8//DAAoKmpCffccw96enpw//3a64uIEt2Cjkm8Xi8AICcnBwDQ19eHqakp2O320Jg1a9agoKAA3d3duvsIBALw+XxhG1EimXdIgsEgamtrsX79eqxduxYA4Ha7YTQaYTKZwsaazWa43W7d/TidTqiqGtry8/Pn2xJRTMw7JA6HAxcuXEBLS8uCGqivr4fX6w1tQ0NDC9ofUbTN67KUmpoaHD9+HF1dXVi16uZ9CBaLBZOTkxgbGwtbTTweDywWi+6+FEWBouh/gMwdr+gHmtK/5P56zk9v/MVPdeumz/T/9CV9Ea0kQgjU1NSgra0NJ06cQGFhYdjjxcXFSE1NhcvlCtUGBgZw6dIl2Gy26HRMtMgiWkkcDgeOHj2KY8eOISsrK3Scoaoq0tPToaoqnn76adTV1SEnJwfZ2dnYvn07bDYbz2xR0oooJK+99hoAoLy8PKze1NSEJ554AgBw4MABpKSkoLq6GoFAABUVFXj1Ve2boREli4hCIoT8oyPT0tLQ2NiIxsbGeTdFlEh47RaRBG+6SgBL7v2+bn1by7E57+PeNx2a2upf98y7J7qJKwmRBENCJMGQEEkwJEQSPHBPAF8+v1y3Xrls7ldErzo1qS3O4ZQ9yXElIZJgSIgkGBIiCYaESIIhIZLg2a1FNlFZqqm5Kn85y+hlsW2G5oQrCZEEQ0IkwZAQSTAkRBI8cF9kw+uXaGoFS+d+gD7bW5em+rSXpfCilOjgSkIkwZAQSTAkRBIMCZEEQ0IkwbNbCcz57b2aWnfFat2xYuS/YtzNnYsrCZEEQ0IkwZAQSTAkRBI8cF9k392p/QCdH+/86wj2oP+xehQ7XEmIJBgSIgmGhEiCISGSSLgD9xufpjWNKd4QQTEzjSkAc/v0toQLid/vBwCcxodx7oTuBH6/H6qq3nKMQcwlSosoGAxieHgYWVlZ8Pv9yM/Px9DQELKzs+PdWlT5fD7OLY6EEPD7/bBarUhJufVRR8KtJCkpKVi1ahUAwGAwAACys7MT9oe9UJxb/MhWkBt44E4kwZAQSSR0SBRFwZ49e6AoSrxbiTrOLXkk3IE7UaJJ6JWEKBEwJEQSDAmRBENCJJHQIWlsbMTq1auRlpaGsrIynDlzJt4tRayrqwuVlZWwWq0wGAxob28Pe1wIgYaGBuTl5SE9PR12ux0XL16MT7MRcDqdKCkpQVZWFnJzc1FVVYWBgYGwMRMTE3A4HFixYgUyMzNRXV0Nj8cTp47nL2FD8u6776Kurg579uzBJ598gqKiIlRUVGB0dDTerUVkfHwcRUVFaGxs1H183759OHToEF5//XX09vYiIyMDFRUVmJiYWOROI9PZ2QmHw4Genh589NFHmJqawoYNGzA+Ph4as2PHDrz//vtobW1FZ2cnhoeHsXnz5jh2PU8iQZWWlgqHwxH6emZmRlitVuF0OuPY1cIAEG1tbaGvg8GgsFgsYv/+/aHa2NiYUBRFvPPOO3HocP5GR0cFANHZ2SmEuD6P1NRU0draGhrzxRdfCACiu7s7Xm3OS0KuJJOTk+jr64Pdbg/VUlJSYLfb0d2tvUc8WQ0ODsLtdofNU1VVlJWVJd08vV4vACAnJwcA0NfXh6mpqbC5rVmzBgUFBUk3t4QMyeXLlzEzMwOz2RxWN5vNcLtvnzdCuDGXZJ9nMBhEbW0t1q9fj7Vr1wK4Pjej0QiTyRQ2NtnmBiTgVcCUfBwOBy5cuIDTp0/Hu5WYSMiVZOXKlViyZInmTIjH44HFYolTV9F3Yy7JPM+amhocP34cJ0+eDN3iAFyf2+TkJMbGxsLGJ9PcbkjIkBiNRhQXF8PlcoVqwWAQLpcLNpstjp1FV2FhISwWS9g8fT4fent7E36eQgjU1NSgra0NJ06cQGFhYdjjxcXFSE1NDZvbwMAALl26lPBz04j3mYPZtLS0CEVRRHNzs+jv7xfbtm0TJpNJuN3ueLcWEb/fL86dOyfOnTsnAIhXXnlFnDt3Tnz99ddCCCFefvllYTKZxLFjx8T58+fFpk2bRGFhobh27VqcO7+15557TqiqKk6dOiVGRkZC29WrV0Njnn32WVFQUCBOnDghzp49K2w2m7DZbHHsen4SNiRCCHH48GFRUFAgjEajKC0tFT09PfFuKWInT54UuP6WFmHb1q1bhRDXTwPv3r1bmM1moSiKeOSRR8TAwEB8m54DvTkBEE1NTaEx165dE88//7xYvny5WLZsmXj00UfFyMhI/JqeJ14qTySRkMckRImEISGSYEiIJBgSIgmGhEiCISGSYEiIJBgSIgmGhEiCISGSYEiIJBgSIon/BemDrL4B7kYBAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(2, 2))\n",
    "plt.imshow(sample.cpu().reshape(28, 28, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d22e69",
   "metadata": {},
   "source": [
    "## Sampling\n",
    "\n",
    "The following method will autoregressively sample from the given model `m`. It can optionally use a priming signal to start with. When `maxp` is `True`, it will always choose the value with highest probability instead of random sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "71212e9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sample(m, priming_signal=None, maxp=False):\n",
    "    f = m.get_recurrent_runner()\n",
    "    # The input is padded with 0 at the start\n",
    "    u = f(torch.zeros(1, 1, device=device))\n",
    "\n",
    "    # Process the priming signal if given\n",
    "    if priming_signal is not None:\n",
    "        for s in priming_signal:\n",
    "            u = f(s.reshape(1, 1))\n",
    "        primed_size = priming_signal.size(0)\n",
    "    else:\n",
    "        primed_size = 0\n",
    "\n",
    "    # Generate the new part\n",
    "    Y = []\n",
    "    for _ in range(784 - primed_size):\n",
    "        if maxp:\n",
    "            p = torch.argmax(u, dim=-1)\n",
    "        else:\n",
    "            dist = torch.distributions.categorical.Categorical(probs=torch.nn.functional.softmax(u, dim=-1))\n",
    "            p = dist.sample()\n",
    "        p = p.reshape(1, 1) / 255.0\n",
    "        Y.append(p)\n",
    "        u = f(p)\n",
    "\n",
    "    generated = torch.cat(Y, dim=1).real.flatten()\n",
    "    # Red: generated part\n",
    "    # Green: given part\n",
    "    if priming_signal is not None:\n",
    "        priming_signal = priming_signal.flatten()\n",
    "        red = torch.cat([torch.zeros_like(priming_signal), generated], dim=0)\n",
    "        green = torch.cat([priming_signal, torch.zeros_like(generated)], dim=0)\n",
    "    else:\n",
    "        red = generated\n",
    "        green = torch.zeros_like(red)\n",
    "    blue = torch.zeros_like(green)\n",
    "    return torch.cat([color.reshape(28, 28, 1) for color in [red, green, blue]], dim=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "c0d26c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "def overfit_sample(start=0, maxp=False):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        img = overfit_image\n",
    "        img = img.to(device).reshape(-1, 784, 1)\n",
    "        sample = generate_sample(model, img[0, :start, :] if start > 0 else None, maxp).cpu()\n",
    "        plt.figure(figsize=(2,2))\n",
    "        plt.axis('off')\n",
    "        plt.imshow(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c53100",
   "metadata": {},
   "source": [
    "Let's sample some images.\n",
    "\n",
    "<b style=\"color: red;\">Red</b> parts are generated, <b style=\"color: green;\">green</b> parts are given in the priming signal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "78624d7b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAD+UlEQVR4nO3dMascZRTH4VljCiGNikIKY8AuTQhBEFSI2EQtzVfQJtoErNOnTOE3sBEECwkhoGAKtbCRGEOCWqhIIBCLKIgGNh8g55CZvbub/c99nvLw3r1v8eOFmbl7Z7FcLpcDhHnicW8AViFcIgmXSMIlknCJJFwiCZdIwiWScIn05NiFi8Vik/uAYRiGYeyDXCcukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukUa/dWcnnClm7zVr/yxm/zZrPylmt5u1PzdztsqJSyThEkm4RBIukYRLpMVy5MtTd+Jdvr8Ws6Mb+l33mvn1Df2+TfmjmF1o1n6/yY2M412+zJpwiSRcIgmXSFmPfKvHu8ebtT8Vs2PN2hPF7FSz9pVi9nuz9oVmPtb9Zn6nmB2e8Lm/NfMduDgby4lLJOESSbhEEi6RhEukrEe+W/R0M69uQHQX4y/vcQ/d373fKmY3mrXPFLMPmrUfP3JHm+eRL7MmXCIJl0jCJZKLs0DvFrNPm7U/FrM3mrV3V9vOWrk4Y9aESyThEkm4RBIukdxV2GHPN/NrE9ZW/27ts9W2sxXuKjBrwiWScIkkXCJlfct3nznbzJ8rZn81a2+uaS+7xolLJOESSbhEEi6RhEskj3x3xKvF7Ktm7cFidqpZe3Wl3Tw+Hvkya8IlknCJJFwieeS7I94uZtVF2DAMw5fF7Ns17iWBE5dIwiWScIkkXCIJl0juKmzZU838dDH7r1l7vpj9v9p2YjlxiSRcIgmXSMIlkouzLfuomVdv87ncrP1mTXtJ5sQlknCJJFwiCZdIwiWSb/lu0DvF7PNm7T/F7K1m7Zz/aNy3fJk14RJJuEQSLpE88l2DZ5v5xWJ2oFl7qZjN+SJsr5y4RBIukYRLJOESSbhE8sh3ouquwHfN2pPF7JdmbfUt327tnHnky6wJl0jCJZJwieSR70QvFbPqIqxzrpnvxwuxvXDiEkm4RBIukYRLJOESyV2FxovN/MqEz6j+T9gXK+yFhzlxiSRcIgmXSMIlkouzxvvN/MiEz/i6mI37a1MexYlLJOESSbhEEi6RhEskdxWGYXi9mH249V0whROXSMIlknCJJFwiuTgbhuG1YnZows9339D9e4W9MI4Tl0jCJZJwiSRcIgmXSO4qTPRDMXuzWXt3kxvZ55y4RBIukYRLJOESyVt32CneusOsCZdIwiWScIkkXCKNfuQ79moPtsGJSyThEkm4RBIukYRLJOESSbhEEi6RhEukB9LUfA//0xDfAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAD+UlEQVR4nO3dMascZRTH4VljCiGNikIKY8AuTQhBEFSI2EQtzVfQJtoErNOnTOE3sBEECwkhoGAKtbCRGEOCWqhIIBCLKIgGNh8g55CZvbub/c99nvLw3r1v8eOFmbl7Z7FcLpcDhHnicW8AViFcIgmXSMIlknCJJFwiCZdIwiWScIn05NiFi8Vik/uAYRiGYeyDXCcukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukUa/dWcnnClm7zVr/yxm/zZrPylmt5u1PzdztsqJSyThEkm4RBIukYRLpMVy5MtTd+Jdvr8Ws6Mb+l33mvn1Df2+TfmjmF1o1n6/yY2M412+zJpwiSRcIgmXSFmPfKvHu8ebtT8Vs2PN2hPF7FSz9pVi9nuz9oVmPtb9Zn6nmB2e8Lm/NfMduDgby4lLJOESSbhEEi6RhEukrEe+W/R0M69uQHQX4y/vcQ/d373fKmY3mrXPFLMPmrUfP3JHm+eRL7MmXCIJl0jCJZKLs0DvFrNPm7U/FrM3mrV3V9vOWrk4Y9aESyThEkm4RBIukdxV2GHPN/NrE9ZW/27ts9W2sxXuKjBrwiWScIkkXCJlfct3nznbzJ8rZn81a2+uaS+7xolLJOESSbhEEi6RhEskj3x3xKvF7Ktm7cFidqpZe3Wl3Tw+Hvkya8IlknCJJFwieeS7I94uZtVF2DAMw5fF7Ns17iWBE5dIwiWScIkkXCIJl0juKmzZU838dDH7r1l7vpj9v9p2YjlxiSRcIgmXSMIlkouzLfuomVdv87ncrP1mTXtJ5sQlknCJJFwiCZdIwiWSb/lu0DvF7PNm7T/F7K1m7Zz/aNy3fJk14RJJuEQSLpE88l2DZ5v5xWJ2oFl7qZjN+SJsr5y4RBIukYRLJOESSbhE8sh3ouquwHfN2pPF7JdmbfUt327tnHnky6wJl0jCJZJwieSR70QvFbPqIqxzrpnvxwuxvXDiEkm4RBIukYRLJOESyV2FxovN/MqEz6j+T9gXK+yFhzlxiSRcIgmXSMIlkouzxvvN/MiEz/i6mI37a1MexYlLJOESSbhEEi6RhEskdxWGYXi9mH249V0whROXSMIlknCJJFwiuTgbhuG1YnZows9339D9e4W9MI4Tl0jCJZJwiSRcIgmXSO4qTPRDMXuzWXt3kxvZ55y4RBIukYRLJOESyVt32CneusOsCZdIwiWScIkkXCKNfuQ79moPtsGJSyThEkm4RBIukYRLJOESSbhEEi6RhEukB9LUfA//0xDfAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "overfit_sample(28*10)\n",
    "overfit_sample(28*10, maxp=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "8e786f6d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAD+UlEQVR4nO3dMascZRTH4VkTCyFVJAELTcAujYgIQhQiNiaWyVeITbQJWKdPaeE3SBMQUgQRQUELtbBISFQiJoWKCEIsVBATWD+A55CZvbub/c99nvLw3r1v8eOFmbl7Z7FcLpcDhHnicW8AViFcIgmXSMIlknCJJFwiCZdIwiWScIl0cOzCxWKxyX3AMAzDMPZBrhOXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSKPfurMLzhWz883aX4vZP83aK8Xst2btj82c7XLiEkm4RBIukYRLJOESabEc+fLUXXiX771idnxDv+vPZv7thn7fpvxSzC43a7/Z5EZG8i5fZk24RBIukYRLpKhHvtXj3Reatd8VsxPN2heL2alm7SvF7Odm7bPNfKyHzfz3YvbMhM/9qZnvwsXZWE5cIgmXSMIlknCJJFwiRT3y3aaLzfxGMeuuxl/e4x66P3z/oZh936w9XMzeadZ+8MgdbZ5HvsyacIkkXCIJl0guzgKdLWZXm7W3i9nrzdr7q21nrVycMWvCJZJwiSRcIgmXSO4q7LCjzfzWhLXV/1v7cLXtbIW7CsyacIkkXCIJl0hR3/Ldby408yPF7I9m7Z017WXXOHGJJFwiCZdIwiWScInkke+OOFnMPmvWPlnMTjVrv1hpN4+PR77MmnCJJFwiCZdIHvnuiDPFrLoIG4Zh+LSYfbXGvSRw4hJJuEQSLpGESyThEsldhS17qpm/Wcz+bdZeKmYPVttOLCcukYRLJOESSbhEcnG2Ze818+p9wh83a79c016SOXGJJFwiCZdIwiWScInkW74b9FYxu9as/buYnW7WzvmPxn3Ll1kTLpGESyThEskj3zV4upm/X8wONGs/KmZzvgjbKycukYRLJOESSbhEEi6RPPKdqLor8HWz9qVidrdZW33Lt1s7Zx75MmvCJZJwiSRcInnkO9Hzxay6COtcbOb78UJsL5y4RBIukYRLJOESSbhEclehcayZfzLhM6r/E3Z9hb3wf05cIgmXSMIlknCJ5OKs8XYzf27CZ3xezMb9tSmP4sQlknCJJFwiCZdIwiWSuwrDMLxWzN7d+i6YwolLJOESSbhEEi6RXJwNw/BqMTs04ee7b+j+tcJeGMeJSyThEkm4RBIukYRLJHcVJrpZzN5o1t7f5Eb2OScukYRLJOESSbhE8tYddoq37jBrwiWScIkkXCIJl0ijH/mOvdqDbXDiEkm4RBIukYRLJOESSbhEEi6RhEsk4RLpP5IvfQ/JCRL1AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAD+UlEQVR4nO3dMascZRTH4VljCiGNikIKY8AuTQhBEFSI2EQtzVfQJtoErNOnTOE3sBEECwkhoGAKtbCRGEOCWqhIIBCLKIgGNh8g55CZvbub/c99nvLw3r1v8eOFmbl7Z7FcLpcDhHnicW8AViFcIgmXSMIlknCJJFwiCZdIwiWScIn05NiFi8Vik/uAYRiGYeyDXCcukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukUa/dWcXnClm7zVr/yxm/zZrPylmt5u1PzdztsuJSyThEkm4RBIukYRLpMVy5MtTd+Fdvr8Ws6Mb+l33mvn1Df2+TfmjmF1o1n6/yY2M5F2+zJpwiSRcIgmXSFGPfKvHu8ebtT8Vs2PN2hPF7FSz9pVi9nuz9oVmPtb9Zn6nmB2e8Lm/NfNduDgby4lLJOESSbhEEi6RhEukqEe+2/R0M6/uQHRX4y/vcQ/dH77fKmY3mrXPFLMPmrUfP3JHm+eRL7MmXCIJl0jCJZKLs0DvFrNPm7U/FrM3mrV3V9vOWrk4Y9aESyThEkm4RBIukdxV2GHPN/NrE9ZW/2/ts9W2sxXuKjBrwiWScIkkXCJFfct3vznbzJ8rZn81a2+uaS+7xolLJOESSbhEEi6RhEskj3x3xKvF7Ktm7cFidqpZe3Wl3Tw+Hvkya8IlknCJJFwieeS7I94uZtVF2DAMw5fF7Ns17iWBE5dIwiWScIkkXCIJl0juKmzZU838dDH7r1l7vpj9v9p2YjlxiSRcIgmXSMIlkouzLfuomVdv87ncrP1mTXtJ5sQlknCJJFwiCZdIwiWSb/lu0DvF7PNm7T/F7K1m7Zz/aNy3fJk14RJJuEQSLpE88l2DZ5v5xWJ2oFl7qZjN+SJsr5y4RBIukYRLJOESSbhE8sh3ouquwHfN2pPF7JdmbfUt327tnHnky6wJl0jCJZJwieSR70QvFbPqIqxzrpnvxwuxvXDiEkm4RBIukYRLJOESyV2FxovN/MqEz6j+T9gXK+yFhzlxiSRcIgmXSMIlkouzxvvN/MiEz/i6mI37a1MexYlLJOESSbhEEi6RhEskdxWGYXi9mH249V0whROXSMIlknCJJFwiuTgbhuG1YnZows9339D9e4W9MI4Tl0jCJZJwiSRcIgmXSO4qTPRDMXuzWXt3kxvZ55y4RBIukYRLJOESyVt32CneusOsCZdIwiWScIkkXCKNfuQ79moPtsGJSyThEkm4RBIukYRLJOESSbhEEi6RhEukB+O/fA/0HCGSAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.manual_seed(7)\n",
    "overfit_sample()\n",
    "overfit_sample()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c6c7a1e",
   "metadata": {},
   "source": [
    "# Full Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "00c9da05",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "model = SaShiMi(\n",
    "    input_dim=1,\n",
    "    hidden_dim=64,\n",
    "    output_dim=256,\n",
    "    state_dim=64,\n",
    "    sequence_length=784,\n",
    "    block_count=2,\n",
    ").to(device)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = torch.nn.CrossEntropyLoss()\n",
    "losses = []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "020d9de5",
   "metadata": {},
   "source": [
    "You can run the following block multiple times to train the same model without restarting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "38306e52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 loss: 0.7941518191695197\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 loss: 0.7467632965842866\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 loss: 0.7763504243016268\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7500 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 loss: 0.7242769702951095\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLtElEQVR4nO3deVxU9f4G8OfMwMwAwiggi4oiLiiIWCSILWKRWOaShdrNJXJvubfLvZbUTbRumem11V8uYdqqkZpbmUpqmhAJLqCICyqorC6AqAPMnN8f5ASJyCDMOTPzvF+v80eHcw6fmRcNj/N9OCOIoiiCiIiISMYUUg9AREREdDsMLERERCR7DCxEREQkewwsREREJHsMLERERCR7DCxEREQkewwsREREJHsMLERERCR7dlIP0BwMBgPOnz8PZ2dnCIIg9ThERETUCKIoory8HO3atYNC0fB7KFYRWM6fPw8fHx+pxyAiIqImyMvLQ4cOHRo8xioCi7OzM4CaB+zi4iLxNERERNQYZWVl8PHxMf4eb4hVBJYby0AuLi4MLERERBamMXUOlm6JiIhI9hhYiIiISPYYWIiIiEj2GFiIiIhI9hhYiIiISPYYWIiIiEj2GFiIiIhI9hhYiIiISPYYWIiIiEj2GFiIiIhI9hhYiIiISPYYWIiIiEj2GFgaYDCIWL7nFGatz5R6FCIiIptmFZ/W3FKO5Jfhzc1HIIrAgz08EOHvIfVIRERENonvsDSgV3stnunvCwCIW5uB8utV0g5ERERkoxhYbmNGlD86ujoiv/Q63v7hqNTjEBER2SQGlttwVNlh3hO9AQDfpObi1xMlEk9ERERkexhYGiG8ixvG9esEAHhlzSFU6KolnoiIiMi2MLA00sxHeqB9awecvXQN87ZwaYiIiMicGFgayUn959LQ58ln8FvOBYknIiIish0MLCa4r5s7xvT1AQC8vOYQrlXqJZ6IiIjINjCwmOjVIT3hrdXgzIWrWLA1W+pxiIiIbAIDi4lcNPZ4e2QQAGD5r6eQduaixBMRERFZPwaWJhjo74En7u4AUQRmfHcI16u4NERERNSSGFiaaNZjAfBwViOnuALvbT8m9ThERERWjYGlibSO9njr8ZqloWW/5OBA3mVpByIiIrJiDCx34OEATwzv0w4GEXj5u4PQVXNpiIiIqCUwsNyh+KGBcG+lwrHCK/j45xNSj0NERGSVGFjukKuTCm8M7wUA+L+dJ5F5rlTiiYiIiKxPkwLLokWL4OvrC41Gg7CwMKSmpt7y2IiICAiCcNM2ZMgQ4zGFhYV45pln0K5dOzg6OmLw4ME4fvx4U0aTxKNB3ng0yAt6g4gZ3x1CZbVB6pGIiIisismBZfXq1YiNjUV8fDzS09MRHByMqKgoFBUV1Xv82rVrkZ+fb9wyMzOhVCoRHR0NABBFESNGjEBOTg7Wr1+P/fv3o1OnToiMjERFRcWdPTozemN4L7RxtEdWfhk+2XlS6nGIiIisismBZeHChZg8eTJiYmIQEBCAxYsXw9HREcuXL6/3eFdXV3h5eRm3bdu2wdHR0RhYjh8/jpSUFHzyySfo27cv/P398cknn+DatWv45ptv7uzRmZF7KzVmDwsEAHy84ziOFpRJPBEREZH1MCmwVFZWIi0tDZGRkX9eQKFAZGQkkpOTG3WNhIQEjBkzBk5OTgAAnU4HANBoNHWuqVarsWfPnnqvodPpUFZWVmeTg2HB7fBwgCeq9CJmJB5CtZ5LQ0RERM3BpMBSUlICvV4PT0/POvs9PT1RUFBw2/NTU1ORmZmJSZMmGff16NEDHTt2RFxcHC5duoTKykrMmzcPZ8+eRX5+fr3XmTt3LrRarXHz8fEx5WG0GEEQ8NaIXnDR2CHjXCmW7s6ReiQiIiKrYNa/EkpISEBQUBBCQ0ON++zt7bF27VocO3YMrq6ucHR0xI4dO/DII49Aoah/vLi4OJSWlhq3vLw8cz2E2/Jw0WDW0Jqlofe3HceJonKJJyIiIrJ8JgUWd3d3KJVKFBYW1tlfWFgILy+vBs+tqKjAqlWrMHHixJu+FhISggMHDuDy5cvIz8/Hli1bcOHCBfj5+dV7LbVaDRcXlzqbnDxxd3tE+LdFpd6AGd8dgt4gSj0SERGRRTMpsKhUKoSEhCApKcm4z2AwICkpCeHh4Q2em5iYCJ1Oh7Fjx97yGK1Wi7Zt2+L48ePYt28fhg8fbsp4siEIAuaODIKz2g77cy9j+Z5TUo9ERERk0UxeEoqNjcWyZcuwcuVKZGVlYfr06aioqEBMTAwAYPz48YiLi7vpvISEBIwYMQJubm43fS0xMRE7d+40/mnzww8/jBEjRmDQoEFNeEjy4K11wGtDegIAFmzNRk7xFYknIiIislx2pp4wevRoFBcXY9asWSgoKECfPn2wZcsWYxE3Nzf3pu5JdnY29uzZg61bt9Z7zfz8fMTGxqKwsBDe3t4YP348Xn/99SY8HHkZ3dcHmzPysft4CV5Zcwirp4RDoRCkHouIiMjiCKIoWnzBoqysDFqtFqWlpbLrs5y9dBVR7/2Ciko94ocGIObezlKPREREJAum/P7mZwm1sA5tHDHz0ZqloXe3ZOPMBcu5ey8REZFcMLCYwdOhHdHPzxXXqvSYuSYDBv7VEBERkUkYWMxAoRAw74necLBXIjnnAr5OzZV6JCIiIovCwGImndycMCPKHwAw94csnL10VeKJiIiILAcDixk9098X93Rqg4pKPeLWZsAK+s5ERERmwcBiRgqFgHef7A21nQK7j5fg233y+UgBIiIiOWNgMTO/tq3wr0HdAQD/3ZSF/NJrEk9EREQkfwwsEph4nx+CfVqjXFeNV7k0REREdFsMLBJQKgQseLI3VEoFdmQXY93+c1KPREREJGsMLBLp5umMf0R2AwDM2XgERWXXJZ6IiIhIvhhYJDTlAT/0au+C0mtV+M/3mVwaIiIiugUGFgnZKxWY/2Qw7JUCth4pxMZD+VKPREREJEsMLBLr6e2C5wd2BQDEr89EyRWdxBMRERHJDwOLDDwX0RU9vJxx6WoV4tcflnocIiIi2WFgkQGVnQILooOhVAjYnJGPHzO4NERERFQbA4tM9GqvxbQBfgCA19dn4lJFpcQTERERyQcDi4z8/aFu6ObRCiVXKjFnI5eGiIiIbmBgkRG1nRLzo4OhEIDvD5zH9iOFUo9EREQkCwwsMtPHpzUm31+zNPTqugyUXq2SeCIiIiLpMbDI0D8f7g4/dycUlevw5uYjUo9DREQkOQYWGdLYKzE/ujcEAfgu7Sx2ZBdJPRIREZGkGFhkKqSTK2L6dwYAvLo2A2XXuTRERES2i4FFxv4d1R0dXR2RX3odc384KvU4REREkmFgkTFHlR3mPdEbAPBNai5+PVEi8URERETSYGCRufAubhjXrxMA4JU1h1Chq5Z4IiIiIvNjYLEAMx/pgfatHXD20jXM28KlISIisj0MLBbASf3n0tDnyWeQknNB4omIiIjMi4HFQtzXzR1PhfoAqFkaulapl3giIiIi82FgsSBxj/aEt1aDMxeuYv5P2VKPQ0REZDYMLBbERWOPt0cGAQA+23sKaWcuSjwRERGReTCwWJiB/h544u4OEEVgxneHcL2KS0NERGT9GFgs0KzHAuDhrEZOcQXe235M6nGIiIhaHAOLBdI62uOtx2uWhpb9koMDeZelHYiIiKiFMbBYqIcDPDG8TzsYRGBG4kHoqrk0RERE1ouBxYLNHhoI91YqHC+6go+STkg9DhERUYthYLFgbZxUeHN4LwDAJ7tOIvNcqcQTERERtQwGFgv3SJA3Hg3ygt4g4t+JB1FZbZB6JCIiombXpMCyaNEi+Pr6QqPRICwsDKmpqbc8NiIiAoIg3LQNGTLEeMyVK1fwwgsvoEOHDnBwcEBAQAAWL17clNFs0hvDe6GNoz2OFpTjk50npR6HiIio2ZkcWFavXo3Y2FjEx8cjPT0dwcHBiIqKQlFRUb3Hr127Fvn5+cYtMzMTSqUS0dHRxmNiY2OxZcsWfPnll8jKysJLL72EF154ARs2bGj6I7Mh7q3UmD0sEADw8Y7jOFpQJvFEREREzcvkwLJw4UJMnjwZMTExxndCHB0dsXz58nqPd3V1hZeXl3Hbtm0bHB0d6wSWvXv3YsKECYiIiICvry+mTJmC4ODgBt+5obqGBbfDwwGeqNKLmJF4CNV6Lg0REZH1MCmwVFZWIi0tDZGRkX9eQKFAZGQkkpOTG3WNhIQEjBkzBk5OTsZ9/fv3x4YNG3Du3DmIoogdO3bg2LFjGDRokCnj2TRBEPDWiF7QOtgj41wplvySI/VIREREzcakwFJSUgK9Xg9PT886+z09PVFQUHDb81NTU5GZmYlJkybV2f/RRx8hICAAHTp0gEqlwuDBg7Fo0SI88MAD9V5Hp9OhrKyszkaAh4sGsx4LAAB8sP04jheWSzwRERFR8zDrXwklJCQgKCgIoaGhdfZ/9NFHSElJwYYNG5CWlob//e9/eP7557F9+/Z6rzN37lxotVrj5uPjY47xLcLIu9tjoH9bVOoNmPHdIegNotQjERER3TGTAou7uzuUSiUKCwvr7C8sLISXl1eD51ZUVGDVqlWYOHFinf3Xrl3Dq6++ioULF2Lo0KHo3bs3XnjhBYwePRoLFiyo91pxcXEoLS01bnl5eaY8DKsmCALeHhkEZ7UdDuRdxvI9p6QeiYiI6I6ZFFhUKhVCQkKQlJRk3GcwGJCUlITw8PAGz01MTIROp8PYsWPr7K+qqkJVVRUUirqjKJVKGAz1F0fVajVcXFzqbPQnb60DXhvSEwCwYGs2coqvSDwRERHRnTF5SSg2NhbLli3DypUrkZWVhenTp6OiogIxMTEAgPHjxyMuLu6m8xISEjBixAi4ubnV2e/i4oIBAwZgxowZ2LlzJ06dOoUVK1bg888/x+OPP97Eh0Wj+/rg/m7u0FUb8MqaQzBwaYiIiCyYnaknjB49GsXFxZg1axYKCgrQp08fbNmyxVjEzc3NvendkuzsbOzZswdbt26t95qrVq1CXFwcnn76aVy8eBGdOnXCW2+9hWnTpjXhIRFQszQ0d2QQot77Bb+fvoSVyacRc29nqcciIiJqEkEURYv/p3dZWRm0Wi1KS0u5PPQXX6ScwevfZ8LBXoktL92PTm5Otz+JiIjIDEz5/c3PErJyT4d2RD8/V1yr0nNpiIiILBYDi5VTKATMe6I3HOyVSMm5iK9Sc6UeiYiIyGQMLDagk5sTZkT5AwDe+SELZy9dlXgiIiIi0zCw2Ihn+vvink5tUFGpR9zaDFhBdYmIiGwIA4uNUCgEvPtkb6jtFNh9vATf7uPN9oiIyHIwsNgQv7at8K9B3QEA/92UhfzSaxJPRERE1DgMLDZm4n1+6OPTGuW6arzKpSEiIrIQDCw2RqkQMP/J3lApFdiRXYy16eekHomIiOi2GFhsUDdPZ/wjshsAYM7Gwygquy7xRERERA1jYLFRUx7wQ6/2Lii7Xo3Xvs/k0hAREckaA4uNslcqMP/JYNgrBWw7UoiNh/KlHomI7tBPhwvw4P92YhVvEElWiIHFhvX0dsHzA7sCAOLXZ6Lkik7iiYioqdbtP4vnvkpHTnEF4jccRt5F3iCSrAsDi417LqIreng549LVKsSvPyz1OETUBF+knME/Vx+E3iDCWW0HXbUBb2w6IvVYRM2KgcXGqewUWBAdDKVCwOaMfPyYwaUhIkuyeNdJvP59JgBgQngnrHmuP+wUNUu9Px8tlHg6oubDwELo1V6L6QO6AABeX5+JixWVEk9ERLcjiiIW/JSNd348CgB4LqILZg8LRHdPZ0y8rzMAYPaGI7hepZdyTKJmw8BCAIAXH+qKbh6tUHKlEnM2cmmISM4MBhFzNh7BxztOAABeHuyPlwf3gCAIAIAXH+oGLxcNci9exdJfcqQclajZMLAQAEBtp8T86GAoBGD9gfPYdoRvJRPJkd4g4pU1h7Bi72kAwBvDA/FcRNc6x7RS2+G1IT0BAIt2nGABl6wCAwsZ9fFpjcn3+wEAXluXgdKrVRJPRES1VVYb8Pdv9iMx7SwUArAgOhjjw33rPfax3t7o38UNumoD5mxkAZcsHwML1fHPh7vDz90JReU6vLmZL3JEcnG9So+pX+zD5ox82CsFLPrb3XgypMMtjxcEAW8MD4SdQsD2LBZwyfIxsFAdGnsl5kf3hiAA36WdxY7sIqlHIrJ5V3TVeOazVOzILobGXoFl4+/BI0Hetz2vqwcLuGQ9GFjoJiGdXBHTv+ZF7tW1GSi7zqUhIqlcvlqJpz/9DSk5F9FKbYeVMaGI8Pdo9Pm1C7hLdrGAS5aLgYXqNSPKH53cHJFfeh1zf8iSehwim1RcrsOYpSk4mHcZrR3t8fXkMIT5uZl0jVZqO/znsZoC7v/tZAGXLBcDC9XLQaXEvCd6AwC+Sc3DnuMlEk9EZFvOXb6GUUuScbSgHG2d1Vg9JRy9O7Ru0rWGBLGAS5aPgYVuqZ+fG8b16wQAeGXNIVzRVUs8EZFtOFVSgVGLk3GqpALtWzsgcWo4/L2cm3w9FnDJGjCwUINmPtID7Vs74Nzla3h3y1GpxyGyekcLyhC9OBnnLl+Dn7sTEqeFw9fd6Y6v29XDGRPvZwGXLBcDCzXISW1nXBr6PPkMUnIuSDwRkfU6kHcZo5ekoOSKDj28nLF6ajjatXZotuv//UEWcMlyMbDQbd3XzR1PhfoAqFkaulbJf5kRNbeUnAt4elkKSq9VoY9Pa6yeEo62zupm/R5OLOCSBWNgoUaJe7QnvLUanLlwFfN/ypZ6HCKrsiO7CBOWp6KiUo9wPzd8OSkMWkf7FvleQ4K8cW9XFnDJ8jCwUKO4aOwxd2QQAOCzvaew7/RFiScisg4/ZORjyuf7oKs24MEeHvgspi9aqe1a7PsJgoA5w/4s4CZlsYBLloGBhRotwt8DT4Z0gCgCL393iKU9ojuUuC8PL3ydjiq9iMd6e2PJuBBo7JUt/n1rF3DnbGQBlywDAwuZ5PUhAfBwViOnpALvbTsm9ThEFmvl3tOY8d0hGERg9D0++GDMXbBXmu8lmQVcsjQMLGQSraM93nq8Zmlo2e4cHMi7LO1ARBZo0Y4TiN9wGADw7L2d8c4TQVAqBLPO8NcCbu4FFnBJ3hhYyGQPB3hieJ92MIjAjMSD0FXz7WSixhBFEfO2HDUW1//+UDe8/lhPCIJ5w8oNtQu4b2w6LMkMRI3FwEJNMntoINxbqXC86Ao+Sjoh9ThEsmcwiJi1/jA+2XkSAPDqoz0Q+3B3ycIK8NcCbhELuCRrDCzUJG2cVHhzeC8AwCe7TiLzXKnEExHJV7XegH9/dxBfpJyBIABvPd4LUx7oIvVYAP5yB9yNh1nAJdliYKEmeyTIG0OCvKE3iPh34kFUVhukHolIdnTVerzw9X6sTT8HpULAe6P64OmwTlKPVceNAm7exWtYvOuk1OMQ1YuBhe7InOGBaONoj6MF5fi/nVwaIqrtWqUekz9Pw5bDBVApFfi/p+/GiLvaSz3WTWoXcD/ZeZIFXJKlJgWWRYsWwdfXFxqNBmFhYUhNTb3lsRERERAE4aZtyJAhxmPq+7ogCJg/f35TxiMzcm+lxuxhgQCAj38+gaz8MoknIpKH8utVmLA8Fb8cK4aDvRIJz9yDqEAvqce6JRZwSe5MDiyrV69GbGws4uPjkZ6ejuDgYERFRaGoqKje49euXYv8/HzjlpmZCaVSiejoaOMxtb+en5+P5cuXQxAEPPHEE01/ZGQ2w4Lb4eEAT1QbRMz47iCq9VwaItt2qaIST3/6G1JPX4Sz2g5fTAzF/d3aSj1Wg2oKuL1gr2QBl+TJ5MCycOFCTJ48GTExMQgICMDixYvh6OiI5cuX13u8q6srvLy8jNu2bdvg6OhYJ7DU/rqXlxfWr1+PgQMHws/Pr+mPjMxGEAS8NaIXtA72yDxXhiW/8CZUZLuKyq5j9NJkHDpbClcnFb6Z0g/3+LpKPVajdPVohWfvYwGX5MmkwFJZWYm0tDRERkb+eQGFApGRkUhOTm7UNRISEjBmzBg4OTnV+/XCwkJs3rwZEydONGU0kpiHiwazHgsAAHyw/TiOF5ZLPBGR+eVdvIroJck4VngFni5qrJ7SD73aa6UeyyQs4JJcmRRYSkpKoNfr4enpWWe/p6cnCgoKbnt+amoqMjMzMWnSpFses3LlSjg7O2PkyJG3PEan06GsrKzORtIbeXd7DPRvi0q9ATO+OwS9QZR6JCKzOVl8BaOWJOPMhavwcXVA4tT+6ObpLPVYJnNS2+H1P/7x8X8s4JKMmPWvhBISEhAUFITQ0NBbHrN8+XI8/fTT0Gg0tzxm7ty50Gq1xs3Hx6clxiUTCYKAt0cGwVlthwN5l5Gwh0tDZBuOnC/DqMXJyC+9ji5tnZA4tT86ujlKPVaTPRrkhXu7uqGSBVySEZMCi7u7O5RKJQoL65axCgsL4eXVcPu9oqICq1atanCpZ/fu3cjOzm7wHRgAiIuLQ2lpqXHLy8tr/IOgFuWtdTD+eeT/th5DTvEViScialnpuZcwZmkyLlRUIrCdC76dGg4v7a3/wWUJ/lrA3X6EBVySnkmBRaVSISQkBElJScZ9BoMBSUlJCA8Pb/DcxMRE6HQ6jB079pbHJCQkICQkBMHBwQ1eS61Ww8XFpc5G8jHqHh/c380dumoDXubSEFmxvSdKMPbT31B2vRohndrg68n94NZKLfVYzaKrRytMvK/mDx/mbGIBl6Rn8pJQbGwsli1bhpUrVyIrKwvTp09HRUUFYmJiAADjx49HXFzcTeclJCRgxIgRcHNzq/e6ZWVlSExMvO27KyR/giBg7sggOKmU2HfmElbuPS31SETNLimrEM+s+B1XK/W4r6s7vpgYCq2DvdRjNasXH+wKby0LuCQPJgeW0aNHY8GCBZg1axb69OmDAwcOYMuWLcYibm5uLvLz8+uck52djT179jS4HLRq1SqIooinnnrK1JFIhjq0ccTMR2uWht796SjOXKiQeCKi5rPx4HlM/SINldUGPBzgiU8n3ANHlZ3UYzU7J7Ud/jOEBVySB0EURYt/v76srAxarRalpaVcHpIRg0HE3z5NQUrORfTzc8XXk/pBoZDuk2mJmsPq33Mxc20GRBEY3qcdFkQHw15pvZ9yIooixiWkYs+JEjzUwwMJz/SVeiSyIqb8/rbe/8tIcgqFgHefCIaDvRIpORfxVWqu1CMR3ZGEPafwypqasPJUaEcsHNXHqsMKULPEO3tYIOyVApKOsoBL0rHu/9NIch3dHPHyYH8AwDs/ZOHsJb6lTJZHFEV8mHQcb246AgCYfH9nvP14Lyht5B1DFnBJDhhYqMVNCPfFPZ3aoKJSj7i1GbCCVUiyIaIoYu6PR7Fw2zEAQOzD3fHqoz0hCLYRVm6oXcD9ZCcLuGR+DCzU4hQKAe8+2RtqOwV2Hy/B6t953xyyDHqDiNe+z8TSPz4f6/XHAvD3h7rZXFgB6hZwP9nFAi6ZHwMLmYVf21b416DuAIC3Nmchv/SaxBMRNaxKb0Dstwfw9W+5EATgnZFBmPjHBwPaqkeDvHBfV3dUVhswZyPvgEvmxcBCZjPxPj/08WmNcl01XuXSEMnY9So9nvsqHesPnIedQsCHY+7CmNCOUo8lORZwSUoMLGQ2SoWA+U/2hkqpwI7sYqxNPyf1SEQ3uVpZjUkr92HbkUKo7BRYMi4EQ4PbST2WbLCAS1JhYCGz6ubpjH9EdgMAzNl4GEVl1yWeiOhPpdeqjPcccVQpseKZvniop+ftT7QxLOCSFBhYyOymPuCHoPZalF2vxmvfZ3JpiGThwhUd/rYsBWlnLsFFY4cvJ4Whf1d3qceSJSe1HV5/7M8CLu9kTebAwEJmZ6dUYH50b9grBWw7UogNB89LPRLZuILS6xi9NAWHz5fBzUmFVVPCcXfHNlKPJWuP9PqzgPvGxiNSj0M2gIGFJNHDywXPD+wKAJi94TCKy3UST0S2KvfCVUQv2YsTRVfgrdXg22nhCGjHj/i4HRZwydwYWEgyz0V0RQ8vZ1y6WoX4DZlSj0M26ERROaKX7EXexWvo5OaIb6eGo0vbVlKPZTG6erTCpPtrCrizN7KASy2LgYUko7JTYEF0MJQKAT9kFOCHjPzbn0TUTDLPlWLUkhQUlunQ3bMVEqeGw8fVUeqxLM6NAu7ZSyzgUstiYCFJ9WqvxfQBXQAAs9Zn4mJFpcQTkS3Yd/oinlqagosVlQhqr8WqKeHwcNFIPZZFclSxgEvmwcBCknvxoa7o7tkKJVcqefdManG7jxdjXEIqynXVCPV1xdeTw+DqpJJ6LIv2SC8v3N/txh1wWcCllsHAQpJT2ykx/8lgKARg/YHz2MbyHrWQrYcLMHHFPlyr0uOB7m2x8tlQOGvspR7L4tUu4P7MAi61EAYWkoVgn9aY/EBNee+1dRkovVol8URkbdYfOIfpX6WjUm/A4EAvLBsfAgeVUuqxrEaXtizgUstiYCHZ+Gdkd/i5O6GoXIc3NvFtZWo+X/+Wi5dWH4DeIGLk3e3x8d/ugtqOYaW5vfhgV7T7o4D7fyzgUjNjYCHZ0NgrMT+6NwQBWJN+Fjuyi6QeiazA0l9O4tV1GRBFYFy/TljwZDDslHzpawmOKjv8548C7mIWcKmZ8f9akpWQTq6I6d8ZAPDq2gyUXefSEDWNKIpYuO0Y3v7hKABgekQXvDE8EAqFIPFk1q12AXf2hsP86A1qNgwsJDszovzRyc0R+aXXMfeHLKnHIQskiiLe3JSFD5OOA6j5mXplcA8IAsNKS6tdwN2RXYztWXynlJoHAwvJjoNKiXlP9AYAfJOahz3HSySeiCyJ3iAibm0Glv96CgAwZ1ig8WMgyDxqF3DnsIBLzYSBhWSpn58bxod3AgC8suYQruiqJZ6ILEGV3oB/rNqPVb/nQSEA85/sjQn9faUeyyaxgEvNjYGFZOuVwT3QvrUDzl2+hnk/HpV6HJK561V6TPsiDZsO5cNeKeDjv92N6Ht8pB7LZtW+Ay4LuNQcGFhItpzUdsaloS9SziD55AWJJyK5qtBVI+az35F0tAhqOwWWjr8HjwZ5Sz2WzRvMAi41IwYWkrX7urnjqdCafyW/suYQrlZyaYjqKr1ahbEJvyE55wKcVEqsfDYUA/09pB6LwAIuNS8GFpK9uEd7wlurQe7Fq1jw0zGpxyEZKbmiw5hlKdifexlaB3t8Nbkf+vm5ST0W1dKlbStMZgGXmgEDC8mei8Yec0cGAQA+23sK+05flHgikoPzl69h1OJkZOWXwb2VGqun9kMfn9ZSj0X1eIEFXGoGDCxkESL8PfBkSAeIIvDyd4f4rzQbd7qkAtGLk5FTUoH2rR2QOC0cPbxcpB6LbuGvBdzTJSzgkukYWMhivD4kAB7OauSUVOC9bVwaslXZBeWIXpKMc5evobO7E76dFo7O7k5Sj0W3UbuAO2cjC7hkOgYWshhaR3u89XjN0tCy3TnYn3tJ4onI3A6dvYzRS5NRXK5DDy9nfDs1HO1bO0g9FjWCIAiYwwIu3QEGFrIoDwd4YnifdjD8sTSkq+bSkK1IPXURf1v2Gy5frUKwT2usmtIPbZ3VUo9FJvCrVcCdveEwrlXy/19qPAYWsjizhwbCvZUKx4uuGD8rhqzbzuwijF/+G67oqtHPzxVfTQpDa0eV1GNRE9wo4J67fA2f7Dwh9ThkQRhYyOK0cVLhzeG9AACLd+Ug81ypxBNRS/oxIx+TP9+H61UGDPRvixUxoWiltpN6LGqiOgXcX3JYwKVGY2Ahi/RIkDeGBHlDbxDx78SDqKw2SD0StYA1aWfx/NfpqNKLGBLkjSXj7oHGXin1WHSH6twBlwVcaiQGFrJYc4YHwtVJhaMF5fg/vrVsdb5IPo1/JR6EQQSiQzrgw6fugsqOL1nWoHYBd2d2MbYdKZR6JLIA/L+fLJZ7KzVmDwsEAHz88wlk5ZdJPBE1l092nsTr6w8DAJ7p74t5T/SGUiFIPBU1J786d8A9wgIu3RYDC1m0ob29MSjAE9UGETO+O4gqPZeGLJkoipj/01HM21Lz6dwvPtgV8UMDoGBYsUos4JIpmhRYFi1aBF9fX2g0GoSFhSE1NfWWx0ZEREAQhJu2IUOG1DkuKysLw4YNg1arhZOTE/r27Yvc3NymjEc2RBAE/HdEL2gd7JF5rgxLf8mReiRqIoNBxJyNR7BoR82t22c+0gP/GuQPQWBYsVaOKjvMGnrjDrgs4FLDTA4sq1evRmxsLOLj45Geno7g4GBERUWhqKj+mwCtXbsW+fn5xi0zMxNKpRLR0dHGY06ePIn77rsPPXr0wM6dO3Ho0CG8/vrr0Gg0TX9kZDM8XDSY9cdfHXyw/TiOFZZLPBGZqlpvwMtrDmHF3tMQBODNEb0wbUAXqcciM4gK/KOAq2cBlxomiCb+dISFhaFv3774+OOPAQAGgwE+Pj548cUXMXPmzNue//7772PWrFnIz8+Hk1PN7bTHjBkDe3t7fPHFF014CEBZWRm0Wi1KS0vh4sLPE7FFoiji2RW/Y0d2MYJ9WmPt9P7sPFiIymoDXlq9Hz9kFECpELAgujcev6uD1GORGeUUX0HU+7+gSi9i6bgQDAr0knokMhNTfn+b9A5LZWUl0tLSEBkZ+ecFFApERkYiOTm5UddISEjAmDFjjGHFYDBg8+bN6N69O6KiouDh4YGwsDB8//33t7yGTqdDWVlZnY1smyAIeHtkEJzVdjiYdxkJe7g0ZAmuVeox5Yt9+CGjACqlAov+djfDig3ya9sKUx5gAZcaZlJgKSkpgV6vh6enZ539np6eKCgouO35qampyMzMxKRJk4z7ioqKcOXKFbzzzjsYPHgwtm7discffxwjR47Erl276r3O3LlzodVqjZuPj48pD4OslLfWAf95rCcA4H9bjyGn+IrEE1FDyq9XYcJnqdiZXQyNvQKfTrgHg3vxX9a26vmBLOBSw8z6V0IJCQkICgpCaGiocZ/BUPNXHcOHD8c///lP9OnTBzNnzsRjjz2GxYsX13uduLg4lJaWGre8vDyzzE/yN+oeH9zfzR26agNe/u4Q9Aauh8vR5auVGPvpb0g9dRHOajt8MTEMD3RvK/VYJCEWcOl2TAos7u7uUCqVKCyse5OfwsJCeHk1/C+jiooKrFq1ChMnTrzpmnZ2dggICKizv2fPnrf8KyG1Wg0XF5c6GxFQszQ0d2QQnFRK7DtzCSv3npZ6JPqLovLrGL0kBQfPlqKNoz2+ntwPfX1dpR6LZCAq0AsPdG/LAi7Vy6TAolKpEBISgqSkJOM+g8GApKQkhIeHN3huYmIidDodxo4de9M1+/bti+zs7Dr7jx07hk6dOpkyHhEAoEMbR8x8tGZp6N2fjuLMBf5LTS7OXrqKUYuTkV1YDg9nNb6dGo6gDlqpxyKZEAQBs4cG8A64VC+Tl4RiY2OxbNkyrFy5EllZWZg+fToqKioQExMDABg/fjzi4uJuOi8hIQEjRoyAm5vbTV+bMWMGVq9ejWXLluHEiRP4+OOPsXHjRjz33HNNeEhEwNOhHdHPzxXXq2qWhgxcGpJcTvEVjFqcjNMXrqJDGwd8N60/unk6Sz0WyQwLuHQrJgeW0aNHY8GCBZg1axb69OmDAwcOYMuWLcYibm5uLvLz8+uck52djT179ty0HHTD448/jsWLF+Pdd99FUFAQPv30U6xZswb33XdfEx4SEaBQCHj3iWA42Cvx26mL+Oq3M1KPZNOy8sswakkKzpdeh19bJyROC0dHN0epxyKZen5gV7Rv7YBzl6/xc8LIyOT7sMgR78NCt/LZr6cwZ+MROKqU+OmlB+Djyl+S5rY/9xKe+ex3lF6rQoC3Cz6fGAr3VmqpxyKZ25KZj2lfpkOlVGDrPx+Ar7uT1CNRC2ix+7AQWZoJ4b7o69sGVyv1eHVdBkt8ZpZ88gLGfvobSq9V4e6OrfHNlH4MK9QoLODSXzGwkFVTKAS8+2Qw1HYK7D5egtW/80/gzWXH0SI881kqKir1uLerG76YGAatg73UY5GFEAQBc4YFQqVUYGd2MbaygGvzGFjI6nV2d8K/B/kDAN7anIX80msST2T9Nh/Kx+TP90FXbUBkTw8kTOgLJ7Wd1GORhens7oTJD3QGALzBAq7NY2Ahm/DsfZ3Rx6c1ynXViFvLpaGW9O3veXjxm3RUG0QMC26HT8aGQGOvlHosslAs4NINDCxkE5QKAfOf7G18e3lN+jmpR7JKn/16Ci+vOQSDCDwV6oP3RveBvZIvM9R0jio7vP7Hp7Ev2ZWDU7wDrs3iKwnZjG6ezvhHZDcAwBsbD6Ow7LrEE1kPURTx8c/HMWfjEQDApPs64+3Hg/iJ2dQsogI9jQXcOSzg2iwGFrIpUx/wQ1B7LcquV+O1dZl84WsGoijinS1HsWDrMQDAS5Hd8NqQnhAEhhVqHizgEsDAQjbGTqnA/OjesFcK2J5ViA0Hz0s9kkUzGES8vj4TS3blAAD+M6QnXorszrBCza6zu5PxDrgs4NomBhayOT28XPDCwJqlodkbDqO4XCfxRJapWm/AvxMP4suUXAgCMHdkECbd7yf1WGTFWMC1bQwsZJOeG9gFPb1dcOlqFeI3ZEo9jsXRVevx/NfpWLv/HJQKAe+P7oOnQjtKPRZZOQeVkgVcG8bAQjbJXqnA/Cd7Q6kQ8ENGAX7IyL/9SQQAuFpZjUkr9+Gnw4VQ2SmweGwIhvdpL/VYZCOiAj0x4MYdcDewgGtLGFjIZvVqr8X0AV0AAK9/n4mLFZUSTyR/ZderMGF5KnYfL4GDvRKfPdMXDwd4Sj0W2RBBEDD7jwLurmMs4NoSBhayaS8+1BXdPVvhQkUlZm84LPU4snaxohJ/W5aC309fgrPGDl9OCsW9Xd2lHotsEAu4tomBhWya2k6J+U8GQyEAGw6ex9bDBVKPJEuFZdcxekkyMs+Vwc1JhVVT+iGkk6vUY5ENq13AXbSDBVxbwMBCNi/YpzUm//Gvtf98n4nSq1USTyQveRevInpxMo4XXYGXiwarp4YjsJ1W6rHIxtUu4C79hQVcW8DAQgTgn5Hd4dfWCUXlOryx6YjU48jGiaIriF6cjNyLV9HR1RGJ08LR1aOV1GMRAahbwI1nAdfqMbAQAdDYKzH/yd4QBGBN+lnsyC6SeiTJHT5fitFLklFQdh3dPFohcVo4fFwdpR6LyKh2AfeXY8X46TALuNaMgYXoDyGdXBHTv+aj7OPWZKDsuu0uDaWduYQxS1NwoaISvdq7YPXUcHi6aKQei+gmtQu4b25iAdeaMbAQ1TIjyh+d3BxRUHYdb2/OknocSfx6ogTjEn5D+fVq9PVtg68n94Ork0rqsYhuiQVc28DAQlSLg0qJeU/0BgCs+j0Pu48XSzyReW0/UoiYFb/jaqUe93dzx8pnQ+GisZd6LKIGOaiUmDWUBVxrx8BC9Bf9/NwwPrwTAGDmmgxc0VVLPJF5rD9wDlO/TENltQFRgZ74dMI9cFTZST0WUaMMCmAB19oxsBDV45XBPdChTc1bzPN+PCr1OC3um9RcvLT6APQGESPvao9Ff7sbajul1GMRNRoLuNaPgYWoHk5qO+PS0BcpZ5B88oLEE7WcT3fnIG5tBkQReDqsIxZEB8NOyZcGsjyd3Z0wdQALuNaKr0pEt3BvV3fjJxC/suYQrlZa19KQKIp4f/sx/PePcvHUAX7474heUCgEiScjarrnIljAtVYMLEQNiHu0B7y1GuRevIr5P2VLPU6zEUURb/+Qhfe3HwcA/HtQd8wc3AOCwLBClu2vBdyc4isST0TNhYGFqAEuGnvMHRkEAFix9zT2nb4o8UR3Tm8Q8eq6DCzbfQoAED80AC882I1hhazGoABPRPjXFHBnbzzCAq6VYGAhuo0Ifw88GdIBogi8/N0hXK+y3HXxKr0B/1x9AN+k5kEhAO8+0Rsx93aWeiyiZiUIAmYPZQHX2jCwEDXC60MC4OGsRk5JBd7bdkzqcZrkepUe079Mx4aD52GnEPDhU3dhVF8fqcciahG+fyngWlsHzRYxsBA1gtbRHm8/XrM0tGx3DvbnXpJ4ItNU6KoxceXv2J5VCLWdAsvG34PHereTeiyiFsUCrnVhYCFqpMgAT4zo0w6GP5aGdNWWsTRUeq0K4xJ+w68nLsBJpcSKmFAM7OEh9VhELa52AXfZL6dYwLVwDCxEJogfGgj3ViocL7qCD5OOSz3ObZVc0eGppSlIz70MrYM9vpwUhvAublKPRWQ2tQu4vAOuZWNgITJBGycV3hzeCwCweFcOMs6WSjzRreWXXsPoJck4kl8G91ZqrJrSD3d1bCP1WERmVbuAu/t4CX46XCD1SNREDCxEJnokyBtDgryhN4iY8d1BVFYbpB7pJrkXriJ6cTJOFlegnVaDb6f2Q09vF6nHIpJE3QJuFgu4FoqBhagJ5gwPhKuTCkcLymVX5jteWI4nF+/F2UvX4OvmiG+nhcOvbSupxyKSFAu4lo+BhagJ3FupMXtYIABg0Y4TyMovk3iiGpnnSjFqSTKKynXw93TGt9PC0aGNo9RjEUnOQaVEPO+Aa9EYWIiaaGhvbwwK8ET1H0tDVXppl4Z+P30RTy1NwaWrVQjuoMXqqf3g4ayRdCYiOXn4jwJulV5kAdcCMbAQNZEgCPjv472gdbBH5rkyLP0lR7JZdh8vxriE31Cuq0ZoZ1d8OSkMrR1Vks1DJEcs4Fq2JgWWRYsWwdfXFxqNBmFhYUhNTb3lsRERERAE4aZtyJAhxmOeeeaZm74+ePDgpoxGZFYezhrj28wfbD+OY4XlZp9hS2YBJq7Yh+tVBkT4t8XKmFA4a+zNPgeRJfB1d8K0Pwq4b2zkHXAticmBZfXq1YiNjUV8fDzS09MRHByMqKgoFBUV1Xv82rVrkZ+fb9wyMzOhVCoRHR1d57jBgwfXOe6bb75p2iMiMrPH72qPgX/c52HGd4dQbcaloXX7z+L5r9NRqTfg0SAvLB13DxxUSrN9fyJLNP2PAu750uss4FoQkwPLwoULMXnyZMTExCAgIACLFy+Go6Mjli9fXu/xrq6u8PLyMm7btm2Do6PjTYFFrVbXOa5NG94vgiyDIAh4e2QQnNV2OJh3GQl7Tpnl+36Zcgax3x6E3iDiyZAO+HDMXVDZcZWX6HZYwLVMJr26VVZWIi0tDZGRkX9eQKFAZGQkkpOTG3WNhIQEjBkzBk5OTnX279y5Ex4eHvD398f06dNx4cKFW15Dp9OhrKyszkYkJW+tA/7zWE8AwP+2HcPJFn4BXLLrJP7zfSZEEXimvy/efaI37JQMK0SN9XCAJwaygGtRTHqFKykpgV6vh6enZ539np6eKCi4fXkpNTUVmZmZmDRpUp39gwcPxueff46kpCTMmzcPu3btwiOPPAK9vv7Papk7dy60Wq1x8/HhJ86S9Ebd44P7u7mjstqAl787BL2h+V8ARVHE/7ZmY+6PRwEAzw/sgvihAVAohGb/XkTWTBAExLOAa1HM+k+yhIQEBAUFITQ0tM7+MWPGYNiwYQgKCsKIESOwadMm/P7779i5c2e914mLi0Npaalxy8vLM8P0RA0TBAHvPNEbTiol0s5cwsq9p5v1+gaDiDc2HcFHP9esub8yuAdmRPWAIDCsEDUFC7iWxaTA4u7uDqVSicLCwjr7CwsL4eXl1eC5FRUVWLVqFSZOnHjb7+Pn5wd3d3ecOFF/GUqtVsPFxaXORiQH7Vs7IO7RmqWhd386ijMXKprlunqDiJlrD+GzX08DAN4cHojpEV2a5dpEtmx6RFd0aFNTwP34ZxZw5cykwKJSqRASEoKkpCTjPoPBgKSkJISHhzd4bmJiInQ6HcaOHXvb73P27FlcuHAB3t7epoxHJAt/C+2IcD83XK+qWRoy3OHSUGW1AX9ftR/f7jsLhQD8LzoY48J9m2dYIhvnoFJi1mM1Bdxlu1nAlTOTl4RiY2OxbNkyrFy5EllZWZg+fToqKioQExMDABg/fjzi4uJuOi8hIQEjRoyAm1vdj7a/cuUKZsyYgZSUFJw+fRpJSUkYPnw4unbtiqioqCY+LCLpKBQC5j3RGw72Svx26iK++u1Mk691vUqPaV+mYfOhfNgrBSz62914IqRDM05LRCzgWgaTA8vo0aOxYMECzJo1C3369MGBAwewZcsWYxE3NzcX+fn5dc7Jzs7Gnj176l0OUiqVOHToEIYNG4bu3btj4sSJCAkJwe7du6FWq5v4sIik1dHNES8P9gcAzP3xKPIuXjX5Gld01Xjms1T8fLQIGnsFPp3QF48E8V1HouYmCAJmDwuEyq6mgLslkwVcORJEK4iSZWVl0Gq1KC0tZZ+FZMNgEDF6aTJ+P30J93V1xxcTQxtdkL18tRITPvsdB/Muo5XaDsuf6YvQzq4tPDGRbVu4NRsf/nwC7bQabP/XADiq7KQeyeqZ8vubN24gaiEKhYB3nwyG2k6BPSdKsOr3xv01W3G5DmOWpuBg3mW0drTH15PDGFaIzIAFXHljYCFqQZ3dnfDvQTVLQ29tzsL5y9caPP7c5WsYtSQZRwvK0dZZjdVTwtG7Q2szTEpENXfADQRQU8Bt6RtAkmkYWIha2LP3dcZdHVvjiq4ar67LuGWh73RJBUYtTsapkgq0b+2AxKnh8PdyNvO0RLYtsqeHsYA7mwVcWWFgIWphSoWA+U/2hkqpwM7sYqxJP3fTMdkF5Yhekoxzl6/Bz90JidPC4evuVM/ViKglsYArXwwsRGbQ1cMZ/4jsBgB4Y+NhFJZdN37tYN5ljF6ajOJyHXp6u+DbaeFo19pBqlGJbF4nNydMG1BzY8Y3N/EOuHLBwEJkJlMf8ENQey3KrlfjtXWZEEURv+VcwNOf/obLV6twV8fWWDW5H9xb8c/5iaT2XEQXFnBlhoGFyEzslArMj+4Ne6WA7VmFmL3hMMYvT8UVXTX6d3HDlxPDoHW0l3pMIgKgsWcBV24YWIjMqIeXC14YWLM0tDL5DHTVBjzUwwPLn+kLJzXv+UAkJ5E9PfBgDw8WcGWCgYXIzJ4b2AUB3jU3SHqstzcWjwuBxl4p8VRE9FeCICB+aAALuDLBwEJkZvZKBVZN7YcvJobigzF3wV7J/w2J5Kp2AfcNFnAlxVdKIgm4aOxxf7e2UCoad6t+IpLOjQJuful1fMQCrmQYWIiIiBpQu4D7KQu4kmFgISIiug0WcKXHwEJERHQbfy3g/sgCrtkxsBARETUC74ArLQYWIiKiRmIBVzoMLERERI2ksVdiNgu4kmBgISIiMkFkgCcLuBJgYCEiIjIRC7jmx8BCRERkok5uTpheq4BboWMBt6UxsBARETXB9Igu8HGtKeB+vIMF3JbGwEJERNQEGnsl4h/7s4B7oogF3JbEwEJERNREkQGeeIgFXLNgYCEiIroD8UMDobJTYM8JFnBbEgMLERHRHejo5sgCrhkwsBAREd2h2gVc3gG3ZTCwEBER3aHaBdyEPSzgtgQGFiIiombAAm7LYmAhIiJqJrULuD9ksIDbnBhYiIiImkntAu5/N7OA25wYWIiIiJoRC7gtg4GFiIioGWnslZg9lAXc5sbAQkRE1Mwe6skCbnNjYCEiImoBLOA2LwYWIiKiFtDRzRHPRbCA21wYWIiIiFrItAEs4DYXBhYiIqIWUruA++luFnDvRJMCy6JFi+Dr6wuNRoOwsDCkpqbe8tiIiAgIgnDTNmTIkHqPnzZtGgRBwPvvv9+U0YiIiGTloZ6eiOzpgWoDC7h3wuTAsnr1asTGxiI+Ph7p6ekIDg5GVFQUioqK6j1+7dq1yM/PN26ZmZlQKpWIjo6+6dh169YhJSUF7dq1M/2REBERydSsx1jAvVMmB5aFCxdi8uTJiImJQUBAABYvXgxHR0csX7683uNdXV3h5eVl3LZt2wZHR8ebAsu5c+fw4osv4quvvoK9vX3THg0REZEM1S7gvrmJBdymMCmwVFZWIi0tDZGRkX9eQKFAZGQkkpOTG3WNhIQEjBkzBk5OTsZ9BoMB48aNw4wZMxAYGHjba+h0OpSVldXZiIiI5GzagC7o6OqIgjIWcJvCpMBSUlICvV4PT0/POvs9PT1RUHD7t7hSU1ORmZmJSZMm1dk/b9482NnZ4e9//3uj5pg7dy60Wq1x8/HxafyDICIikoDGXon4oQEAbhRwyyWeyLKY9a+EEhISEBQUhNDQUOO+tLQ0fPDBB1ixYgUEQWjUdeLi4lBaWmrc8vLyWmpkIiKiZlO7gBvPAq5JTAos7u7uUCqVKCwsrLO/sLAQXl5eDZ5bUVGBVatWYeLEiXX27969G0VFRejYsSPs7OxgZ2eHM2fO4F//+hd8fX3rvZZarYaLi0udjYiIyBLEDw2E2k6BX09cYAHXBCYFFpVKhZCQECQlJRn3GQwGJCUlITw8vMFzExMTodPpMHbs2Dr7x40bh0OHDuHAgQPGrV27dpgxYwZ++uknU8YjIiKSPR9XR0xnAddkdqaeEBsbiwkTJuCee+5BaGgo3n//fVRUVCAmJgYAMH78eLRv3x5z586tc15CQgJGjBgBNze3Ovvd3Nxu2mdvbw8vLy/4+/ubOh4REZHsTRvQBWvTzyH34lV8+PNxxD3SU+qRZM/kwDJ69GgUFxdj1qxZKCgoQJ8+fbBlyxZjETc3NxcKRd03brKzs7Fnzx5s3bq1eaYmIiKyYBp7JWYPC8CzK/YhYfcpRId0QFcPZ6nHkjVBtILGT1lZGbRaLUpLS9lnISIiizFp5e/YnlWEe7u64cuJYY3+4xNrYcrvb36WEBERkURqF3A3Z+RLPY6sMbAQERFJxMfVEc9FdAUA/HdTFgu4DWBgISIiktDUAX7GO+B++PNxqceRLQYWIiIiCd0o4AJAwu5TvAPuLTCwEBERSezBHp6I7OnJO+A2gIGFiIhIBuKHBrCA2wAGFiIiIhlgAbdhDCxEREQywQLurTGwEBERyQQLuLfGwEJERCQjtQu4s9azgHsDAwsREZHM3Cjg7j3JAu4NDCxEREQyU7uA++amI7jCAi4DCxERkRzdKOAWlunwURILuAwsREREMlSngLuHBVwGFiIiIpliAfdPDCxEREQyVruAu+mQ7RZwGViIiIhkzMfVEc8P/OMOuJttt4DLwEJERCRzUx7wQyc32y7gMrAQERHJnMZeifihfxZwjxfaXgGXgYWIiMgCPNjDEw8H1BRw4zfYXgGXgYWIiMhCzHrMdgu4DCxEREQWwpYLuAwsREREFsRWC7gMLERERBZEY6/E7KGBAGyrgMvAQkREZGEG9vAwFnBt5Q64DCxEREQW6EYBNznHNgq4DCxEREQWyNYKuAwsREREFqp2AfdDKy/gMrAQERFZKI29ErOH1RRwl1t5AZeBhYiIyIIN9LeNAi4DCxERkYWrXcDdaKUFXAYWIiIiC+fj6ogX/ijgvmWlBVwGFiIiIisw2coLuAwsREREVsDaC7gMLERERFZioL8HBllpAZeBhYiIyIq8bqUFXAYWIiIiK2KtBdwmBZZFixbB19cXGo0GYWFhSE1NveWxEREREAThpm3IkCHGY2bPno0ePXrAyckJbdq0QWRkJH777bemjEZERGTzJj/gB18rK+CaHFhWr16N2NhYxMfHIz09HcHBwYiKikJRUVG9x69duxb5+fnGLTMzE0qlEtHR0cZjunfvjo8//hgZGRnYs2cPfH19MWjQIBQXFzf9kREREdkojb0S8bUKuMesoIAriCY2csLCwtC3b198/PHHAACDwQAfHx+8+OKLmDlz5m3Pf//99zFr1izk5+fDycmp3mPKysqg1Wqxfft2PPTQQ7e95o3jS0tL4eLiYsrDISIislpTPt+HrUcK0c/PFd9M7gdBEKQeqQ5Tfn+b9A5LZWUl0tLSEBkZ+ecFFApERkYiOTm5UddISEjAmDFjbhlWKisrsXTpUmi1WgQHB9d7jE6nQ1lZWZ2NiIiI6nr9sQBo7BVIyblo8QVckwJLSUkJ9Ho9PD096+z39PREQUHBbc9PTU1FZmYmJk2adNPXNm3ahFatWkGj0eC9997Dtm3b4O7uXu915s6dC61Wa9x8fHxMeRhEREQ2wcfVEc9H1BRw/7vJsgu4Zv0roYSEBAQFBSE0NPSmrw0cOBAHDhzA3r17MXjwYIwaNeqWvZi4uDiUlpYat7y8vJYenYiIyCLdKOAWlevwwfZjUo/TZCYFFnd3dyiVShQWFtbZX1hYCC8vrwbPraiowKpVqzBx4sR6v+7k5ISuXbuiX79+SEhIgJ2dHRISEuo9Vq1Ww8XFpc5GREREN6t9B9zPfj1tsQVckwKLSqVCSEgIkpKSjPsMBgOSkpIQHh7e4LmJiYnQ6XQYO3Zso76XwWCATqczZTwiIiKqR0SdO+BmWuQdcE1eEoqNjcWyZcuwcuVKZGVlYfr06aioqEBMTAwAYPz48YiLi7vpvISEBIwYMQJubm519ldUVODVV19FSkoKzpw5g7S0NDz77LM4d+5cnT99JiIioqarXcDdcPC81OOYzM7UE0aPHo3i4mLMmjULBQUF6NOnD7Zs2WIs4ubm5kKhqJuDsrOzsWfPHmzduvWm6ymVShw9ehQrV65ESUkJ3Nzc0LdvX+zevRuBgYFNfFhERERU24074C7Yegxvbc7CQz090UptcgyQjMn3YZEj3oeFiIjo9nTVekS99wtOX7iKyfd3xmtDAiSdp8Xuw0JERESWS233ZwF3uYUVcBlYiIiIbEiEvweiAj2ht7ACLgMLERGRjbHEAi4DCxERkY3p0KamgAsAb23OQvn1Koknuj0GFiIiIhtU+w64HyYdl3qc22JgISIiskF/LeBmF8i7gMvAQkREZKMsqYDLwEJERGTDbhRwfzsl7wIuAwsREZENs5QCLgMLERGRjatdwP1guzwLuAwsRERENq52AfezvfIs4DKwEBERkewLuAwsREREBEDeBVwGFiIiIgJQU8B98cFuAORXwGVgISIiIqNJ93dGZ3cn2RVwGViIiIjISG2nRPzQAADyKuAysBAREVEdEf4eGBzoJasCLgMLERER3eT1ofIq4DKwEBER0U3at3YwFnD/K4MCLgMLERER1etGAbdYBgVcBhYiIiKq11/vgHu6pEKyWewk+85EREQkewO6t8W4fp1wj28bdHJzlGwOBhYiIiJq0Jsjekk9ApeEiIiISP4YWIiIiEj2GFiIiIhI9hhYiIiISPYYWIiIiEj2GFiIiIhI9hhYiIiISPYYWIiIiEj2GFiIiIhI9hhYiIiISPYYWIiIiEj2GFiIiIhI9hhYiIiISPas4tOaRVEEAJSVlUk8CRERETXWjd/bN36PN8QqAkt5eTkAwMfHR+JJiIiIyFTl5eXQarUNHiOIjYk1MmcwGHD+/Hk4OztDEIRmvXZZWRl8fHyQl5cHFxeXZr22teFz1Xh8rhqPz5Vp+Hw1Hp+rxmup50oURZSXl6Ndu3ZQKBpuqVjFOywKhQIdOnRo0e/h4uLCH+hG4nPVeHyuGo/PlWn4fDUen6vGa4nn6nbvrNzA0i0RERHJHgMLERERyR4Dy22o1WrEx8dDrVZLPYrs8blqPD5XjcfnyjR8vhqPz1XjyeG5sorSLREREVk3vsNCREREssfAQkRERLLHwEJERESyx8BCREREssfAAmDRokXw9fWFRqNBWFgYUlNTGzw+MTERPXr0gEajQVBQEH744QczTSo9U56rFStWQBCEOptGozHjtNL55ZdfMHToULRr1w6CIOD777+/7Tk7d+7E3XffDbVaja5du2LFihUtPqccmPpc7dy586afK0EQUFBQYJ6BJTR37lz07dsXzs7O8PDwwIgRI5CdnX3b82zxNaspz5WtvmZ98skn6N27t/GmcOHh4fjxxx8bPEeKnymbDyyrV69GbGws4uPjkZ6ejuDgYERFRaGoqKje4/fu3YunnnoKEydOxP79+zFixAiMGDECmZmZZp7c/Ex9roCauyLm5+cbtzNnzphxYulUVFQgODgYixYtatTxp06dwpAhQzBw4EAcOHAAL730EiZNmoSffvqphSeVnqnP1Q3Z2dl1frY8PDxaaEL52LVrF55//nmkpKRg27ZtqKqqwqBBg1BRUXHLc2z1NaspzxVgm69ZHTp0wDvvvIO0tDTs27cPDz74IIYPH47Dhw/Xe7xkP1OijQsNDRWff/5543/r9XqxXbt24ty5c+s9ftSoUeKQIUPq7AsLCxOnTp3aonPKganP1WeffSZqtVozTSdfAMR169Y1eMzLL78sBgYG1tk3evRoMSoqqgUnk5/GPFc7duwQAYiXLl0yy0xyVlRUJAIQd+3adctjbPk1q7bGPFd8zfpTmzZtxE8//bTer0n1M2XT77BUVlYiLS0NkZGRxn0KhQKRkZFITk6u95zk5OQ6xwNAVFTULY+3Fk15rgDgypUr6NSpE3x8fBpM7LbOVn+u7kSfPn3g7e2Nhx9+GL/++qvU40iitLQUAODq6nrLY/izVaMxzxXA1yy9Xo9Vq1ahoqIC4eHh9R4j1c+UTQeWkpIS6PV6eHp61tnv6el5y/XwgoICk463Fk15rvz9/bF8+XKsX78eX375JQwGA/r374+zZ8+aY2SLcqufq7KyMly7dk2iqeTJ29sbixcvxpo1a7BmzRr4+PggIiIC6enpUo9mVgaDAS+99BLuvfde9OrV65bH2eprVm2Nfa5s+TUrIyMDrVq1glqtxrRp07Bu3ToEBATUe6xUP1NW8WnNJE/h4eF1Enr//v3Rs2dPLFmyBG+++aaEk5El8/f3h7+/v/G/+/fvj5MnT+K9997DF198IeFk5vX8888jMzMTe/bskXoU2Wvsc2XLr1n+/v44cOAASktL8d1332HChAnYtWvXLUOLFGz6HRZ3d3colUoUFhbW2V9YWAgvL696z/Hy8jLpeGvRlOfqr+zt7XHXXXfhxIkTLTGiRbvVz5WLiwscHBwkmspyhIaG2tTP1QsvvIBNmzZhx44d6NChQ4PH2upr1g2mPFd/ZUuvWSqVCl27dkVISAjmzp2L4OBgfPDBB/UeK9XPlE0HFpVKhZCQECQlJRn3GQwGJCUl3XLtLjw8vM7xALBt27ZbHm8tmvJc/ZVer0dGRga8vb1bakyLZas/V83lwIEDNvFzJYoiXnjhBaxbtw4///wzOnfufNtzbPVnqynP1V/Z8muWwWCATqer92uS/Uy1aKXXAqxatUpUq9XiihUrxCNHjohTpkwRW7duLRYUFIiiKIrjxo0TZ86caTz+119/Fe3s7MQFCxaIWVlZYnx8vGhvby9mZGRI9RDMxtTnas6cOeJPP/0knjx5UkxLSxPHjBkjajQa8fDhw1I9BLMpLy8X9+/fL+7fv18EIC5cuFDcv3+/eObMGVEURXHmzJniuHHjjMfn5OSIjo6O4owZM8SsrCxx0aJFolKpFLds2SLVQzAbU5+r9957T/z+++/F48ePixkZGeI//vEPUaFQiNu3b5fqIZjN9OnTRa1WK+7cuVPMz883blevXjUew9esGk15rmz1NWvmzJnirl27xFOnTomHDh0SZ86cKQqCIG7dulUURfn8TNl8YBFFUfzoo4/Ejh07iiqVSgwNDRVTUlKMXxswYIA4YcKEOsd/++23Yvfu3UWVSiUGBgaKmzdvNvPE0jHluXrppZeMx3p6eoqPPvqomJ6eLsHU5nfjT2//ut14fiZMmCAOGDDgpnP69OkjqlQq0c/PT/zss8/MPrcUTH2u5s2bJ3bp0kXUaDSiq6urGBERIf7888/SDG9m9T1PAOr8rPA1q0ZTnitbfc169tlnxU6dOokqlUps27at+NBDDxnDiijK52dKEEVRbNn3cIiIiIjujE13WIiIiMgyMLAQERGR7DGwEBERkewxsBAREZHsMbAQERGR7DGwEBERkewxsBAREZHsMbAQERGR7DGwEBERkewxsBAREZHsMbAQERGR7DGwEBERkez9P3v9rsdKS2yHAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.train()\n",
    "for epoch in tqdm(range(4), leave=False):\n",
    "    train_loss = 0.0\n",
    "    for img, _ in tqdm(dataloader, leave=False):\n",
    "        # Convert image to a signal with L = 784 and dimension 1\n",
    "        img = img.to(device).reshape(-1, 784, 1)\n",
    "        # Shift and pad with zero to generate input\n",
    "        x = torch.cat([torch.zeros(img.size(0), 1, 1, device=device), img[:, :-1, :]], dim=1)\n",
    "        # For each sample i, predict the next sample (i+1) from the slice between 0 to i\n",
    "        y_hat = model(x)\n",
    "        \n",
    "        y = torch.mul(img, 255.0).to(torch.int64).to(device)[:,:,0]\n",
    "        loss = criterion(y_hat.reshape(-1, 256), y.reshape(-1))\n",
    "\n",
    "        loss_val = loss.detach().cpu().item()\n",
    "        train_loss += loss_val / len(dataloader)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    losses.append(train_loss)\n",
    "    print(f\"Epoch {len(losses)} loss: {train_loss}\")\n",
    "\n",
    "plt.plot(losses)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "da0a5b9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def image_sample(img=None, start=0, maxp=False):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        if img is not None and start > 0:\n",
    "            img = img.to(device).reshape(-1, 784, 1)[0, :start, :]\n",
    "        sample = generate_sample(model, img, maxp).cpu()\n",
    "        plt.figure(figsize=(2,2))\n",
    "        plt.axis('off')\n",
    "        plt.imshow(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "a2ad4f4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAES0lEQVR4nO3dv6scZRTH4XdFI7FQTCN2IYqFYGzSCMHKoIIKQtAmRRoLQbTRysbKwsKIFv4BVjaCBItgEUH8VVoIWiiYxiYGRLERvdbKOTJzdzbxO/s85eG9u8PNJy/MzN3ZzcHBwcGAMDfd6AOAwxAukYRLJOESSbhEEi6RhEsk4RJJuES6eerCzWazy+OAMcYYU2/k2nGJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIk/8ed9VeLmZHm7Uni9nZGe/1bjP/opi9N+N194wdl0jCJZJwiSRcIgmXSJupz8ddxad832/mc64K7Mr3xeyRZu2VXR7IjeVTvqyacIkkXCIJl0jrveVbnYgtcRL2bTG71Kw9UcyebNbeU8zONWtfb+Z7xI5LJOESSbhEEi6RhEuk/KsKp5r50zNe45ti9lSz9mox+61Ze6SYfdmsfbCYHWvWYsclk3CJJFwiCZdI+Sdndzfz6s+Hq5OwMcZ4tJj9dLjD+Yfq08P3z/j5jxY4hpWy4xJJuEQSLpGESyThEin/qsLFZn5vMfu1WXttoWP5t2eL2S07eq89Y8clknCJJFwiCZdI+SdnnR+v43u90szvm/EaX02cMcaw4xJKuEQSLpGESyThEmm/Huy8gBeK2dvN2uo39mez9nIxOzPpiNbFg51ZNeESSbhEEi6R1nvLd0svNfM3i9m004n/9vUCr7FP7LhEEi6RhEsk4RJJuERyVaHx0Iy1c64qdDfOq2c439Gs/WXG+62VHZdIwiWScIkkXCI5OWu808yrL915foH3O1/Mqq8NHmOMNxZ4v3R2XCIJl0jCJZJwiSRcIrmq0PhsxvzDZu2rxezhZu1fxezOZi12XEIJl0jCJZJwieTkbAEfN/NTxex0s7b6O92jhzucvWDHJZJwiSRcIgmXSMIlkgc779ATxay7PVz9I/zcrL3rcIcTwYOdWTXhEkm4RBIukdzy3aHPi9m1Zm31CKbbFjyWtbHjEkm4RBIukYRLJOESyVWFHTpezLorBdUN9e4KBHZcQgmXSMIlknCJFHVydnsxe7FZ+0Axe6ZZWz3+qPNpMbvQrH2tmN06470+mLF239hxiSRcIgmXSMIlknCJFPUp3xPF7Ltm7Zz/kUt8F++2P/9DMTvZrP19y2P4P/MpX1ZNuEQSLpGES6SoW77V9+heatY+vuV7zTkJ604nqtfobi+/VczWfBK2LTsukYRLJOESSbhEEi6Rom75znG2mD3WrP2jmD3XrK1+Wd3//uoKwplm7SfNfN+45cuqCZdIwiWScIm02pMzMjk5Y9WESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEmny10VNfaYTXA92XCIJl0jCJZJwiSRcIgmXSMIlknCJJFwi/Q1xw3de2WR4igAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAADSklEQVR4nO3dMcokVRhA0SoxEhFEjDSaQA0UR9fgdsTQNUxsbGJmMrnuQGORAZlIjAwMRBCFKTfwCsqxXju3+pzwo4IXXD6o1/33v27bti0Q89L/fQB4HsIlSbgkCZck4ZIkXJKES5JwSRIuSS8ffXBd15nngGVZluXoB7k2LknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSdPivfJnsncHsyc6znw1mX5x4lgAblyThkiRckoRLkpezF8VHg9mznWd/mXmQBhuXJOGSJFyShEuScElyq/CieDiY/bHz7OOJ54iwcUkSLknCJUm4JHk5u7UPduafDmZfzTxIm41LknBJEi5JwiVJuCS5Vbi1d3fmrwxmX888SJuNS5JwSRIuScIlad22bTv04LrOPst9+H5n/uZg9v7Os3vf072AgznauDQJlyThkiRckoRLkluFid4bzH7cefbvweztnWd/fb7jJLhV4NKES5JwSRIuSb6PO9Frg9neq8fTwezKL2H/lY1LknBJEi5JwiVJuCS5VZjo48Fs74Pzb2Ye5IJsXJKES5JwSRIuSV7OJnr1Xzz7w7RTXJONS5JwSRIuScIlSbgkuVWY6PXBbO+L5F/OPMgF2bgkCZck4ZIkXJK8nE301mB24d9kvikblyThkiRckoRLknBJcqsw0YeD2aObn+KabFyShEuScEkSLklezk7wxs78wWD208yD3BEblyThkiRckoRLknBJcqtwgs935qPfDnsy8yB3xMYlSbgkCZck4ZLk5ewEf+3MR/9h58+ZB7kjNi5JwiVJuCQJlyThkuRW4QSjL4wvy7I8G8zcKpzDxiVJuCQJlyThkuTlbKLRVvjt5qe4JhuXJOGSJFyShEuScElyq3CCT3bm3w5mv888yB2xcUkSLknCJUm4JHk5O8HPO/PvbnqK+2LjkiRckoRLknBJEi5J67Zt26EH19EvYcG5DuZo49IkXJKES5JwSRIuScIlSbgkCZck4ZIkXJKES5JwSRIuScIlSbgkCZck4ZIkXJKES5JwSRIuSYd/O+zoX1/CLdi4JAmXJOGSJFyShEuScEkSLknCJUm4JP0DG4NAhRfyDyoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAEYUlEQVR4nO3dP6scZRjG4XeNfwpRBI2tWAgpVNQigkVaURtJ0DSKYGPnN/ATCMFCS0EL0UYEQXtFLAQFG4NYiChGkYCFWojx2KR8HpiJuydzz1xX+fCePZOTHy/M7M7O7ujo6GhAmBuu9wHAtRAukYRLJOESSbhEEi6RhEsk4RJJuES6cerC3W53yOOAMcYYU9/IteMSSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEmnyXb5cdWsxe7VZ+1Ix+7JZ+0wx+2HSEW2SHZdIwiWScIkkXCLtpj68xFcwXXVfMftmxs93W8XLxeyNGa+7Er6CiVUTLpGESyThEkm4RPKWb+dkM3/rOA+Cjh2XSMIlknCJJFwiOTkbo3679elm7ekDHcOZYtZtK18Xs0/3eCwB7LhEEi6RhEsk4RJJuETyQfIxxrhSzP490O/qtoo5v6+6+/d8s7a7q3ihfJCcVRMukYRLJOESaVsnZx838yeK2aFOzi438z+K2T17+H0n9vAax8jJGasmXCIJl0jCJZJwibTaqwrV0b7XrD038efHqC82dJ/hfr+Yvdusfb6YXWjWzpF2p4CrCqyacIkkXCIJl0irPTmrnGrm1cdxb2rW/lnM9vFwnDuLWfd90XcVs+4/0ckZLIhwiSRcIgmXSMIl0qauKqR5s5m/MOM1XFWABREukYRLJOESycnZglWPDR5jjC+K2W3N2nuL2Y/XdjjHwskZqyZcIgmXSMIlknCJlPaO4KZ818xvLmZbu+ZjxyWScIkkXCIJl0hOzgL9U8w+atb+dMgDuY7suEQSLpGESyThEkm4RHJVYcGea+a3F7NuB5r2sew8dlwiCZdIwiWScInk5GzBnm3m1ZN/XjnkgSyQHZdIwiWScIkkXCIJl0ibuqpwSzOvvnfrr2ZtN/+/Xixmjzdrfy9mP+/vUCLYcYkkXCIJl0jCJdIiT84eauaPFrOnmrV3F7MTzdpHill3d+z3xaw7MbpUzB5o1j5WzKq7eceon8bza7N2rey4RBIukYRLJOESSbhEWuTjou5o5p8Vs1MzXvdiM79SzB5s1lZ/rO4vM+cO22rtV83a0zNeN43HRbFqwiWScIkkXCIt8i3fTnUi9kGz9vVi9kmz9mQxe7JZ+1ox656jW+lO5Kr5LzNed2vsuEQSLpGESyThEkm4RFrkVYXuTtqzxexCs7a7glD5rZi93aytPrh+rllbfRC8+zB7dVXh82YtdlxCCZdIwiWScIm0yJOzv5v5h8Xs4WZt9Q/r7pqtnGnm9894jcvFrLpDd4wxvi1m78z4XVtjxyWScIkkXCIJl0jCJdIi7/Jlu9zly6oJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiTf7usKl3X8JxsOMSSbhEEi6RhEsk4RJJuEQSLpGESyThEuk/IoWTUBlGygsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "image_sample(MNIST_TEST[1][0], 28*10)\n",
    "image_sample(MNIST_TEST[2][0], 28*10)\n",
    "image_sample(MNIST_TEST[3][0], 28*10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "7a9ba373",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAADwUlEQVR4nO3dv6u2cxzA8evWk0IiMjBQj5QYZFR+LHZlEINVMSoLm3/CQkmUwSaxUwwmNoPBgzIpkx9xG6yfb+7jOedx3vd5vcZPn07XuXt31XVd51z3br/f7zeIue7/PgD4L4RLknBJEi5JwiVJuCQJlyThkiRcki4durjb7c7yOGDbtm079EGuMy5JwiVJuCQJlyThkiRckoRLknBJEi5JwiVJuCQJlyThkiRckoRLknBJEi5JwiVJuCQJlyThkiRckoRLknBJEi5JwiVJuCQJlyThkiRckoRLknBJEi5JwiVJuCQJlyThknTwt+5wcpeH2ZOL3Y+H2ZVTPJZj44xLknBJEi5JwiVJuCS5q3BCdw6z1xe7zw2zGxe7Xw+zVxe7Hy3mF4kzLknCJUm4JAmXpNTF2XRhdMNid5o/v9j9fJg9sth9ZZjtFrvfD7MvF7uPD7MnFrsuzpxxiRIuScIlSbgkCZekc3lX4bbF/Mdh9sdi92p/sb8W81+H2XuL3ReG2UuL3UeH2c+LXZxxiRIuScIlSbgk7fb7/f6gxd3qwebpu30xf3uYPbDY/WSYvbXYnR7N/rnYvWWYfbvYnby7mN88zJ46wc89Fgfm6IxLk3BJEi5JwiVJuCSdy7sKx+KxYfbOYveHYTY9Bj527ipw1IRLknBJEi5JLs7O0FfD7MHF7l3D7KdTPJYKF2ccNeGSJFyShEuScEk6l//lW/P0Yj7dQXhtsXsR7yBcDWdckoRLknBJEi5JLs62bbtpmN2z2L11mL252J1eD/XFIQfEv3LGJUm4JAmXJOGSJFySLtQfkt+7mH84zO4/wc9dfYDTJ/bdYvebYTa9p2zbtu39YfbBYvfKYn5e+UNyjppwSRIuScIl6UJdnK1ewDw93p1eibRt8yuUVo98fx9mq0/x5WF232L37mG2uvB8Y5h9ttj9dJhd62/+cXHGURMuScIlSbgkCZekC3VX4Y7F/Jdh9ttZHshVun6YPbzYfXGYPbvYnb4r+fJBR3R63FXgqAmXJOGSJFySLtTFGf94aDGfXg/1zFkeyMDFGUdNuCQJlyThkiRcktxV4FxxV4GjJlyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEvSpUMXD/2OVbgWnHFJEi5JwiVJuCQJlyThkiRckoRLknBJ+hvo6nUkAL+y/QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAEh0lEQVR4nO3dP+jtcxzH8e9x5UZ0JUxCIYMVm2JhEO7AoqxW8meQkYEMMllsFslwN6WU0eJ2y8Tgb4RIiEviGJTp/e6c7+98z9f3dX6Px/ju0+nz+/2efer7/Z5zfqv1er0eIMwF//cG4CiESyThEkm4RBIukYRLJOESSbhEEi6RLtx24Wq12uc+YBiGYdj2Qa4Tl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwibf3FzktwdzF7Z/Zd5Lu6mZ8sZl/ucyM7cOISSbhEEi6RhEukRV6cPd7MnylmTzVrXy9mp5q1P23aULBHi9n1zdqbitlD021lUk5cIgmXSMIlknCJJFwiLfKuwlfN/Kpi9lyz9r5idkmz9qNi9lqz9uNi9lezdk5vNvO7itkVzdrzE+1lDk5cIgmXSMIlknCJtFqv1+utFq5W+97Lf95o5g8Wsyl2Vf0Cfm7WvlfM3mrW/ljMuvfCvl3MTjRrny5mjzVrq5+tO61+LWaXNWv3ZcscnbhkEi6RhEsk4RJJuERa5F2FT5r5dSNeY8zV9N/FrPtpt7vm7Y35Lf7ezKtP407ht2J2Z7P27J724K4CB024RBIukYRLpEW+H/eLZn5tMTvTrH2ymF3TrH2gmN3WrL2jmI25kBtzcdZdhI15jTEXky8Ws31dhO3KiUsk4RJJuEQSLpGES6RFPvK9uJlfXsyqN2sPQ/+4dB+qTx8PwzD8UcxOj3iNi5q1z2/a0AavNPMndnzdKXjky0ETLpGESyThEmmRF2f865Zmfq6YdSdQ9cd9uFnbfY3TnFyccdCESyThEkm4RBIukRb5RvLj6JFi9nKztjptumvxZ4vZEu4e7MqJSyThEkm4RBIukTzyndntzfzdYtb9l6DqD/Z5s/aGjTtaFo98OWjCJZJwiSRcIgmXSB75zuzVZl7dQeiur6s7CKePtJtcTlwiCZdIwiWScInk4mxm3f8IHuPTYvbhBK+bxIlLJOESSbhEEi6RhEskdxVm9t0Er3FugtdI58QlknCJJFwiCZdIPuU7s2+a+ZXFrLuQu7WYfX207SyOT/ly0IRLJOESSbhEEi6RPPLdoxuL2clmbXXPpvsC5kO5g7ALJy6RhEsk4RJJuERycbZH7xezU83a6kHn2Qn3cmicuEQSLpGESyThEkm4RPJG8glUbwIfhvrR7Ilm7flidunRthPNG8k5aMIlknCJJFwieeQ7ge+b+Z/FrLs4+2WivRwXTlwiCZdIwiWScIkkXCK5qzCBm5t59ZD8h2btvRPt5bhw4hJJuEQSLpGESyQXZxO4v5lXX7f0bbPWJ3rHceISSbhEEi6RhEsk4RLJXYUJnGnmLxSzD/a4j129VMy6Oyb3FLPPptvKRk5cIgmXSMIlknCJ5CuYWBRfwcRBEy6RhEsk4RJJuETa+pHvtld7MAcnLpGESyThEkm4RBIukYRLJOESSbhEEi6R/gGiPo+vsDs9JAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAEQ0lEQVR4nO3dPatcZRSG4XfyScAqmiYgRFBBUSuTShEtBBsrQRAbtQhY2KVQsLFQm/yBJAiCNgFBEcTOykIb7S1sBG1UBBGNmrFIF9bC/YZzkjx7rqtcLIeR3GzYH2f2ZrvdbgeEOXCrvwDcCOESSbhEEi6RhEsk4RJJuEQSLpGES6RDSxc3m81+fg8YY4yx9EauIy6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RBIukYRLJOESSbhEEi6RhEsk4RJJuEQSLpGESyThEkm4RFr8w87sryPF7Fize7CY/d7sXilmZ5vdF4vZI83uR8Xs5WZ3PzjiEkm4RBIukYRLpM124WtOvHVnbzzczF8vZs83u9U/2AfN7lfF7LVm9/5mvvQ77MWZvrfusGrCJZJwiSRcIgmXSK4q7IG7m/n5YvZks3u8mHX/MD8Xs3+b3b+L2V3N7tFmvpSrCvA/hEsk4RJJuETyPO6kE8Xs1Wb3uWJ2tdmtTkmqZ2nHGOPNYnah2a080MzvKWafTXzuzeSISyThEkm4RBIukYRLJLd8J1UPZp9udpfdvLzm02L2drP79cTnpnHLl1UTLpGESyThEskt38ZTzbw6Eetu4/5VzLrbwx8Ws3+aXRxxCSVcIgmXSMIlknCJ5KpC43Azr64gdDfDvyxm79/Y1+E6jrhEEi6RhEsk4RLJ87iN7o033xaze5vd6k04F5vdd4pZ9VNLa+d5XFZNuEQSLpGESyThEslVhUmniln317jd654qnxez6t26Y4zx68TnpnFVgVUTLpGESyThEsnJ2aRni9nHze7MTzBVuy80u5cnPjeNkzNWTbhEEi6RhEsk4RIp6q98H1w4G2OMX4rZyWb3jmL2ULNb3cbtfuPrt2J2Z7Nb+Wlid9c44hJJuEQSLpGES6Sok7P3itnMG2/24qb198Xsk2b3TDE73uz+Wcx+XPSNdpMjLpGESyThEkm4RBIukaKuKtw3sVtdQegeUf6hmFWvehpjjGeK2ePN7kvNvHKpmH038d/vGkdcIgmXSMIlknCJFPVXvm8Us8ea3aeLWfc/Wv3YcvfO3XPF7K1m90gx+6bZfbSZ7xp/5cuqCZdIwiWScIkkXCJFXVWY8UQx+6LZnXktU/cgeKV6wLz7seY/Jj53zVxVYNWESyThEkm4RFrtyVnllWZenXC9O/G51bO0Y4xxduIzuMbJGasmXCIJl0jCJZJwibRTVxU6B4vZsWb3ajG70ux2P/hMz1UFVk24RBIukYRLJCdn3FacnLFqwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSMIlknCJJFwiCZdIwiWScIl0aOni0t90gpvBEZdIwiWScIkkXCIJl0jCJZJwiSRcIgmXSP8BrvmfsYDIx4QAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.manual_seed(0)\n",
    "image_sample()\n",
    "image_sample()\n",
    "image_sample()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "bc314e10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAADUklEQVR4nO3cMaokRRzA4R5XBVMzUxURgzXxCIKRgiCYeAzNjbyCBzASNhODPcHuHkAwMdDEREw0lTEwUqpg5tHl21/P94V/6s3r4EdBdc/06Xw+nzeIeeG+LwDuQrgkCZck4ZIkXJKES5JwSRIuScIl6cVLF55Op5XXAdu2bdulD3LtuCQJlyThkiRckoRLknBJEi5JwiVJuCQJlyThkiRckoRLknBJEi5JF38f99A+H8xemax9OJh9csX/+noyfzKYfXPF594YOy5JwiVJuCQJlyThknS69P24h/iV77eT+TV3BVb5aTB7f7L2l5UXcr/8ypdDEy5JwiVJuCQd95Hv6CC2xyHsx8Hs8WTt64PZh5O1bwxmn03WfjWZ3xA7LknCJUm4JAmXJOGS1L+r8N5k/vEVn/HDYPbRZO1vg9mfk7UvD2ZPJ2vfHcxenazFjkuTcEkSLknCJal/OHttMh99fXh0CNu2bftgMPv1bpfzL6NfD79zxd9/v8M1HJQdlyThkiRckoRLknBJ6t9V+G4yf3Mw+2Oy9vedruW/Ph3MXlr0v26MHZck4ZIkXJKES1L/cDbz8//4v76YzN+64jOeXThj2zY7LlHCJUm4JAmXJOGSdFsvdl7krx0+Y/R0+NEOn1vjxc4cmnBJEi5JwiXJ4WwHexzOHuzwGUfgcMahCZck4ZIkXJKES9Jxv0j+HLjsfMxd2HFJEi5JwiVJuCQ5nC3kIfk6dlyShEuScEkSLknCJcldhYU88l3HjkuScEkSLknCJcnhbCGPfNex45IkXJKES5JwSRIuSe4qLOSR7zp2XJKES5JwSRIuSQ5nC3nku44dlyThkiRckoRLknBJEi5JwiVJuCQJlyThkiRckoRLknBJEi5JwiVJuCQJlyThkiRckoRLknBJ8ivf58SXF874hx2XJOGSJFyShEuScElyV2Gh0YudZ+8Te3vlhRyQHZck4ZIkXJKES5LD2Q4e3PcF3CA7LknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JF387rDzefSaYrgfdlyShEuScEkSLknCJUm4JAmXJOGSJFyS/gYR1kC+F0ljdAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAK4AAACuCAYAAACvDDbuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAACsElEQVR4nO3csY3VQBRAURsRbU4LkIC0dEMTiGLogAwCCqECGtgSNmVowBZeZA//LueEIwejr6sneezvdYwxFoh58a83AH9DuCQJlyThkiRckoRLknBJEi5JwiXp5dEL13W9ch+wLMuyHH2Qa+KSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLkmH/+XLxV5vrP3cufbTxtrnE/cSYOKSJFyShEuScElyc3Yr3m+s/dq59uHKjTSYuCQJlyThkiRckoRLklOFW3G/sfa4c+33C/cRYeKSJFyShEuScElyczbbu531jxtrX67cSJuJS5JwSRIuScIlSbgkOVWY7c3O+t3G2tcrN9Jm4pIkXJKES5JwSVrHGOPQhet69V7+Dz921l9trL3duXbvPd1n4GCOJi5NwiVJuCQJlyThkuRUYbK9z4F921j7cOVGbpRTBZ414ZIkXJKES5L3cW+ECfI0fi+ShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGS5EXyyfxX+hwmLknCJUm4JAmXJDdnkx37wBB/YuKSJFyShEuScEkSLklOFSbzyPccJi5JwiVJuCQJlyQ3Z5N55HsOE5ck4ZIkXJKES5JwSXKqMJlHvucwcUkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuScEkSLknCJUm4JAmXJOGSJFyShEuSDztP5sPO5zBxSRIuScIlSbgkCZck4ZIkXJKES5JwSRIuScIlSbgkCZck4ZIkXJKES5JwSRIuScIlSbgkCZekw//yHWNcuQ94EhOXJOGSJFyShEuScEkSLknCJUm4JAmXpN+UFSdoFQgDKgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 200x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "image_sample(MNIST_TEST[1][0], 28*10, maxp=True)\n",
    "image_sample(MNIST_TEST[2][0], 28*10, maxp=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dcfb4d4",
   "metadata": {},
   "source": [
    "### Saving & Loading the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "cddfe78a",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model, \"./models/sashimi-mnist.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "ef777711",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = torch.load(\"./models/sashimi-mnist.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcad02b0",
   "metadata": {},
   "source": [
    "# NLL\n",
    "\n",
    "Here, we compute the average negative log likelihood (NLL) value on the test dataset.\n",
    "\n",
    "Note that [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) simply calculates the NLL by applying softmax internally:\n",
    "\n",
    "$$\n",
    "l_n = - \\log \\frac{\\exp(x_{n,y_n})}{\\sum_{c=1}^C \\exp(x_{n,c})}\n",
    "$$\n",
    "\n",
    "PyTorch uses natural log by default. We convert this to NLL in bits by dividing the result by $\\log(2)$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "6833d411",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average NLL: 0.7191216349601746\n",
      "Average NLL in base 2: 1.037473201751709\n",
      "Average NLL converted to probability: 0.4871799945831299\n"
     ]
    }
   ],
   "source": [
    "model.eval()\n",
    "criterion = torch.nn.CrossEntropyLoss()\n",
    "with torch.no_grad():\n",
    "    nll = 0.0\n",
    "    for img, _ in tqdm(testloader, leave=False):\n",
    "        # Convert image to a signal with L = 784 and dimension 1\n",
    "        img = img.to(device).reshape(-1, 784, 1)\n",
    "        # Shift and pad with zero to generate input\n",
    "        x = torch.cat([torch.zeros(img.size(0), 1, 1, device=device), img[:, :-1, :]], dim=1)\n",
    "        y = torch.mul(img, 255.0).to(torch.int64).to(device)[:,:,0]\n",
    "        # For each sample i, predict the next sample (i+1) from the slice between 0 to i\n",
    "        y_hat = model(x)\n",
    "        nll += criterion(y_hat.reshape(-1, 256), y.reshape(-1))\n",
    "    nll /= len(MNIST_TEST)\n",
    "\n",
    "    print(\"Average NLL:\", nll.item())\n",
    "    print(\"Average NLL in base 2:\", (nll / torch.tensor(2.0).log()).item())\n",
    "    print(\"Average NLL converted to probability:\", (-nll).exp().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ed4631",
   "metadata": {},
   "source": [
    "The result is worse than `S4-MNIST-Demo`. That's probably because SaShiMi is larger than vanilla S4 and hence it's harder to train. It requires more hyperparameter optimization and longer training to achieve a better performance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}