notebooks/07_Sequence_to_Sequence_Tasks_with_Transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Sequence-to-Sequence Tasks with Transformers using Python, PyTorch, and Huggingface\n",
        "\n",
        "## 1. Introduction to Sequence-to-Sequence Tasks\n",
        "\n",
        "### What is Sequence-to-Sequence?\n",
        "\n",
        "Sequence-to-sequence (seq2seq) is a concept in machine learning where both the input and the output are sequences. The core idea is to use a neural network model to map an input sequence to an output sequence. These models are particularly useful for tasks where the length of the input sequence can differ from the length of the output sequence.\n",
        "\n",
        "#### Applications of Seq2Seq:\n",
        "- **Machine Translation**: Translating a sentence from one language to another.\n",
        "- **Text Summarization**: Condensing a long article into a short summary.\n",
        "- **Speech Recognition**: Converting spoken language into written text.\n",
        "- **Image Captioning**: Describing the content of an image using a sentence.\n",
        "\n",
        "### How does Seq2Seq work?\n",
        "\n",
        "A typical seq2seq model consists of two main components:\n",
        "1. **Encoder**: This part of the network compresses the input into a \"context\" or \"memory\" which the decoder then uses to produce the output.\n",
        "2. **Decoder**: This part of the network unfolds the \"context\" to produce the output sequence.\n",
        "\n",
        "The encoder processes the input sequence and compresses this information into a context vector, which is a fixed-size vector. The decoder then uses this context vector to produce the output sequence. The entire model is trained end-to-end.\n",
        "\n",
        "---\n",
        "\n",
        "## 2. Understanding the Transformer Architecture\n",
        "\n",
        "The Transformer architecture, introduced in the paper [\"Attention Is All You Need\"](https://arxiv.org/abs/1706.03762), revolutionized the field of deep learning. It uses self-attention mechanisms to weigh input tokens differently, allowing for more flexible and context-aware representations.\n",
        "\n",
        "![Transformer Architecture](https://miro.medium.com/max/2824/1*1cHMDM5GSfZ8z9dZREKwIQ.png)\n",
        "\n",
        "---\n",
        "\n",
        "## 3. Setting Up the Environment\n",
        "\n",
        "First, let's set up our environment in Google Colab:\n"
      ],
      "metadata": {
        "id": "Npy6OUdcslfR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q torch torchvision\n",
        "!pip install -q transformers\n",
        "!pip install -q sentencepiece"
      ],
      "metadata": {
        "id": "7ZMlsHyWsl1Y"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Package Versions\n",
        "To ensure reproducibility, it's essential to know the versions of the packages used in this tutorial. Here's how you can check the versions of the key packages:"
      ],
      "metadata": {
        "id": "Pc7zt5SUwBQD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import transformers\n",
        "\n",
        "print(f\"PyTorch Version: {torch.__version__}\")\n",
        "print(f\"Transformers Version: {transformers.__version__}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MhBAIWO7wAYs",
        "outputId": "77f71b7e-aca9-4ebc-8474-fa45ca75d853"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PyTorch Version: 2.0.1+cu118\n",
            "Transformers Version: 4.33.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. Implementing a Seq2Seq Task with Transformers\n",
        "\n",
        "For this tutorial, we'll focus on machine translation as our seq2seq task. We'll translate English sentences to French.\n",
        "\n",
        "### 4.1. Loading the Model and Tokenizer\n",
        "\n",
        "We'll use the `t5-small` model from Huggingface's model hub. T5 (Text-to-Text Transfer Transformer) is designed for various seq2seq tasks.\n"
      ],
      "metadata": {
        "id": "VQHypHfMswd5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import T5ForConditionalGeneration, T5Tokenizer\n",
        "\n",
        "model_name = \"t5-small\"\n",
        "model = T5ForConditionalGeneration.from_pretrained(model_name)\n",
        "tokenizer = T5Tokenizer.from_pretrained(model_name)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6u_S-zj5snsx",
        "outputId": "8d3736b1-af2f-42eb-9611-4d167d5a0bec"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.2. Translation Function\n",
        "\n",
        "Let's define a function to handle the translation:"
      ],
      "metadata": {
        "id": "JAwdIZOjs4Yd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def translate(text, model, tokenizer):\n",
        "    input_ids = tokenizer.encode(\"translate English to French: \" + text, return_tensors=\"pt\")\n",
        "    # Using the generate method, which handles the encoder-decoder process internally\n",
        "    outputs = model.generate(input_ids)\n",
        "    translated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
        "    return translated_text"
      ],
      "metadata": {
        "id": "_VXNTDCas2dM"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.3. Testing the Translation"
      ],
      "metadata": {
        "id": "I7UV1s4-s8AH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Hello, how are you?\"\n",
        "translated_text = translate(text, model, tokenizer)\n",
        "print(f\"Translated Text: {translated_text}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g9O4eH02s6Tq",
        "outputId": "41309437-7d0b-4a0a-8e8e-29eef3985ab3"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1260: UserWarning: Using the model-agnostic default `max_length` (=20) to control thegeneration length. We recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Translated Text: Bonjour, comment êtes-vous?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5. Visualizing the Results\n",
        "\n",
        "Visualizing attention weights can provide insights into what the model \"focuses\" on when translating.\n"
      ],
      "metadata": {
        "id": "cPLG90gwtAB8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "def visualize_attention(text, model, tokenizer):\n",
        "    # Encode the text with the prefix\n",
        "    input_ids = tokenizer.encode(\"translate English to French: \" + text, return_tensors=\"pt\")\n",
        "\n",
        "    # Generate the translation to get the decoder_input_ids\n",
        "    with torch.no_grad():\n",
        "        decoder_input_ids = model.generate(input_ids)\n",
        "\n",
        "    # Get the model's outputs with attentions\n",
        "    outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, output_attentions=True)\n",
        "\n",
        "    # Extract the attention weights from the outputs\n",
        "    attentions = outputs[-1]  # attentions are the last item in the outputs tuple\n",
        "\n",
        "    # Visualize the first layer's attention weights for the first head\n",
        "    attention = attentions[0][0][0].squeeze().detach().numpy()\n",
        "\n",
        "    plt.figure(figsize=(10, 5))\n",
        "    sns.heatmap(attention, cmap=\"YlGnBu\")\n",
        "    plt.xlabel(\"Attention Heads\")\n",
        "    plt.ylabel(\"Tokens\")\n",
        "    plt.title(\"Attention Weights\")\n",
        "    plt.show()\n",
        "\n",
        "visualize_attention(text, model, tokenizer)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "1AF3G-Vls99e",
        "outputId": "b42e206c-43af-4258-bcb4-859721e32aa4"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1260: UserWarning: Using the model-agnostic default `max_length` (=20) to control thegeneration length. We recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x500 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwUAAAHWCAYAAAAmUCXRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABNRklEQVR4nO3dd3wUdf7H8fckJJuQQEBKEiJFQKrUIBxNQCKoHIIVEaUplkOlKAoWAqIEUDi8E0VQQD0QLKiIHkUOUCFKtwAGkHpAAogQCLCB7Pz+8MfergkhCUMm2Xk9f495/G6/MzvznkTNfvZbxjBN0xQAAAAAxwqyOwAAAAAAe1EUAAAAAA5HUQAAAAA4HEUBAAAA4HAUBQAAAIDDURQAAAAADkdRAAAAADgcRQEAAADgcBQFAAAAgMNRFACAxQzD0KhRo+yOcdm0b99e7du3L/B7r7nmGmsDAQAuGUUBgCLl9ddfl2EYatGiRY77t2zZolGjRmn37t05vnfWrFmXN+D/+/LLL4vUB/8JEybIMAxt3LjRr900TZUtW1aGYWjXrl1++86cOSOXy6V77rmnMKPmyYEDBzRq1Cht2rTJ7igA4AgUBQCKlNmzZ6tatWpas2aNduzYkW3/li1bNHr06CJRFIwePTrHfadPn9Zzzz1XKDnOa9OmjSTp22+/9WvfvHmzjh07phIlSmjVqlV++9auXavMzEzve/NqyZIlWrJkyaUFvogDBw5o9OjRFAUAUEgoCgAUGbt27dLq1as1adIkVahQQbNnz7Y7UoGEhYWpRIkShXrNZs2aKSwsLFtRsGrVKpUrV04dO3bMtu/86/wWBaGhoQoNDb20wACAIoWiAECRMXv2bJUtW1ZdunTRHXfcka0omDVrlu68805JUocOHWQYhgzD0IoVK1StWjVt3rxZK1eu9Lb7jns/duyYBg8erMqVK8vlcqlmzZoaP368PB6P95jdu3fLMAy98sormjZtmmrUqCGXy6Vrr71Wa9eu9R7Xt29fTZkyRZK81zIMw7s/pzkFGzdu1E033aTSpUsrMjJSHTt21HfffZft/gzD0KpVqzR06FBVqFBBERERuvXWW3X48OFcf3ahoaG69tprs/UGrFq1Si1btlTr1q1z3FemTBnvGH+Px6PJkyerfv36CgsLU3R0tB566CH9/vvvfu/LaU7Bnj17dMsttygiIkIVK1bUkCFDtHjxYu/v58+2bNmiDh06qGTJkoqLi9OECRO8+1asWKFrr71WktSvXz/vz/d8L9D27dt1++23KyYmRmFhYbryyit199136/jx47n+jAAAF1a4X2UBQC5mz56t2267TaGhoerZs6feeOMNrV271vsB8brrrtPjjz+uf/zjH3rmmWdUt25dSVLdunU1efJkPfbYY4qMjNSzzz4rSYqOjpYknTp1Su3atdP+/fv10EMPqUqVKlq9erVGjBihgwcPavLkyX455syZoxMnTuihhx6SYRiaMGGCbrvtNu3cuVMhISF66KGHdODAAS1dulTvvffeRe9r8+bNatu2rUqXLq2nnnpKISEhevPNN9W+fXutXLky2/yJxx57TGXLllViYqJ2796tyZMn69FHH9W8efNyvU6bNm30zTffaPfu3apWrZqkPz74P/DAA2revLkSExN17NgxlSlTRqZpavXq1WrZsqWCgv74fuihhx7SrFmz1K9fPz3++OPatWuXXnvtNW3cuFGrVq1SSEhIjtfNyMjQ9ddfr4MHD2rQoEGKiYnRnDlztHz58hyP//3333XjjTfqtttu01133aWPPvpITz/9tBo0aKCbbrpJdevW1QsvvKCRI0fqwQcfVNu2bSVJrVq1UmZmpjp37iy3263HHntMMTEx2r9/vxYuXKhjx44pKirqor8PAEAOTAAoAtatW2dKMpcuXWqapml6PB7zyiuvNAcNGuR33IcffmhKMpcvX57tHPXr1zfbtWuXrX3MmDFmRESEuW3bNr/24cOHm8HBwebevXtN0zTNXbt2mZLMcuXKmUePHvUe99lnn5mSzM8//9zbNnDgQPNC/wmVZCYmJnpfd+/e3QwNDTV//fVXb9uBAwfMUqVKmdddd523bebMmaYkMyEhwfR4PN72IUOGmMHBweaxY8dyvN55X3zxhSnJfO+990zTNM2DBw+aksyVK1eaJ06cMIODg80vvvjCNE3T/Pnnn01J5ksvvWSapml+8803piRz9uzZfudctGhRtvZ27dr5/ZwnTpxoSjI//fRTb9vp06fNOnXqZPtdtWvXzpRkvvvuu942t9ttxsTEmLfffru3be3ataYkc+bMmX55Nm7caEoyP/zww1x/FgCA/GH4EIAiYfbs2YqOjlaHDh0k/TEEp0ePHpo7d66ysrIu6dwffvih2rZtq7Jly+rIkSPeLSEhQVlZWfr666/9ju/Ro4fKli3rfX3+m+qdO3fm+9pZWVlasmSJunfvrurVq3vbY2Njdc899+jbb79Venq633sefPBBv+FIbdu2VVZWlvbs2ZPrtVq1aqWgoCDvXIHz3+5fe+21ioyMVMOGDb1DiM7///PzCT788ENFRUXphhtu8PsZxcfHKzIy8oLf+kvSokWLFBcXp1tuucXbFhYWpgEDBuR4fGRkpO69917v69DQUDVv3jxPP9/zPQGLFy/WqVOnLno8ACBvKAoA2C4rK0tz585Vhw4dtGvXLu3YsUM7duxQixYtlJaWpmXLll3S+bdv365FixapQoUKfltCQoIk6dChQ37HV6lSxe/1+QLhz2Pr8+Lw4cM6deqUateunW1f3bp15fF4tG/fPkuuX6ZMGdWvX9/vg3+TJk0UHh4u6Y+iwXff+Q/j0h8/o+PHj6tixYrZfk4nT57M9jPytWfPHtWoUcOvkJGkmjVr5nj8lVdeme3YsmXL5unne9VVV2no0KF66623VL58eXXu3FlTpkxhPgEAXCLmFACw3X/+8x8dPHhQc+fO1dy5c7Ptnz17tjp16lTg83s8Ht1www166qmnctxfq1Ytv9fBwcE5HmeaZoEz5MelXL9NmzaaOnWqjh07plWrVqlVq1befa1atdKMGTN09uxZffvtt4qPj1dYWJikP35GFStWvOCKTxUqVCjAneTsUn++EydOVN++ffXZZ59pyZIlevzxx5WUlKTvvvtOV155pWU5AcBJKAoA2G727NmqWLGid0UfX/Pnz9cnn3yiqVOnKjw8PNs3zL4utK9GjRo6efKkt2fACrnl8FWhQgWVLFlSKSkp2fb98ssvCgoKUuXKlS3L1aZNG73xxhv66quvtHHjRg0bNsy7r1WrVjp9+rS++OIL7dy5U7fffrt3X40aNfTVV1+pdevW3p6FvKpataq2bNki0zT9fi45PWciry72823QoIEaNGig5557TqtXr1br1q01depUvfjiiwW+JgA4GcOHANjq9OnTmj9/vv7617/qjjvuyLY9+uijOnHihBYsWCBJioiIkPTHEqN/FhERkWP7XXfdpeTkZC1evDjbvmPHjuncuXP5zp1bDl/BwcHq1KmTPvvsM78HrqWlpWnOnDlq06aNSpcune/rX8j5OQKTJk3S2bNn/XoKqlWrptjYWO/yn77PJ7jrrruUlZWlMWPGZDvnuXPncr3Pzp07a//+/d7fkfTH05KnT59e4Pu40M83PT092++rQYMGCgoKktvtLvD1AMDp6CkAYKsFCxboxIkTfpNUff3lL3/xPsisR48eaty4sYKDgzV+/HgdP35cLpdL119/vSpWrKj4+Hi98cYbevHFF1WzZk1VrFhR119/vYYNG6YFCxbor3/9q/r27av4+HhlZGTop59+0kcffaTdu3erfPny+codHx8vSXr88cfVuXNnBQcH6+67787x2BdffFFLly5VmzZt9Le//U0lSpTQm2++Kbfb7bc+vxWqVKmiypUrKzk5WdWqVVOlSpX89rdq1Uoff/yxDMNQ69atve3t2rXTQw89pKSkJG3atEmdOnVSSEiItm/frg8//FCvvvqq7rjjjhyv+dBDD+m1115Tz549NWjQIMXGxmr27NneoUl57VXxVaNGDZUpU0ZTp05VqVKlFBERoRYtWuiHH37Qo48+qjvvvFO1atXSuXPn9N577yk4ONiv5wMAkE/2Ln4EwOm6du1qhoWFmRkZGRc8pm/fvmZISIh55MgR0zRNc/r06Wb16tXN4OBgvyUvU1NTzS5dupilSpUyJfktm3nixAlzxIgRZs2aNc3Q0FCzfPnyZqtWrcxXXnnFzMzMNE3zf0uSvvzyy9ky6E/LjJ47d8587LHHzAoVKpiGYfgtT/rnY03TNDds2GB27tzZjIyMNEuWLGl26NDBXL16td8x55ckXbt2rV/78uXLL7gMa0569uxpSjLvueeebPsmTZpkSjLr1q2b43unTZtmxsfHm+Hh4WapUqXMBg0amE899ZR54MAB7zF/XpLUNE1z586dZpcuXczw8HCzQoUK5hNPPGF+/PHHpiTzu+++83tv/fr1s123T58+ZtWqVf3aPvvsM7NevXpmiRIlvMuT7ty50+zfv79Zo0YNMywszLziiivMDh06mF999VWefjYAgJwZpllIM+cAAI4yefJkDRkyRP/9738VFxdndxwAQC4oCgAAl+z06dN+E5TPnDmjJk2aKCsrS9u2bbMxGQAgL5hTAAC4ZLfddpuqVKmixo0b6/jx4/rXv/6lX3755YJLnAIAihaKAgDAJevcubPeeustzZ49W1lZWapXr57mzp2rHj162B0NAJAHDB8CAAAAHI7nFAAAAAAOR1EAAAAAOBxFAQAAAOBwATnRONOzzu4IlgoJKmV3BEsZyv/TTYsqU4E1JSeQfjcAEIjK137d7giWOZIy2e4IFxRepadl5zq9933LznU5BWRRAAAAABSUYThvMI3z7hgAAACAH3oKAAAAAB+GA783pygAAAAAfDB8CAAAAIDj0FMAAAAA+HBiTwFFAQAAAODDMJy3RLfzyiAAAAAAfugpAAAAAPw473tzigIAAADAhxPnFDjvjgEAAAD4oacAAAAA8OHEngKKAgAAAMAHTzQuZEeOHNGMGTOUnJys1NRUSVJMTIxatWqlvn37qkKFCnbGAwAAABzBtqJg7dq16ty5s0qWLKmEhATVqlVLkpSWlqZ//OMfGjdunBYvXqxmzZrleh632y232+3XZoRkyuUKvWzZAQAAELgYPlSIHnvsMd15552aOnVqtgdEmKaphx9+WI899piSk5NzPU9SUpJGjx7t1/bcyAF6PvFByzMDAAAg8DmxKDBM0zTtuHB4eLg2btyoOnXq5Lj/l19+UZMmTXT69Olcz5NzT8HPAdVTEBJUyu4IljIUOE8JNGXLvz6XTSD9bgAgEJWv/brdESxzJGWy3REuqFytxy0712/b/mHZuS4n23oKYmJitGbNmgsWBWvWrFF0dPRFz+NyueRyufzaMj2BUxAAAACgcDmxp8C2ouDJJ5/Ugw8+qPXr16tjx47eAiAtLU3Lli3T9OnT9corr9gVDwAAAA7lxJ5z24qCgQMHqnz58vr73/+u119/XVlZWZKk4OBgxcfHa9asWbrrrrvsigcAAAA4hq1Lkvbo0UM9evTQ2bNndeTIEUlS+fLlFRISYmcsAAAAOBjDh2wSEhKi2NhYu2MAAAAAjiwKnHfHAAAAAPwUiZ4CAAAAoKhwYk8BRQEAAADgx3lFgfPuGAAAAIAfegoAAAAAHwwfAgAAABzOiUWB8+4YAAAAgB96CgAAAAAfhgO/N6coAAAAAHw4cfhQQBYFUdUm2h3BUhl7nrc7gqUMI3D+sTNk2B0BAOAgR1L+ZncEBKjA+XQGAAAAWMAwnPelH0UBAAAA4MOJw4ecd8cAAAAA/NBTAAAAAPhg9SEAAADA4Rg+BAAAAMBx6CkAAAAAfDixp4CiAAAAAPDhxDkFzrtjAAAAAH7oKQAAAAB8MXwIAAAAcDYnzilw3h0DAAAA8ENPAQAAAODDMAy7IxQ6igIAAADAB6sPFTH79u1T//79cz3G7XYrPT3dbzPNrEJKCAAAABR/RbooOHr0qN55551cj0lKSlJUVJTfdi59SyElBAAAQKAxjCDLtuLC1uFDCxYsyHX/zp07L3qOESNGaOjQoX5tFes/cEm5AAAA4GDMKShc3bt3l2EYMk3zgsdcbKKHy+WSy+X603uCLckHAAAAOIGtfRqxsbGaP3++PB5PjtuGDRvsjAcAAAAnCrJwKyZsjRofH6/169dfcP/FehEAAAAAyxmGdVsxYevwoWHDhikjI+OC+2vWrKnly5cXYiIAAADAeWwtCtq2bZvr/oiICLVr166Q0gAAAAAqVt/wW4WHlwEAAAC+itFcAKs48JYBAAAA+KKnAAAAAPBhMnwIAAAAcDjn1QQMHwIAAACcjp4CAAAAwFeQ87oKKAoAAAAAXw6cU8DwIQAAAMDh6CkAAAAAfDmvoyAwi4LI8Bi7I1jqTNbvdkewVMkSFeyOAAAAcGEOnFPA8CEAAADA4QKypwAAAAAoMAdONKYoAAAAAHw5ryZg+BAAAABQlEyZMkXVqlVTWFiYWrRooTVr1uR6/OTJk1W7dm2Fh4ercuXKGjJkiM6cOZOva9JTAAAAAPiycaLxvHnzNHToUE2dOlUtWrTQ5MmT1blzZ6WkpKhixYrZjp8zZ46GDx+uGTNmqFWrVtq2bZv69u0rwzA0adKkPF+XngIAAADAl2Hhlk+TJk3SgAED1K9fP9WrV09Tp05VyZIlNWPGjByPX716tVq3bq177rlH1apVU6dOndSzZ8+L9i78GUUBAAAAcJm43W6lp6f7bW63O8djMzMztX79eiUkJHjbgoKClJCQoOTk5Bzf06pVK61fv95bBOzcuVNffvmlbr755nzlpCgAAAAAfJiGYdmWlJSkqKgovy0pKSnH6x45ckRZWVmKjo72a4+OjlZqamqO77nnnnv0wgsvqE2bNgoJCVGNGjXUvn17PfPMM/m6Z4oCAAAAwFeQYdk2YsQIHT9+3G8bMWKEZVFXrFihsWPH6vXXX9eGDRs0f/58ffHFFxozZky+zsNEYwAAAOAycblccrlceTq2fPnyCg4OVlpaml97WlqaYmJicnzP888/r/vuu08PPPCAJKlBgwbKyMjQgw8+qGeffVZBQXnrA6CnAAAAAPBl00Tj0NBQxcfHa9myZd42j8ejZcuWqWXLljm+59SpU9k++AcHB0uSTNPM87XpKQAAAAB82fhE46FDh6pPnz5q1qyZmjdvrsmTJysjI0P9+vWTJPXu3VtxcXHeeQldu3bVpEmT1KRJE7Vo0UI7duzQ888/r65du3qLg7ygKAAAAACKiB49eujw4cMaOXKkUlNT1bhxYy1atMg7+Xjv3r1+PQPPPfecDMPQc889p/3796tChQrq2rWrXnrppXxd1zDz069QTFSoPcTuCJbas/kOuyNYqmSJCnZHAAAAtqtld4ALqtn9XcvOtePT3pad63KyfU7B6dOn9e2332rLli3Z9p05c0bvvpv7LyWntV9Nz7nLFRcAAACBzsaHl9nF1qJg27Ztqlu3rq677jo1aNBA7dq108GDB737jx8/7h0/dSE5rf166ujayx0dAAAACBi2FgVPP/20rrnmGh06dEgpKSkqVaqUWrdurb179+b5HDmt/VryimsvY2oAAAAENMOwbismbJ1ovHr1an311VcqX768ypcvr88//1x/+9vf1LZtWy1fvlwREREXPUdOa78aQcyfBgAAQAEVow/zVrG1p+D06dMqUeJ/H+ANw9Abb7yhrl27ql27dtq2bZuN6QAAAABnsPUr9Tp16mjdunWqW7euX/trr70mSbrlllvsiAUAAAAns30pnsJn6y3feuutev/993Pc99prr6lnz575ehIbAAAAcMkcOKeA5xQUAzynAAAABJ4i/JyCu2Zbdq4dH/Sy7FyXEzNyAQAAAF/F5wt+y1AUAAAAAD7MIOdVBQ6cRgEAAADAFz0FAAAAgK9iNEHYKhQFAAAAgC/n1QQMHwIAAACcjp4CAAAAwJcDJxpTFAAAAAC+HDingOFDAAAAgMMFZE9BRHi03REsFRoUaXcES508+1+7I1gmMuRKuyMAAACrOa+jIDCLAgAAAKDAHDingOFDAAAAgMPRUwAAAAD4cmBPAUUBAAAA4MN0Xk3A8CEAAADA6egpAAAAAHwxfAgAAABwOB5eBgAAAMBp6CkAAAAAfDF8CAAAAHA4B46lceAtAwAAAPBFTwEAAADgi4nGAAAAAJyGngIAAADAFxONC9/WrVv13XffqWXLlqpTp45++eUXvfrqq3K73br33nt1/fXX5/p+t9stt9vt12Z6zskIsv3WAAAAUAyZDB8qXIsWLVLjxo315JNPqkmTJlq0aJGuu+467dixQ3v27FGnTp30n//8J9dzJCUlKSoqym87fmhF4dwAAAAAEABsLQpeeOEFDRs2TL/99ptmzpype+65RwMGDNDSpUu1bNkyDRs2TOPGjcv1HCNGjNDx48f9tqiK7QvnBgAAABB4gizciglbo27evFl9+/aVJN111106ceKE7rjjDu/+Xr166ccff8z1HC6XS6VLl/bbGDoEAACAAgsyrNuKCdvrF+P/x2wFBQUpLCxMUVFR3n2lSpXS8ePH7YoGAAAAOIKtRUG1atW0fft27+vk5GRVqVLF+3rv3r2KjY21IxoAAACcyjCs24oJW8fZPPLII8rKyvK+vuaaa/z2//vf/77o6kMAAACApYrRsB+r2FoUPPzww7nuHzt2bCElAQAAAJyLGbkAAACAL+d1FFAUAAAAAL5MBw4fsn31IQAAAAD2oqcAAAAA8OXAngKKAgAAAMBXMVpK1CoMHwIAAAAcjp4CAAAAwJcDvzanKAAAAAB8MXwIAAAAgNMEZE/Brk232h3BUm0/O2x3BEutvKWi3REsc9fyVLsjWOqDDjF2RwAAwH6sPgQAAAA4nAOLAoYPAQAAAA5HTwEAAADgw3TgRGOKAgAAAMCXA8fSOPCWAQAAAPiipwAAAADwxfAhAAAAwOFYfQgAAACA09BTAAAAAPhyYE8BRQEAAADgy3k1AcOHAAAAAKejpwAAAADwYTJ8yH6macpw4DJQAAAAKCIc+Fm0yA0fcrlc2rp1q90xAAAAAMewradg6NChObZnZWVp3LhxKleunCRp0qRJuZ7H7XbL7Xb7tYW6MuVyhVoTFAAAAM7C8KHCM3nyZDVq1EhlypTxazdNU1u3blVERESehhElJSVp9OjRfm0jEwdq1KjHrIwLAAAAp3BeTWBfUTB27FhNmzZNEydO1PXXX+9tDwkJ0axZs1SvXr08nWfEiBHZeh1CXXsszQoAAAAEMtvmFAwfPlzz5s3TI488oieffFJnz54t0HlcLpdKly7ttzF0CAAAAAUVFGTdVhBTpkxRtWrVFBYWphYtWmjNmjW5Hn/s2DENHDhQsbGxcrlcqlWrlr788sv83XPBolrj2muv1fr163X48GE1a9ZMP//8MysPAQAAwFaGYd2WX/PmzdPQoUOVmJioDRs2qFGjRurcubMOHTqU4/GZmZm64YYbtHv3bn300UdKSUnR9OnTFRcXl6/r2r4kaWRkpN555x3NnTtXCQkJysrKsjsSAAAAYItJkyZpwIAB6tevnyRp6tSp+uKLLzRjxgwNHz482/EzZszQ0aNHtXr1aoWEhEiSqlWrlu/rFpklSe+++26tW7dO8+fPV9WqVe2OAwAAAIeysqfA7XYrPT3db/vzypnnZWZmav369UpISPC2BQUFKSEhQcnJyTm+Z8GCBWrZsqUGDhyo6OhoXXPNNRo7dmy+v2gvMkWBJF155ZXq1q2bIiIi7I4CAAAAhzIMw7ItKSlJUVFRfltSUlKO1z1y5IiysrIUHR3t1x4dHa3U1NQc37Nz50599NFHysrK0pdffqnnn39eEydO1Isvvpive7Z9+BAAAAAQqHJaKdPlcll2fo/Ho4oVK2ratGkKDg5WfHy89u/fr5dfflmJiYl5Pg9FAQAAAODDynVvXC5XnouA8uXLKzg4WGlpaX7taWlpiomJyfE9sbGxCgkJUXBwsLetbt26Sk1NVWZmpkJD87YqZ5EaPgQAAADYza7Vh0JDQxUfH69ly5Z52zwej5YtW6aWLVvm+J7WrVtrx44d8ng83rZt27YpNjY2zwWBRFEAAAAAFBlDhw7V9OnT9c4772jr1q165JFHlJGR4V2NqHfv3hoxYoT3+EceeURHjx7VoEGDtG3bNn3xxRcaO3asBg4cmK/rMnwIAAAA8GHY+LV5jx49dPjwYY0cOVKpqalq3LixFi1a5J18vHfvXgX5PBWtcuXKWrx4sYYMGaKGDRsqLi5OgwYN0tNPP52v6xqmaZqW3kkRYCrF7giWavvZYbsjWGrlLWXsjmCZniuO2h3BUh90yHm8IgAA1qtld4ALqv3W15adK+WB6yw71+XE8CEAAADA4Rg+BAAAAPgIsnD1oeKCogAAAADwYeWSpMVFQBYFT3y/3+4Ilvr6luiLH1SMeMxzdkewzD//cszuCJZ6eFX+Hole1E1tHWd3BAAAioWALAoAAACAgqKnAAAAAHA4w4FVAasPAQAAAA5HTwEAAADgw86Hl9mFogAAAADw4cDRQwwfAgAAAJyOngIAAADAhxN7CigKAAAAAB9OLAoYPgQAAAA4HD0FAAAAgI8gegou7p133tEXX3zhff3UU0+pTJkyatWqlfbs2WNpOAAAAKCwGYZ1W3GR76Jg7NixCg8PlyQlJydrypQpmjBhgsqXL68hQ4ZYHhAAAADA5ZXv4UP79u1TzZo1JUmffvqpbr/9dj344INq3bq12rdvb3U+AAAAoFAVp2/4rZLvnoLIyEj99ttvkqQlS5bohhtukCSFhYXp9OnT1qYDAAAACpkRZFi2FRf57im44YYb9MADD6hJkybatm2bbr75ZknS5s2bVa1atUsKk5GRoQ8++EA7duxQbGysevbsqXLlyuX6HrfbLbfb7dd2LjNTJUJDLykLAAAA4BT57imYMmWKWrZsqcOHD+vjjz/2fmhfv369evbsma9z1atXT0ePHpX0x7Cka665RkOGDNHSpUuVmJioevXqadeuXbmeIykpSVFRUX7b2nfez+9tAQAAAJKcOdHYME3TtOviQUFBSk1NVcWKFXXvvfdq165d+vLLLxUVFaWTJ0/q1ltvVYUKFTRnzpwLniOnnoLnflgdUD0FrzSPtjuCpTzmObsjWOY3d2CtuJW4oZTdESw1tXWc3REAABdUy+4AF9Tio28tO9f3d7Sx7FyXU4GeU3Ds2DGtWbNGhw4dksfj8bYbhqH77ruvQEGSk5M1depURUVFSfpj7sLo0aN199135/o+l8sll8vl1xZIBQEAAABwueW7KPj888/Vq1cvnTx5UqVLl5bh0y9SkKLg/PvPnDmj2NhYv31xcXE6fPhwfiMCAAAABVachv1YJd9zCp544gn1799fJ0+e1LFjx/T77797t/PzA/KjY8eOatq0qdLT05WSkuK3b8+ePRedaAwAAABYKciwbisu8t1TsH//fj3++OMqWbLkJV88MTHR73VkZKTf688//1xt27a95OsAAAAAuLB8FwWdO3fWunXrVL169Uu++J+Lgj97+eWXL/kaAAAAQH44cfhQvouCLl26aNiwYdqyZYsaNGigkJAQv/233HKLZeEAAACAwmbke4B98ZfvomDAgAGSpBdeeCHbPsMwlJWVdempAAAAABSafBcFvkuQAgAAAIGG4UP5dObMGYWFhVmVBQAAALCd4cCqIN8jprKysjRmzBjFxcUpMjJSO3fulCQ9//zzevvtty0PCAAAAODyyndR8NJLL2nWrFmaMGGCQn2eHHzNNdforbfesjQcAAAAUNgMw7qtuMh3UfDuu+9q2rRp6tWrl4KDg73tjRo10i+//GJpOAAAAKCwURTkwf79+1WzZs1s7R6PR2fPnrUkFAAAAIDCk++ioF69evrmm2+ytX/00Udq0qSJJaEAAAAAuzixpyDfqw+NHDlSffr00f79++XxeDR//nylpKTo3Xff1cKFCy9Hxnyb2CLO7gjIhaHAeZZFOVdVuyNY6olr9tkdwVI1eq23O4Jlfp0db3cEAHCMoGL0Yd4q+e4p6Natmz7//HN99dVXioiI0MiRI7V161Z9/vnnuuGGGy5HRgAAAACXUb57Cv773/+qbdu2Wrp0abZ93333nf7yl79YEgwAAACwAz0FedCpUycdPXo0W/uqVat04403WhIKAAAAsEuQYVq2FRf5Lgr+8pe/qFOnTjpx4oS37euvv9bNN9+sxMRES8MBAAAAuPzyXRS89dZbqlKlirp27Sq3263ly5erS5cueuGFFzRkyJDLkREAAAAoNEGGdVtxke+iICgoSHPnzlVISIiuv/563XLLLUpKStKgQYMuRz4AAACgUAVZuBUXeZpo/OOPP2ZrGzVqlHr27Kl7771X1113nfeYhg0bWpsQAAAAwGWVp6KgcePGMgxDpvm/yRLnX7/55puaNm2aTNOUYRjKygqcNegBAADgPMVpgrBV8lQU7Nq163LnAAAAAIqE4jQXwCp5KgqqVg2sp7YCAAAA+J98P7xMkn799VdNnjxZW7dulSTVq1dPgwYNUo0aNSwNBwAAABS24jRB2Cr5vufFixerXr16WrNmjRo2bKiGDRvq+++/V/369XN8yjEAAABQnDhxSdJ89xQMHz5cQ4YM0bhx47K1P/3007rhhhssCwcAAADg8st3T8HWrVt1//33Z2vv37+/tmzZYkkoAAAAwC6GYVq2FRf5LgoqVKigTZs2ZWvftGmTKlasmK9zbdiwwW9lo/fee0+tW7dW5cqV1aZNG82dO/ei53C73UpPT/fb3O7MfOUAAAAAznPi8KE8FwUvvPCCTp06pQEDBujBBx/U+PHj9c033+ibb77RuHHj9NBDD2nAgAH5uni/fv3066+/SpLeeustPfTQQ2rWrJmeffZZXXvttRowYIBmzJiR6zmSkpIUFRXltyUlvZmvHAAAAICTGabvE8lyERwcrIMHD6pChQqaPHmyJk6cqAMHDkiSKlWqpGHDhunxxx+XYeS9JCpZsqS2bt2qqlWrqmnTpnrkkUf8Cos5c+bopZde0ubNmy94DrfbLbfb7dcW6tojlys0zzlQuDym++IHFRN5/Nen2Nh1Yp/dESx1499O2B3BMr/Ojrc7AgBYrJbdAS7o7uVfW3auuR2us+xcl1OeJxqf//BjGIaGDBmiIUOG6MSJP/7glipVqkAXL1mypI4cOaKqVatq//79at68ud/+Fi1aXPTBaS6XSy6Xyz+rKAgAAABQME58onG+5hT8uRegVKlSBS4IJOmmm27SG2+8IUlq166dPvroI7/9H3zwgWrWrFng8wMAAAC4uHwtSVqrVq2LDg86evRons83fvx4tW7dWu3atVOzZs00ceJErVixQnXr1lVKSoq+++47ffLJJ/mJCAAAAFyS4jRB2Cr5KgpGjx6tqKgoyy5eqVIlbdy4UePGjdPnn38u0zS1Zs0a7du3T61bt9aqVavUrFkzy64HAAAAXIwTn2icr6Lg7rvvzveyoxdTpkwZjRs3LtvD0AAAAAAUjjwXBflZVQgAAAAorhg+lItAW3oRAAAAyIkTVx/Kc1Hg8XguZw4AAAAANsnXnAIAAAAg0DF8CAAAAHA4J64+5MR7BgAAAOCDngIAAADABxONAQAAAIdz4pwChg8BAAAADheQPQWmec7uCJYyjMD6NQUZLrsjWOacedruCJaKi4iyO4Kl3p8cOF/1VH3lv3ZHsNSeJ6+0OwIAXJDdPQVTpkzRyy+/rNTUVDVq1Ej//Oc/1bx584u+b+7cuerZs6e6deumTz/9NF/XpKcAAAAA8BFk4ZZf8+bN09ChQ5WYmKgNGzaoUaNG6ty5sw4dOpTr+3bv3q0nn3xSbdu2LcBVKQoAAACAImPSpEkaMGCA+vXrp3r16mnq1KkqWbKkZsyYccH3ZGVlqVevXho9erSqV69eoOtSFAAAAAA+ggzTss3tdis9Pd1vc7vdOV43MzNT69evV0JCwv+yBAUpISFBycnJF8z7wgsvqGLFirr//vsLfs8FficAAAAQgIIM67akpCRFRUX5bUlJSTle98iRI8rKylJ0dLRfe3R0tFJTU3N8z7fffqu3335b06dPv6R7DqwZrAAAAEARMmLECA0dOtSvzeWyZtGVEydO6L777tP06dNVvnz5SzoXRQEAAADgw8qhNC6XK89FQPny5RUcHKy0tDS/9rS0NMXExGQ7/tdff9Xu3bvVtWtXb5vH45EklShRQikpKapRo0aers3wIQAAAMCHlcOH8iM0NFTx8fFatmyZt83j8WjZsmVq2bJltuPr1Kmjn376SZs2bfJut9xyizp06KBNmzapcuXKeb42PQUAAABAETF06FD16dNHzZo1U/PmzTV58mRlZGSoX79+kqTevXsrLi5OSUlJCgsL0zXXXOP3/jJlykhStvaLoSgAAAAAfBiGadu1e/ToocOHD2vkyJFKTU1V48aNtWjRIu/k47179yooyPrBPoZpmvbd9WXiMTfbHcFSgfZE40ByzhNYTzQ+6zlhdwRL/fx7ut0RLHPnO+F2R7AUTzQGINWyO8AFjVi37OIH5VFSs46WnetyYk4BAAAA4HB8BQ0AAAD4cOK35hQFAAAAgI8gG+cU2MWJhRAAAAAAH7YWBY899pi++eabSzqH2+1Wenq63+Z2Z1qUEAAAAE5j13MK7GRrUTBlyhS1b99etWrV0vjx45WamprvcyQlJSkqKspvG5c0/TKkBQAAgBNQFNhgyZIluvnmm/XKK6+oSpUq6tatmxYuXOh9RPPFjBgxQsePH/fbho8YcJlTAwAAAIHD9qKgQYMGmjx5sg4cOKB//etfcrvd6t69uypXrqxnn31WO3bsyPX9LpdLpUuX9ttcrtBCSg8AAIBAE2zhVlzYXhScFxISorvuukuLFi3Szp07NWDAAM2ePVu1a9e2OxoAAAAcJMgwLduKiyJTFPiqUqWKRo0apV27dmnRokV2xwEAAAACmq3PKahataqCgy/csWIYhm644YZCTAQAAACnK04ThK1ia1Gwa9cuOy8PAAAAZOPEoqBIDh8CAAAAUHhs7SkAAAAAippgB/YUUBQAAAAAPhg+BAAAAMBx6CkAAAAAfBSn5wtYhaIAAAAA8MHwIQAAAACOQ08BAAAA4OPCj9YNXBQFAAAAgA8nDh8KyKLAMALrtgw58J/MYiLYCLU7gqU8RpjdESzVpFyk3REs89Og3+yOYKmaXVfbHcFSOz5vZXcEALgkgfXpGQAAALhErD4EAAAAOJwTn2jM6kMAAACAw9FTAAAAAPhgojEAAADgcE4sChg+BAAAADgcPQUAAACADyf2FFAUAAAAAD6CHbgkKcOHAAAAAIejpwAAAADw4cRvzSkKAAAAAB9OnFPgxEIIAAAAgA96CgAAAAAfTuwpoCgAAAAAfLD6kA1ee+019e7dW3PnzpUkvffee6pXr57q1KmjZ555RufOncv1/W63W+np6X6b251ZGNEBAACAgGBrUfDiiy/qmWee0alTpzRkyBCNHz9eQ4YMUa9evdSnTx+99dZbGjNmTK7nSEpKUlRUlN+WlPRmId0BAAAAAk2QYd1WXBimadrWP1KzZk1NmDBBt912m3744QfFx8frnXfeUa9evSRJn3zyiZ566ilt3779gudwu91yu91+baGuPXK5Qi9r9sJkqBj9E+UwHjP3nqzi5qwnw+4IlgoyAmeE5Oms3+yOYKmmt/3X7giW2vF5K7sjAMVQLbsDXNDne/9t2bm6VrnJsnNdTrb+xTxw4ICaNWsmSWrUqJGCgoLUuHFj7/6mTZvqwIEDuZ7D5XLJ5XL5tZkKnIIAAAAAuNxsHT4UExOjLVu2SJK2b9+urKws72tJ2rx5sypWrGhXPAAAADiQE4cP2dpT0KtXL/Xu3VvdunXTsmXL9NRTT+nJJ5/Ub7/9JsMw9NJLL+mOO+6wMyIAAAAcJrgYfZi3iq1FwejRoxUeHq7k5GQNGDBAw4cPV6NGjfTUU0/p1KlT6tq160UnGgMAAAC4NLZONL5cTKXYHcFSTDQuuphoXLQx0bjoYqIxgKI80XjJ/i8tO1enuJstO9flFDh/MQEAAAAL2P4gLxs48Z4BAAAA+KCnAAAAAPBRnFYNsgpFAQAAAODDiasPMXwIAAAAcDh6CgAAAAAfQUbALc55URQFAAAAgA8nzilg+BAAAADgcPQUAAAAAD6c2FMQoEWBx+4AljIDrkMncH4/pgLricaGEVj/rGWZZ+yOYKHAGt/69bxydkewVPXXD9odwTI7/xZrdwTAdoH11zBvnHjPAAAAAHwEaE8BAAAAUDAGw4cAAAAAZ3NgTcDwIQAAAMDp6CkAAAAAfDB8CAAAAHA4Jw6lceI9AwAAAPBBTwEAAADgwzAC69kweUFRAAAAAPhw4JQChg8BAAAATkdPAQAAAODDiasP0VMAAAAAOBw9BQAAAIAPB3YUUBQAAAAAvoIcWBXYOnzo4MGDGjlypK6//nrVrVtX9evXV9euXfX2228rKyvLzmgAAACALaZMmaJq1aopLCxMLVq00Jo1ay547PTp09W2bVuVLVtWZcuWVUJCQq7HX4htRcG6detUt25dffnllzp79qy2b9+u+Ph4RURE6Mknn9R1112nEydOXPQ8brdb6enpfpvbnVkIdwAAAIBAZFi45de8efM0dOhQJSYmasOGDWrUqJE6d+6sQ4cO5Xj8ihUr1LNnTy1fvlzJycmqXLmyOnXqpP379+frurYVBYMHD9aQIUO0bt06ffPNN5o1a5a2bdumuXPnaufOnTp16pSee+65i54nKSlJUVFRfltS0rRCuAMAAAAEIsOwbsuvSZMmacCAAerXr5/q1aunqVOnqmTJkpoxY0aOx8+ePVt/+9vf1LhxY9WpU0dvvfWWPB6Pli1blq/r2lYUbNiwQffdd5/39T333KMNGzYoLS1NZcuW1YQJE/TRRx9d9DwjRozQ8ePH/bYRIx68nNEBAACAPMl5VIs7x2MzMzO1fv16JSQkeNuCgoKUkJCg5OTkPF3v1KlTOnv2rK644op85bStKKhYsaIOHjzofZ2WlqZz586pdOnSkqSrr75aR48eveh5XC6XSpcu7be5XKGXLTcAAAACm5XDh3Ie1ZKU43WPHDmirKwsRUdH+7VHR0crNTU1T9mffvppVapUya+wyAvbVh/q3r27Hn74Yb388styuVwaM2aM2rVrp/DwcElSSkqK4uLi7IoHAAAAh7Jy8aERI0Zo6NChfm0ul8vCK/zPuHHjNHfuXK1YsUJhYWH5eq9tRcGLL76ogwcPqmvXrsrKylLLli31r3/9y7vfMIwLVlEAAABAceByufJcBJQvX17BwcFKS0vza09LS1NMTEyu733llVc0btw4ffXVV2rYsGG+c9pWFERGRmrevHk6c+aMzp07p8jISL/9nTp1sikZAAAAnMyu5xSEhoYqPj5ey5YtU/fu3SXJO2n40UcfveD7JkyYoJdeekmLFy9Ws2bNCnRt2x9elt+uDQAAAOBysvPZZUOHDlWfPn3UrFkzNW/eXJMnT1ZGRob69esnSerdu7fi4uK8I2rGjx+vkSNHas6cOapWrZp37kFkZGS2L91zY3tRAAAAAOAPPXr00OHDhzVy5EilpqaqcePGWrRokXfy8d69exUU9L+1gt544w1lZmbqjjvu8DtPYmKiRo0alefrGqZpmpbcQRFiaqvdESxm64OnLwOP3QEs4zHP2h3BUlkBdj8eM3AeZJjpOWl3BEudPHvG7giWajOrtN0RLLPzb7F2R4Bj1LI7wAXtSP/csnPVLN3VsnNdTvQUAAAAAD7sHD5kl0D7ChoAAABAPtFTAAAAAPgwHNhVQFEAAAAA+HDiUBon3jMAAAAAH/QUAAAAAD4YPgQAAAA4nANrgsAsCgwF2x0BuQqc30+QEVgj8ALt351zCpzHsIQHl7c7gqVOn9tndwRLfdzjhN0RLNNqfmD9d2D1bRXtjgAUCwFZFAAAAAAFxfAhAAAAwOEcWBOw+hAAAADgdPQUAAAAAD6CHNhVQFEAAAAA+HBgTcDwIQAAAMDp6CkAAAAAfBhG4CxpnVcUBQAAAIAPhg8BAAAAcBx6CgAAAAAfPLzMBpmZmfr000+VnJys1NRUSVJMTIxatWqlbt26KTQ01OaEAAAAcBIH1gT2Dh/asWOH6tatqz59+mjjxo3yeDzyeDzauHGjevfurfr162vHjh12RgQAAAACnq09BY888ogaNGigjRs3qnTp0n770tPT1bt3bw0cOFCLFy+2KSEAAACcxomTbm0tClatWqU1a9ZkKwgkqXTp0hozZoxatGhhQzIAAAA4lRPnFNhaCJUpU0a7d+++4P7du3erTJkyuZ7D7XYrPT3db3O7M60NCgAAAAQwW4uCBx54QL1799bf//53/fjjj0pLS1NaWpp+/PFH/f3vf1ffvn314IMP5nqOpKQkRUVF+W1JSW8W0h0AAAAg8BgWbsWDYZqmrY9sGz9+vF599VWlpqbK+P++GtM0FRMTo8GDB+upp57K9f1ut1tut9uvzeXaK5eLVYtw+ZkKrCcemuY5uyNY6px52u4IljEUbHcESx1177M7gqUOnAqcEcgDV5axO4KlVt9W0e4IuKBadge4oN/dCy07V1nXXy071+Vke1Fw3q5du/yWJL3qqqsu4WzbrAkFXARFQdFGUVB0URQUXRQFKDwUBUWJ7c8pOO+qq67KVgjs27dPiYmJmjFjhk2pAAAA4DSGETiFfl4V6Ts+evSo3nnnHbtjAAAAwFGcN6fA1p6CBQsW5Lp/586dhZQEAAAAcC5bi4Lu3bvLMAzlNq3BcOJCsQAAALCNUYy+4beKrcOHYmNjNX/+fHk8nhy3DRs22BkPAAAAjuS84UO2FgXx8fFav379BfdfrBcBAAAAwKWzdfjQsGHDlJGRccH9NWvW1PLlywsxEQAAAJzOiasP2VoUtG3bNtf9ERERateuXSGlAQAAAKTiNOzHKs4rgwAAAAD4KTIPLwMAAACKAieuPkRRAAAAAPhwYlHA8CEAAADA4egpAAAAAPw473tzwwzABwF4zC12R7BUkEHtBgBAQby6ebfdESw1aWGw3REss+fpjnZHuKCMcystO1dEieKxkqbzyiAAAAAAfvgKGgAAAPDjvInGFAUAAACAD1YfAgAAAOA49BQAAAAAfpz3vTlFAQAAAOCD4UMAAAAAHIeeAgAAAMCHYTivp4CiAAAAAPDjvKKA4UMAAACAwxXpoiAtLU0vvPCC3TEAAADgIIaCLNuKiyKdNDU1VaNHj7Y7BgAAABzFsHArHmydU/Djjz/muj8lJaWQkgAAAADOZWtR0LhxYxmGIdM0s+073+7E2d8AAACwjxM/f9paFFxxxRWaMGGCOnbsmOP+zZs3q2vXrrmew+12y+12+7WFhGbK5Qq1LCcAAACcxHlFga1zCuLj43XgwAFVrVo1xy0uLi7HXgRfSUlJioqK8tvGJU0vpDsAAAAAij9bewoefvhhZWRkXHB/lSpVNHPmzFzPMWLECA0dOtSvLSR0pyX5AAAA4DzFadUgq9haFNx666257i9btqz69OmT6zEul0sul8uvzWMydAgAAAAFxfChImXfvn3q37+/3TEAAACAgFaki4KjR4/qnXfesTsGAAAAHMSw8P+KC1uHDy1YsCDX/Tt3MjcAAAAAhYslSQtZ9+7dL/icgvOc+EsBAAAACpOtw4diY2M1f/58eTyeHLcNGzbYGQ8AAACOFGThVjzY/pyC9evXX3D/xXoRAAAAAKsxp6CQDRs2LNfnFNSsWVPLly8vxEQAAACA89haFLRt2zbX/REREWrXrl0hpQEAAAAkJz6nwNaiAAAAAChqnLjQTfGZ/QAAAADgsqCnAAAAAPDjvO/NKQoAAAAAH8Vp1SCrOK8MAgAAAODPRIGcOXPGTExMNM+cOWN3FEsE0v0E0r2YJvdTlAXSvZgm91OUBdK9mCb3U5QF0r0gfwzT5OlgBZGenq6oqCgdP35cpUuXtjvOJQuk+wmke5G4n6IskO5F4n6KskC6F4n7KcoC6V6QPwwfAgAAAByOogAAAABwOIoCAAAAwOEoCgrI5XIpMTFRLpfL7iiWCKT7CaR7kbifoiyQ7kXifoqyQLoXifspygLpXpA/TDQGAAAAHI6eAgAAAMDhKAoAAAAAh6MoAAAAAByOogAAAABwOIqCApoyZYqqVaumsLAwtWjRQmvWrLE7UoF8/fXX6tq1qypVqiTDMPTpp5/aHanAkpKSdO2116pUqVKqWLGiunfvrpSUFLtjFdgbb7yhhg0bqnTp0ipdurRatmypf//733bHssS4ceNkGIYGDx5sd5QCGTVqlAzD8Nvq1Kljd6xLsn//ft17770qV66cwsPD1aBBA61bt87uWAVSrVq1bL8fwzA0cOBAu6PlW1ZWlp5//nldddVVCg8PV40aNTRmzBgV5zVCTpw4ocGDB6tq1aoKDw9Xq1attHbtWrtj5cnF/maapqmRI0cqNjZW4eHhSkhI0Pbt2+0JexEXu5f58+erU6dOKleunAzD0KZNm2zJicJDUVAA8+bN09ChQ5WYmKgNGzaoUaNG6ty5sw4dOmR3tHzLyMhQo0aNNGXKFLujXLKVK1dq4MCB+u6777R06VKdPXtWnTp1UkZGht3RCuTKK6/UuHHjtH79eq1bt07XX3+9unXrps2bN9sd7ZKsXbtWb775pho2bGh3lEtSv359HTx40Lt9++23dkcqsN9//12tW7dWSEiI/v3vf2vLli2aOHGiypYta3e0Alm7dq3f72bp0qWSpDvvvNPmZPk3fvx4vfHGG3rttde0detWjR8/XhMmTNA///lPu6MV2AMPPKClS5fqvffe008//aROnTopISFB+/fvtzvaRV3sb+aECRP0j3/8Q1OnTtX333+viIgIde7cWWfOnCnkpBd3sXvJyMhQmzZtNH78+EJOBtuYyLfmzZubAwcO9L7OysoyK1WqZCYlJdmY6tJJMj/55BO7Y1jm0KFDpiRz5cqVdkexTNmyZc233nrL7hgFduLECfPqq682ly5darZr184cNGiQ3ZEKJDEx0WzUqJHdMSzz9NNPm23atLE7xmUzaNAgs0aNGqbH47E7Sr516dLF7N+/v1/bbbfdZvbq1cumRJfm1KlTZnBwsLlw4UK/9qZNm5rPPvusTakK5s9/Mz0ejxkTE2O+/PLL3rZjx46ZLpfLfP/9921ImHe5/f3ftWuXKcncuHFjoWZC4aOnIJ8yMzO1fv16JSQkeNuCgoKUkJCg5ORkG5Phz44fPy5JuuKKK2xOcumysrI0d+5cZWRkqGXLlnbHKbCBAweqS5cufv/+FFfbt29XpUqVVL16dfXq1Ut79+61O1KBLViwQM2aNdOdd96pihUrqkmTJpo+fbrdsSyRmZmpf/3rX+rfv78Mw7A7Tr61atVKy5Yt07Zt2yRJP/zwg7799lvddNNNNicrmHPnzikrK0thYWF+7eHh4cW6t02Sdu3apdTUVL//vkVFRalFixZ8PkCxUMLuAMXNkSNHlJWVpejoaL/26Oho/fLLLzalwp95PB4NHjxYrVu31jXXXGN3nAL76aef1LJlS505c0aRkZH65JNPVK9ePbtjFcjcuXO1YcOGYjN2ODctWrTQrFmzVLt2bR08eFCjR49W27Zt9fPPP6tUqVJ2x8u3nTt36o033tDQoUP1zDPPaO3atXr88ccVGhqqPn362B3vknz66ac6duyY+vbta3eUAhk+fLjS09NVp04dBQcHKysrSy+99JJ69epld7QCKVWqlFq2bKkxY8aobt26io6O1vvvv6/k5GTVrFnT7niXJDU1VZJy/Hxwfh9QlFEUICANHDhQP//8c7H/5ql27dratGmTjh8/ro8++kh9+vTRypUri11hsG/fPg0aNEhLly7N9g1hceT7LW3Dhg3VokULVa1aVR988IHuv/9+G5MVjMfjUbNmzTR27FhJUpMmTfTzzz9r6tSpxb4oePvtt3XTTTepUqVKdkcpkA8++ECzZ8/WnDlzVL9+fW3atEmDBw9WpUqViu3v5r333lP//v0VFxen4OBgNW3aVD179tT69evtjgY4GsOH8ql8+fIKDg5WWlqaX3taWppiYmJsSgVfjz76qBYuXKjly5fryiuvtDvOJQkNDVXNmjUVHx+vpKQkNWrUSK+++qrdsfJt/fr1OnTokJo2baoSJUqoRIkSWrlypf7xj3+oRIkSysrKsjviJSlTpoxq1aqlHTt22B2lQGJjY7MVmnXr1i3WQ6Ikac+ePfrqq6/0wAMP2B2lwIYNG6bhw4fr7rvvVoMGDXTfffdpyJAhSkpKsjtagdWoUUMrV67UyZMntW/fPq1Zs0Znz55V9erV7Y52Sc5/BuDzAYorioJ8Cg0NVXx8vJYtW+Zt83g8WrZsWbEe6x0ITNPUo48+qk8++UT/+c9/dNVVV9kdyXIej0dut9vuGPnWsWNH/fTTT9q0aZN3a9asmXr16qVNmzYpODjY7oiX5OTJk/r1118VGxtrd5QCad26dbble7dt26aqVavalMgaM2fOVMWKFdWlSxe7oxTYqVOnFBTk/6c6ODhYHo/HpkTWiYiIUGxsrH7//XctXrxY3bp1szvSJbnqqqsUExPj9/kgPT1d33//PZ8PUCwwfKgAhg4dqj59+qhZs2Zq3ry5Jk+erIyMDPXr18/uaPl28uRJv283d+3apU2bNumKK65QlSpVbEyWfwMHDtScOXP02WefqVSpUt4xnFFRUQoPD7c5Xf6NGDFCN910k6pUqaITJ05ozpw5WrFihRYvXmx3tHwrVapUtrkdERERKleuXLGc8/Hkk0+qa9euqlq1qg4cOKDExEQFBwerZ8+edkcrkCFDhqhVq1YaO3as7rrrLq1Zs0bTpk3TtGnT7I5WYB6PRzNnzlSfPn1UokTx/VPXtWtXvfTSS6pSpYrq16+vjRs3atKkSerfv7/d0Qps8eLFMk1TtWvX1o4dOzRs2DDVqVOnWPwNvdjfzMGDB+vFF1/U1VdfrauuukrPP/+8KlWqpO7du9sX+gIudi9Hjx7V3r17deDAAUnyfnEQExNDz0egsnv5o+Lqn//8p1mlShUzNDTUbN68ufndd9/ZHalAli9fbkrKtvXp08fuaPmW031IMmfOnGl3tALp37+/WbVqVTM0NNSsUKGC2bFjR3PJkiV2x7JMcV6StEePHmZsbKwZGhpqxsXFmT169DB37Nhhd6xL8vnnn5vXXHON6XK5zDp16pjTpk2zO9IlWbx4sSnJTElJsTvKJUlPTzcHDRpkVqlSxQwLCzOrV69uPvvss6bb7bY7WoHNmzfPrF69uhkaGmrGxMSYAwcONI8dO2Z3rDy52N9Mj8djPv/882Z0dLTpcrnMjh07Ftl/Bi92LzNnzsxxf2Jioq25cfkYplmMH4sIAAAA4JIxpwAAAABwOIoCAAAAwOEoCgAAAACHoygAAAAAHI6iAAAAAHA4igIAAADA4SgKAAAAAIejKAAAAAAcjqIAAGzQvn17DR482O4Yl51T7hMAijuKAgABJzk5WcHBwerSpUu2faNGjVLjxo2ztRuGoU8//dTyLCtWrJBhGDp27Jhf+/z58zVmzBjLr+dr9+7dMgxDmzZtyraPD+sAAF8UBQACzttvv63HHntMX3/9tQ4cOGB3nBxdccUVKlWqlN0xAACQRFEAIMCcPHlS8+bN0yOPPKIuXbpo1qxZ3n2zZs3S6NGj9cMPP8gwDBmGoVmzZqlatWqSpFtvvVWGYXhfS9Jnn32mpk2bKiwsTNWrV9fo0aN17tw5737DMPTWW2/p1ltvVcmSJXX11VdrwYIFkv74pr5Dhw6SpLJly8owDPXt21dS9m/qf//9d/Xu3Vtly5ZVyZIlddNNN2n79u1+2cuUKaPFixerbt26ioyM1I033qiDBw9a8nNzu9168sknFRcXp4iICLVo0UIrVqzw7v/tt9/Us2dPxcXFqWTJkmrQoIHef/99v3NkZGSod+/eioyMVGxsrCZOnJjtOq+//rquvvpqhYWFKTo6WnfccYcl+QEAl4aiAEBA+eCDD1SnTh3Vrl1b9957r2bMmCHTNCVJPXr00BNPPKH69evr4MGDOnjwoHr06KG1a9dKkmbOnKmDBw96X3/zzTfq3bu3Bg0apC1btujNN9/UrFmz9NJLL/ldc/To0brrrrv0448/6uabb1avXr109OhRVa5cWR9//LEkKSUlRQcPHtSrr76aY+6+fftq3bp1WrBggZKTk2Wapm6++WadPXvWe8ypU6f0yiuv6L333tPXX3+tvXv36sknn7Tk5/boo48qOTlZc+fO1Y8//qg777xTN954o7cwOXPmjOLj4/XFF1/o559/1oMPPqj77rtPa9as8Z5j2LBhWrlypT777DMtWbJEK1as0IYNG7z7161bp8cff1wvvPCCUlJStGjRIl133XWW5AcAXCITAAJIq1atzMmTJ5umaZpnz541y5cvby5fvty7PzEx0WzUqFG290kyP/nkE7+2jh07mmPHjvVre++998zY2Fi/9z333HPe1ydPnjQlmf/+979N0zTN5cuXm5LM33//3e887dq1MwcNGmSapmlu27bNlGSuWrXKu//IkSNmeHi4+cEHH5imaZozZ840JZk7duzwHjNlyhQzOjr6gj+LXbt2mZLM8PBwMyIiwm8LCgryXn/Pnj1mcHCwuX///mz3P2LEiAuev0uXLuYTTzxhmqZpnjhxwgwNDfXmNU3T/O2338zw8HDvdT7++GOzdOnSZnp6+gXPCQCwRwkb6xEAsFRKSorWrFmjTz75RJJUokQJ9ejRQ2+//bbat2+f7/P98MMPWrVqlV/PQFZWls6cOaNTp06pZMmSkqSGDRt690dERKh06dI6dOhQnq+zdetWlShRQi1atPC2lStXTrVr19bWrVu9bSVLllSNGjW8r2NjY/N0nXnz5qlu3bp+bb169fL+759++klZWVmqVauW3zFut1vlypWT9Md9jx07Vh988IH279+vzMxMud1u78/g119/VWZmpt89XHHFFapdu7b39Q033KCqVauqevXquvHGG3XjjTd6h10BAOxFUQAgYLz99ts6d+6cKlWq5G0zTVMul0uvvfaaoqKi8nW+kydPavTo0brtttuy7QsLC/P+75CQEL99hmHI4/HkM/3F5XQd8/+HRuWmcuXKqlmzpl9beHi493+fPHlSwcHBWr9+vYKDg/2Oi4yMlCS9/PLLevXVVzV58mQ1aNBAERERGjx4sDIzM/Ocv1SpUtqwYYNWrFihJUuWaOTIkRo1apTWrl2rMmXK5Pk8AADrURQACAjnzp3Tu+++q4kTJ6pTp05++7p37673339fDz/8sEJDQ5WVlZXt/SEhIdnamzZtqpSUlGwfqPMjNDRUknK85nl169bVuXPn9P3336tVq1aS/pjYm5KSonr16hX42nnVpEkTZWVl6dChQ2rbtm2Ox6xatUrdunXTvffeK0nyeDzatm2bN1+NGjUUEhKi77//XlWqVJH0x+Tpbdu2qV27dt7zlChRQgkJCUpISFBiYqLKlCmj//znPzkWXgCAwkNRACAgLFy4UL///rvuv//+bD0Ct99+u95++209/PDDqlatmnbt2qVNmzbpyiuvVKlSpeRyuVStWjUtW7ZMrVu3lsvlUtmyZTVy5Ej99a9/VZUqVXTHHXcoKChIP/zwg37++We9+OKLecpVtWpVGYahhQsX6uabb1Z4eLj32/fzrr76anXr1k0DBgzQm2++qVKlSmn48OGKi4tTt27dLPsZXUitWrXUq1cv9e7dWxMnTlSTJk10+PBhLVu2TA0bNlSXLl109dVX66OPPtLq1atVtmxZTZo0SWlpad6iIDIyUvfff7+GDRumcuXKqWLFinr22WcVFPS/9SwWLlyonTt36rrrrlPZsmX15ZdfyuPx+A0xAgDYg9WHAASEt99+WwkJCTkOEbr99tu1bt06/fjjj7r99tt14403qkOHDqpQoYJ3Wc2JEydq6dKlqly5spo0aSJJ6ty5sxYuXKglS5bo2muv1V/+8hf9/e9/V9WqVfOcKy4uTqNHj9bw4cMVHR2tRx99NMfjZs6cqfj4eP31r39Vy5YtZZqmvvzyy2xDhi6XmTNnqnfv3nriiSdUu3Ztde/eXWvXrvV+6//cc8+padOm6ty5s9q3b6+YmBh1797d7xwvv/yy2rZtq65duyohIUFt2rRRfHy8d3+ZMmU0f/58XX/99apbt66mTp2q999/X/Xr1y+UewQAXJhh5mVAKgAAAICARU8BAAAA4HAUBQAAAIDDURQAAAAADkdRAAAAADgcRQEAAADgcBQFAAAAgMNRFAAAAAAOR1EAAAAAOBxFAQAAAOBwFAUAAACAw1EUAAAAAA73f6NyAp7dxpRyAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6. Conclusion\n",
        "\n",
        "Transformers have revolutionized the way we handle seq2seq tasks. With the power of Huggingface's Transformers library and PyTorch, implementing such tasks has become more accessible than ever. The visualization of attention weights offers a glimpse into the inner workings of these models, helping us understand their decision-making process.\n",
        "\n",
        "> **Note**: This tutorial is a concise introduction to seq2seq tasks with Transformers. In practice, you might want to fine-tune models on specific datasets, handle longer sequences with more advanced techniques, and explore other architectures and models available in the Huggingface model hub.\n"
      ],
      "metadata": {
        "id": "t0yHRJH1tDpY"
      }
    }
  ]
}