6010ML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jLBoiWOVs1qQ"
   },
   "source": [
    "Conquering IMDB Reviews: TextCNN, RNN and Transformers\n",
    "===  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WANG, Yisong 20641044  \n",
    "1st, Jun, 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![1](https://raw.githubusercontent.com/wangys96/6010S-NLP/master/pic/1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content  \n",
    "- [Introduction](#1.-Introduction)\n",
    "- [Data Preparation](#2.-Data-Preparation)\n",
    "- [Models](#3.-Models)\n",
    " - [TextCNN](#3.2-TextCNN)\n",
    " - [TextRNN](#3.3-TextRNN)\n",
    " - [BERT](#3.4-BERT)\n",
    "- [Conclusion](#Conclusion)\n",
    "- [Reference](#References)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project we explored and implemented three popular NLP models:TextCNN, RNN and BERT+RNN. We focused on the IMDB sentiment classification task.  \n",
    "\n",
    "Before the booming age of neuron networks, people use traditional machine learning method to solve the NLP problem. TF-IDF are used to convert the sentences to the sparse frequency vectors, then Naive Bayes and SVM can be applied on the data. The traditional ML method merely reach 80% accuracy on IMDB. Then the Word2Vec algorithm, that converts sentences to dense vectors, got on the stage, which largely improved the quality of feature extracted. With the vectors, a one-layer fully connected NN could achieve 85% accuracy. In recent 2 years the pre-trained attention models like GPT and BERT dominate the NLP field. As an pre-trained encoder, they can be applied on many tasks with a little work of transformation and training, thus they are usually called the transformers. The enormous number of parameters make the their performance exceed any traditional models. With simple RNN claasification layers the transformers have about 92% accuracy on IMDB. Thanks to the work of open-source contributers, we can easily access these models by import PyTorch and Transformers.\n",
    "\n",
    "We first preprocess IMDB data with `torchtext`, then explore the TextCNN model, then RNN model and finally BERT. For CNN and RNN, we use pretrained `GloVe` model to extract the word vectors. The project is finished on Google Colab with a K80 GPU. `transformers` and `torchviz` are required, which can be install by `!pip install`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_h3W58m42JLY"
   },
   "source": [
    "## 2. Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally, we have 4 steps in `torchtext` to prepare data for NN:  \n",
    "1. define the text field\n",
    "2. build and split the dataset\n",
    "3. build the vocabulary and the vectorizer\n",
    "4. build data iterator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yXCZSLU7toOo"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import random\n",
    "import torch\n",
    "from torchtext import data, datasets\n",
    "torch.manual_seed(19260817)\n",
    "torch.cuda.manual_seed(19260817)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ld5yTCpG25Jd"
   },
   "source": [
    "#### Download Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `spacy` with default setting to split the sentence. Words are cut by simply the space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HuRqc5ub1jcJ"
   },
   "outputs": [],
   "source": [
    "TEXT = data.Field(tokenize='spacy')\n",
    "LABEL = data.LabelField(dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We download the IMDB dataset and split the train set into 80% part for training and 20% part for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "id": "t9LPk0x0uhZk",
    "outputId": "58050cb5-f3d0-450e-81c4-15f2172d3173"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading aclImdb_v1.tar.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "aclImdb_v1.tar.gz: 100%|██████████| 84.1M/84.1M [00:07<00:00, 10.9MB/s]\n"
     ]
    }
   ],
   "source": [
    "all_train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)\n",
    "train_data, valid_data = all_train_data.split(split_ratio=0.8,random_state=random.seed(19260817))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "id": "4u3w6uInxxCC",
    "outputId": "9ecfbfc5-9859-4fa1-9c2f-d9f2417b6bfd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Obs: 20000\n",
      "Validation Obs: 5000\n",
      "Testing Obs: 25000\n"
     ]
    }
   ],
   "source": [
    "print(f'Training Obs: {len(train_data)}')\n",
    "print(f'Validation Obs: {len(valid_data)}')\n",
    "print(f'Testing Obs: {len(test_data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hTD4xOTR23A8"
   },
   "source": [
    "#### Build Vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We adopted the trained `GloVe` to vectorize the raw text. It has been trained on a large global word corpus. The theory is that these pre-trained vectors already have words with similar semantic meaning close together in vector space, e.g. \"terrible\", \"awful\", \"dreadful\" are nearby. This gives our embedding layer a good initialization as it does not have to learn these relations from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "id": "sUUq-0IU3BTb",
    "outputId": "1bf70ce9-8b67-4f98-a84d-5c269ad73b67"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".vector_cache/glove.6B.zip: 862MB [06:30, 2.21MB/s]                          \n",
      "100%|█████████▉| 398661/400000 [00:21<00:00, 19296.45it/s]"
     ]
    }
   ],
   "source": [
    "TEXT.build_vocab(train_data, max_size=25000, vectors=\"glove.6B.100d\", unk_init=torch.Tensor.normal_)\n",
    "LABEL.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dWExMRvF7feg"
   },
   "source": [
    "#### Batch Iterator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `BucketIterator`, which put sentences with similar length into a same batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aSHofWRk7jvo"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data),\n",
    "    batch_size = BATCH_SIZE,\n",
    "    device = device\n",
    ")\n",
    "all_train_iterator = data.BucketIterator(\n",
    "    all_train_data,\n",
    "    batch_size = BATCH_SIZE,\n",
    "    device = device\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F-0IdQlbqKj-"
   },
   "source": [
    "## 3. Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter we trained the model and select the best one based on validation performance to run on the testset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i72LysBf8k0u"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchviz import make_dot, make_dot_from_trace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dn1bMSNGqQXb"
   },
   "source": [
    "### 3.1 Utility Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pEYydfOPusdh"
   },
   "outputs": [],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DVPAkQjSrkKu"
   },
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    \"\"\"\n",
    "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
    "    \"\"\"\n",
    "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
    "    correct = (rounded_preds == y).float()\n",
    "    return correct.sum() / len(correct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "om1OmZHqqUst"
   },
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion):\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    total_len = 0\n",
    "    model.train() \n",
    "\n",
    "    for batch in iterator:\n",
    "        optimizer.zero_grad()\n",
    "        predictions = model(batch.text).squeeze(1)\n",
    "        loss = criterion(predictions, batch.label)\n",
    "        acc = binary_accuracy(predictions, batch.label)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        epoch_loss += loss.item() * len(batch.label)\n",
    "        epoch_acc += acc.item() * len(batch.label)\n",
    "        total_len += len(batch.label)\n",
    "    return epoch_loss / total_len, epoch_acc / total_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jjgTOPzVqpj4"
   },
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    total_len = 0\n",
    "    model.eval() \n",
    "\n",
    "    with torch.no_grad():\n",
    "        for batch in iterator:      \n",
    "            predictions = model(batch.text).squeeze(1)\n",
    "            loss = criterion(predictions, batch.label)\n",
    "            acc = binary_accuracy(predictions, batch.label)\n",
    "\n",
    "            epoch_loss += loss.item() * len(batch.label)\n",
    "            epoch_acc += acc.item() * len(batch.label)\n",
    "            total_len += len(batch.label)\n",
    "    model.train()\n",
    "    \n",
    "    return epoch_loss / total_len, epoch_acc / total_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zz1UTQQ7qwSw"
   },
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):  \n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T8aD1hsV8lPm"
   },
   "source": [
    "### 3.2 TextCNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditionally, CNNs are used to analyse images and are made up of one or more convolutional layers, followed by one or more linear layers. The convolutional layers use filters which scan across an image and produce a processed version of the image. This processed version of the image can be fed into another convolutional layer or a linear layer. Each filter has a shape, e.g. a 3x3 filter covers a 3 pixel wide and 3 pixel high area of the image, and each element of the filter has a weight associated with it, the 3x3 filter would have 9 weights. In traditional image processing these weights were specified by hand by engineers, however the main advantage of the convolutional layers in neural networks is that these weights are learned via backpropagation.\n",
    "\n",
    "The intuitive idea behind learning the weights is that your convolutional layers act like feature extractors, extracting parts of the image that are most important for your CNN's goal, e.g. if using a CNN to detect faces in an image, the CNN may be looking for features such as the existance of a nose, mouth or a pair of eyes in the image.\n",
    "\n",
    "In the same way that a 3x3 filter can look over a patch of an image, a 1x2 filter can look over a 2 sequential words in a piece of text, i.e. a bi-gram. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![TextCNN](https://raw.githubusercontent.com/wangys96/6010S-NLP/master/pic/textcnn.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our model, we will have different sizes of filters, heights of 3, 4 and 5, with 100 of each of them. The intuition is that we will be looking for the occurence of different tri-grams, 4-grams and 5-grams that are relevant for analysing sentiment of movie reviews. Then we use max pooling on the output of the convolutional layers. As our model has 100 filters of 3 different sizes, that means we have 300 different n-grams the model thinks are important. We concatenate these together into a single vector and pass them through a linear layer to predict the sentiment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gxXHgExBvOYA"
   },
   "source": [
    "#### Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IrrMjD2um3bm"
   },
   "outputs": [],
   "source": [
    "class TextCNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes,\n",
    "               output_dim, dropout, pad_idx):\n",
    "        super().__init__()\n",
    "\n",
    "        self.embedding = nn.Embedding(vocab_size,embedding_dim,padding_idx=pad_idx)\n",
    "        self.convs = nn.ModuleList([\n",
    "            nn.Conv2d(in_channels=1,\n",
    "                      out_channels=n_filters,\n",
    "                      kernel_size=(fs,embedding_dim))                  \n",
    "            for fs in filter_sizes\n",
    "        ])    \n",
    "        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "    \n",
    "    def forward(self, text):\n",
    "        text = text.permute(1, 0)        # [batch size, sent len]\n",
    "        embedded = self.embedding(text)  # [batch size, sent len, emb dim]\n",
    "        embedded = embedded.unsqueeze(1) # [batch size, 1, sent len, emb dim]\n",
    "\n",
    "        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]\n",
    "        # conv_n = [batch size, n_filters, sent len - filter_sizes[n]]\n",
    "\n",
    "        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]\n",
    "        # pooled_n = [batch size, n_filters]\n",
    "\n",
    "        cat = self.dropout(torch.cat(pooled, dim=1))\n",
    "        # cat = [batch size, n_filters * len(filter_sizes)]\n",
    "    \n",
    "    return self.fc(cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KP1ul2rkoJIS"
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "N_FILTERS = 100\n",
    "FILTER_SIZES = [3,4,5]\n",
    "OUTPUT_DIM = 1\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "model = TextCNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)\n",
    "model.embedding.weight.data.copy_(TEXT.vocab.vectors)\n",
    "\n",
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "SuRd7xMxvWsm",
    "outputId": "774c2eca-6453-413a-d348-9ee180e54e19"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Parameters:  2620801\n"
     ]
    }
   ],
   "source": [
    "print(\"Network Parameters: \",count_parameters(model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "_U7NFtHdps8I",
    "outputId": "46973e79-f1a2-4331-db99-246b85e38888"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"842pt\" height=\"822pt\"\n",
       " viewBox=\"0.00 0.00 842.00 822.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 818)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-818 838,-818 838,4 -4,4\"/>\n",
       "<!-- 139756444322952 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>139756444322952</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"#000000\" points=\"497.5,-21 393.5,-21 393.5,0 497.5,0 497.5,-21\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-7.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddmmBackward</text>\n",
       "</g>\n",
       "<!-- 139756444321496 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>139756444321496</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"360.5,-91 306.5,-91 306.5,-57 360.5,-57 360.5,-91\"/>\n",
       "<text text-anchor=\"middle\" x=\"333.5\" y=\"-77.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">fc.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"333.5\" y=\"-64.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 139756444321496&#45;&gt;139756444322952 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>139756444321496&#45;&gt;139756444322952</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M360.6121,-58.6284C377.8733,-48.8419 400.177,-36.1965 417.5905,-26.3237\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"419.7256,-29.1366 426.6985,-21.1598 416.2731,-23.0472 419.7256,-29.1366\"/>\n",
       "</g>\n",
       "<!-- 139756444323456 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>139756444323456</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"512.5,-84.5 378.5,-84.5 378.5,-63.5 512.5,-63.5 512.5,-84.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-70.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">FusedDropoutBackward</text>\n",
       "</g>\n",
       "<!-- 139756444323456&#45;&gt;139756444322952 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>139756444323456&#45;&gt;139756444322952</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-63.2281C445.5,-54.5091 445.5,-41.9699 445.5,-31.3068\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-31.1128 445.5,-21.1128 442.0001,-31.1129 449.0001,-31.1128\"/>\n",
       "</g>\n",
       "<!-- 139756443933048 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>139756443933048</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"486.5,-154.5 404.5,-154.5 404.5,-133.5 486.5,-133.5 486.5,-154.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-140.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">CatBackward</text>\n",
       "</g>\n",
       "<!-- 139756443933048&#45;&gt;139756444323456 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>139756443933048&#45;&gt;139756444323456</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-133.3685C445.5,-123.1925 445.5,-107.5606 445.5,-94.8912\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-94.7315 445.5,-84.7315 442.0001,-94.7316 449.0001,-94.7315\"/>\n",
       "</g>\n",
       "<!-- 139756443934504 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>139756443934504</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"355,-218 244,-218 244,-197 355,-197 355,-218\"/>\n",
       "<text text-anchor=\"middle\" x=\"299.5\" y=\"-204.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443934504&#45;&gt;139756443933048 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>139756443934504&#45;&gt;139756443933048</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M323.9371,-196.8715C347.9401,-186.4319 384.763,-170.4164 411.5929,-158.7473\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"413.2089,-161.8612 420.9832,-154.6631 410.417,-155.442 413.2089,-161.8612\"/>\n",
       "</g>\n",
       "<!-- 139756443932712 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>139756443932712</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"321,-275 210,-275 210,-254 321,-254 321,-275\"/>\n",
       "<text text-anchor=\"middle\" x=\"265.5\" y=\"-261.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443932712&#45;&gt;139756443934504 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>139756443932712&#45;&gt;139756443934504</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M271.8951,-253.7787C276.4066,-246.2153 282.5772,-235.8705 287.9403,-226.8794\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"291.0166,-228.5544 293.1335,-218.1732 285.0048,-224.9684 291.0166,-228.5544\"/>\n",
       "</g>\n",
       "<!-- 139756443933104 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>139756443933104</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"333.5,-332 153.5,-332 153.5,-311 333.5,-311 333.5,-332\"/>\n",
       "<text text-anchor=\"middle\" x=\"243.5\" y=\"-318.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MaxPool2DWithIndicesBackward</text>\n",
       "</g>\n",
       "<!-- 139756443933104&#45;&gt;139756443932712 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>139756443933104&#45;&gt;139756443932712</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M247.638,-310.7787C250.4651,-303.4542 254.2989,-293.5211 257.6899,-284.7352\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"261.0449,-285.7627 261.3805,-275.1732 254.5145,-283.2421 261.0449,-285.7627\"/>\n",
       "</g>\n",
       "<!-- 139756443934168 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>139756443934168</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"297.5,-389 173.5,-389 173.5,-368 297.5,-368 297.5,-389\"/>\n",
       "<text text-anchor=\"middle\" x=\"235.5\" y=\"-375.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">UnsqueezeBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443934168&#45;&gt;139756443933104 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>139756443934168&#45;&gt;139756443933104</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M237.0047,-367.7787C238.0104,-360.6134 239.3664,-350.9517 240.5793,-342.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"244.0781,-342.5626 242.002,-332.1732 237.146,-341.5896 244.0781,-342.5626\"/>\n",
       "</g>\n",
       "<!-- 139756443932376 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>139756443932376</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"275,-446 164,-446 164,-425 275,-425 275,-446\"/>\n",
       "<text text-anchor=\"middle\" x=\"219.5\" y=\"-432.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443932376&#45;&gt;139756443934168 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>139756443932376&#45;&gt;139756443934168</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M222.5095,-424.7787C224.5431,-417.5338 227.2932,-407.7367 229.7394,-399.0221\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"233.1711,-399.747 232.504,-389.1732 226.4316,-397.8552 233.1711,-399.747\"/>\n",
       "</g>\n",
       "<!-- 139756443762816 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>139756443762816</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"235.5,-503 141.5,-503 141.5,-482 235.5,-482 235.5,-503\"/>\n",
       "<text text-anchor=\"middle\" x=\"188.5\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">ReluBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443762816&#45;&gt;139756443932376 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>139756443762816&#45;&gt;139756443932376</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M194.3309,-481.7787C198.401,-474.2949 203.9521,-464.088 208.8056,-455.1638\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"211.9922,-456.6303 213.6953,-446.1732 205.8428,-453.2858 211.9922,-456.6303\"/>\n",
       "</g>\n",
       "<!-- 139756443762928 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>139756443762928</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"233.5,-560 141.5,-560 141.5,-539 233.5,-539 233.5,-560\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.5\" y=\"-546.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443762928&#45;&gt;139756443762816 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>139756443762928&#45;&gt;139756443762816</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M187.6881,-538.7787C187.8138,-531.6134 187.9833,-521.9517 188.1349,-513.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"191.6367,-513.2331 188.3128,-503.1732 184.6378,-513.1102 191.6367,-513.2331\"/>\n",
       "</g>\n",
       "<!-- 139756443763040 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>139756443763040</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"266,-617 109,-617 109,-596 266,-596 266,-617\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.5\" y=\"-603.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">CudnnConvolutionBackward</text>\n",
       "</g>\n",
       "<!-- 139756443763040&#45;&gt;139756443762928 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>139756443763040&#45;&gt;139756443762928</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M187.5,-595.7787C187.5,-588.6134 187.5,-578.9517 187.5,-570.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"191.0001,-570.1732 187.5,-560.1732 184.0001,-570.1732 191.0001,-570.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763208 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>139756443763208</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"489.5,-680.5 365.5,-680.5 365.5,-659.5 489.5,-659.5 489.5,-680.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"427.5\" y=\"-666.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">UnsqueezeBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443763208&#45;&gt;139756443763040 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>139756443763208&#45;&gt;139756443763040</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M387.5995,-659.443C346.3345,-648.5249 281.8129,-631.4536 237.2465,-619.6621\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"238.0585,-616.2566 227.4959,-617.0823 236.268,-623.0237 238.0585,-616.2566\"/>\n",
       "</g>\n",
       "<!-- 139756443763488 -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>139756443763488</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"550,-617 393,-617 393,-596 550,-596 550,-617\"/>\n",
       "<text text-anchor=\"middle\" x=\"471.5\" y=\"-603.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">CudnnConvolutionBackward</text>\n",
       "</g>\n",
       "<!-- 139756443763208&#45;&gt;139756443763488 -->\n",
       "<g id=\"edge26\" class=\"edge\">\n",
       "<title>139756443763208&#45;&gt;139756443763488</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M434.964,-659.2281C441.3234,-650.0503 450.616,-636.6394 458.2379,-625.6396\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"461.3276,-627.3259 464.1462,-617.1128 455.5739,-623.339 461.3276,-627.3259\"/>\n",
       "</g>\n",
       "<!-- 139756443764048 -->\n",
       "<g id=\"node37\" class=\"node\">\n",
       "<title>139756443764048</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"725,-617 568,-617 568,-596 725,-596 725,-617\"/>\n",
       "<text text-anchor=\"middle\" x=\"646.5\" y=\"-603.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">CudnnConvolutionBackward</text>\n",
       "</g>\n",
       "<!-- 139756443763208&#45;&gt;139756443764048 -->\n",
       "<g id=\"edge38\" class=\"edge\">\n",
       "<title>139756443763208&#45;&gt;139756443764048</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M464.1557,-659.3715C501.615,-648.51 559.8894,-631.6131 600.4419,-619.8547\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"601.4601,-623.2038 610.0898,-617.0573 599.5107,-616.4807 601.4601,-623.2038\"/>\n",
       "</g>\n",
       "<!-- 139756443763320 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>139756443763320</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"487.5,-744 367.5,-744 367.5,-723 487.5,-723 487.5,-744\"/>\n",
       "<text text-anchor=\"middle\" x=\"427.5\" y=\"-730.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">EmbeddingBackward</text>\n",
       "</g>\n",
       "<!-- 139756443763320&#45;&gt;139756443763208 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>139756443763320&#45;&gt;139756443763208</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M427.5,-722.7281C427.5,-714.0091 427.5,-701.4699 427.5,-690.8068\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"431.0001,-690.6128 427.5,-680.6128 424.0001,-690.6129 431.0001,-690.6128\"/>\n",
       "</g>\n",
       "<!-- 139756443931312 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>139756443931312</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"479.5,-814 375.5,-814 375.5,-780 479.5,-780 479.5,-814\"/>\n",
       "<text text-anchor=\"middle\" x=\"427.5\" y=\"-800.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">embedding.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"427.5\" y=\"-787.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (25002, 100)</text>\n",
       "</g>\n",
       "<!-- 139756443931312&#45;&gt;139756443763320 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>139756443931312&#45;&gt;139756443763320</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M427.5,-779.9832C427.5,-772.1157 427.5,-762.6973 427.5,-754.4019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"431.0001,-754.3686 427.5,-744.3687 424.0001,-754.3687 431.0001,-754.3686\"/>\n",
       "</g>\n",
       "<!-- 139756443932152 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>139756443932152</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"234,-687 141,-687 141,-653 234,-653 234,-687\"/>\n",
       "<text text-anchor=\"middle\" x=\"187.5\" y=\"-673.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">convs.0.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"187.5\" y=\"-660.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (100, 1, 3, 100)</text>\n",
       "</g>\n",
       "<!-- 139756443932152&#45;&gt;139756443763040 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>139756443932152&#45;&gt;139756443763040</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M187.5,-652.9832C187.5,-645.1157 187.5,-635.6973 187.5,-627.4019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"191.0001,-627.3686 187.5,-617.3687 184.0001,-627.3687 191.0001,-627.3686\"/>\n",
       "</g>\n",
       "<!-- 139756443763096 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>139756443763096</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"91,-617 0,-617 0,-596 91,-596 91,-617\"/>\n",
       "<text text-anchor=\"middle\" x=\"45.5\" y=\"-603.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">ViewBackward</text>\n",
       "</g>\n",
       "<!-- 139756443763096&#45;&gt;139756443762928 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>139756443763096&#45;&gt;139756443762928</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M71.8763,-595.9123C94.3928,-586.874 126.9788,-573.7937 151.8224,-563.8213\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"153.2885,-567.0043 161.2649,-560.031 150.6808,-560.5081 153.2885,-567.0043\"/>\n",
       "</g>\n",
       "<!-- 139756443932600 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>139756443932600</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"83,-687 8,-687 8,-653 83,-653 83,-687\"/>\n",
       "<text text-anchor=\"middle\" x=\"45.5\" y=\"-673.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">convs.0.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"45.5\" y=\"-660.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (100)</text>\n",
       "</g>\n",
       "<!-- 139756443932600&#45;&gt;139756443763096 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>139756443932600&#45;&gt;139756443763096</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M45.5,-652.9832C45.5,-645.1157 45.5,-635.6973 45.5,-627.4019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"49.0001,-627.3686 45.5,-617.3687 42.0001,-627.3687 49.0001,-627.3686\"/>\n",
       "</g>\n",
       "<!-- 139756443930864 -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>139756443930864</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"501,-218 390,-218 390,-197 501,-197 501,-218\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-204.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443930864&#45;&gt;139756443933048 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>139756443930864&#45;&gt;139756443933048</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-196.7281C445.5,-188.0091 445.5,-175.4699 445.5,-164.8068\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-164.6128 445.5,-154.6128 442.0001,-164.6129 449.0001,-164.6128\"/>\n",
       "</g>\n",
       "<!-- 139756443932264 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>139756443932264</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"501,-275 390,-275 390,-254 501,-254 501,-275\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-261.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443932264&#45;&gt;139756443930864 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>139756443932264&#45;&gt;139756443930864</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-253.7787C445.5,-246.6134 445.5,-236.9517 445.5,-228.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-228.1732 445.5,-218.1732 442.0001,-228.1732 449.0001,-228.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443934112 -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>139756443934112</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"535.5,-332 355.5,-332 355.5,-311 535.5,-311 535.5,-332\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-318.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MaxPool2DWithIndicesBackward</text>\n",
       "</g>\n",
       "<!-- 139756443934112&#45;&gt;139756443932264 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>139756443934112&#45;&gt;139756443932264</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-310.7787C445.5,-303.6134 445.5,-293.9517 445.5,-285.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-285.1732 445.5,-275.1732 442.0001,-285.1732 449.0001,-285.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443762872 -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>139756443762872</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"507.5,-389 383.5,-389 383.5,-368 507.5,-368 507.5,-389\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-375.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">UnsqueezeBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443762872&#45;&gt;139756443934112 -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>139756443762872&#45;&gt;139756443934112</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-367.7787C445.5,-360.6134 445.5,-350.9517 445.5,-342.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-342.1732 445.5,-332.1732 442.0001,-342.1732 449.0001,-342.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763152 -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>139756443763152</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"501,-446 390,-446 390,-425 501,-425 501,-446\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-432.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443763152&#45;&gt;139756443762872 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>139756443763152&#45;&gt;139756443762872</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-424.7787C445.5,-417.6134 445.5,-407.9517 445.5,-399.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-399.1732 445.5,-389.1732 442.0001,-399.1732 449.0001,-399.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763432 -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>139756443763432</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"492.5,-503 398.5,-503 398.5,-482 492.5,-482 492.5,-503\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">ReluBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443763432&#45;&gt;139756443763152 -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>139756443763432&#45;&gt;139756443763152</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-481.7787C445.5,-474.6134 445.5,-464.9517 445.5,-456.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-456.1732 445.5,-446.1732 442.0001,-456.1732 449.0001,-456.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763376 -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>139756443763376</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"491.5,-560 399.5,-560 399.5,-539 491.5,-539 491.5,-560\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-546.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443763376&#45;&gt;139756443763432 -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>139756443763376&#45;&gt;139756443763432</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M445.5,-538.7787C445.5,-531.6134 445.5,-521.9517 445.5,-513.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"449.0001,-513.1732 445.5,-503.1732 442.0001,-513.1732 449.0001,-513.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763488&#45;&gt;139756443763376 -->\n",
       "<g id=\"edge25\" class=\"edge\">\n",
       "<title>139756443763488&#45;&gt;139756443763376</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M466.6096,-595.7787C463.2323,-588.3746 458.6391,-578.3049 454.5996,-569.4491\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"457.703,-567.8189 450.3685,-560.1732 451.3342,-570.7239 457.703,-567.8189\"/>\n",
       "</g>\n",
       "<!-- 139756443933608 -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>139756443933608</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"601,-687 508,-687 508,-653 601,-653 601,-687\"/>\n",
       "<text text-anchor=\"middle\" x=\"554.5\" y=\"-673.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">convs.1.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"554.5\" y=\"-660.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (100, 1, 4, 100)</text>\n",
       "</g>\n",
       "<!-- 139756443933608&#45;&gt;139756443763488 -->\n",
       "<g id=\"edge27\" class=\"edge\">\n",
       "<title>139756443933608&#45;&gt;139756443763488</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M532.2575,-652.9832C520.3564,-643.8781 505.7403,-632.6959 493.8264,-623.581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"495.7752,-620.6652 485.7063,-617.3687 491.5218,-626.2248 495.7752,-620.6652\"/>\n",
       "</g>\n",
       "<!-- 139756443763656 -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>139756443763656</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"375,-617 284,-617 284,-596 375,-596 375,-617\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.5\" y=\"-603.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">ViewBackward</text>\n",
       "</g>\n",
       "<!-- 139756443763656&#45;&gt;139756443763376 -->\n",
       "<g id=\"edge28\" class=\"edge\">\n",
       "<title>139756443763656&#45;&gt;139756443763376</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M351.0469,-595.9123C369.0337,-587.074 394.8863,-574.3705 414.9988,-564.4877\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"416.6371,-567.5824 424.0685,-560.031 413.55,-561.2999 416.6371,-567.5824\"/>\n",
       "</g>\n",
       "<!-- 139756443932656 -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>139756443932656</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"348,-687 273,-687 273,-653 348,-653 348,-687\"/>\n",
       "<text text-anchor=\"middle\" x=\"310.5\" y=\"-673.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">convs.1.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"310.5\" y=\"-660.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (100)</text>\n",
       "</g>\n",
       "<!-- 139756443932656&#45;&gt;139756443763656 -->\n",
       "<g id=\"edge29\" class=\"edge\">\n",
       "<title>139756443932656&#45;&gt;139756443763656</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M315.5917,-652.9832C317.9722,-645.0273 320.8272,-635.4855 323.3294,-627.1228\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"326.7344,-627.9523 326.248,-617.3687 320.0282,-625.9457 326.7344,-627.9523\"/>\n",
       "</g>\n",
       "<!-- 139756443931480 -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>139756443931480</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"647,-218 536,-218 536,-197 647,-197 647,-218\"/>\n",
       "<text text-anchor=\"middle\" x=\"591.5\" y=\"-204.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443931480&#45;&gt;139756443933048 -->\n",
       "<g id=\"edge30\" class=\"edge\">\n",
       "<title>139756443931480&#45;&gt;139756443933048</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M567.0629,-196.8715C543.0599,-186.4319 506.237,-170.4164 479.4071,-158.7473\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"480.583,-155.442 470.0168,-154.6631 477.7911,-161.8612 480.583,-155.442\"/>\n",
       "</g>\n",
       "<!-- 139756443931648 -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>139756443931648</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"664,-275 553,-275 553,-254 664,-254 664,-275\"/>\n",
       "<text text-anchor=\"middle\" x=\"608.5\" y=\"-261.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443931648&#45;&gt;139756443931480 -->\n",
       "<g id=\"edge31\" class=\"edge\">\n",
       "<title>139756443931648&#45;&gt;139756443931480</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M605.3024,-253.7787C603.1417,-246.5338 600.2197,-236.7367 597.6206,-228.0221\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"600.8954,-226.7557 594.6832,-218.1732 594.1873,-228.7564 600.8954,-226.7557\"/>\n",
       "</g>\n",
       "<!-- 139756443762984 -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>139756443762984</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"735.5,-332 555.5,-332 555.5,-311 735.5,-311 735.5,-332\"/>\n",
       "<text text-anchor=\"middle\" x=\"645.5\" y=\"-318.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MaxPool2DWithIndicesBackward</text>\n",
       "</g>\n",
       "<!-- 139756443762984&#45;&gt;139756443931648 -->\n",
       "<g id=\"edge32\" class=\"edge\">\n",
       "<title>139756443762984&#45;&gt;139756443931648</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M638.5406,-310.7787C633.5793,-303.1357 626.7744,-292.6524 620.8956,-283.596\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"623.8087,-281.6553 615.4282,-275.1732 617.9372,-285.4667 623.8087,-281.6553\"/>\n",
       "</g>\n",
       "<!-- 139756443763544 -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>139756443763544</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"707.5,-389 583.5,-389 583.5,-368 707.5,-368 707.5,-389\"/>\n",
       "<text text-anchor=\"middle\" x=\"645.5\" y=\"-375.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">UnsqueezeBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443763544&#45;&gt;139756443762984 -->\n",
       "<g id=\"edge33\" class=\"edge\">\n",
       "<title>139756443763544&#45;&gt;139756443762984</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M645.5,-367.7787C645.5,-360.6134 645.5,-350.9517 645.5,-342.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"649.0001,-342.1732 645.5,-332.1732 642.0001,-342.1732 649.0001,-342.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763712 -->\n",
       "<g id=\"node34\" class=\"node\">\n",
       "<title>139756443763712</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"701,-446 590,-446 590,-425 701,-425 701,-446\"/>\n",
       "<text text-anchor=\"middle\" x=\"645.5\" y=\"-432.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward1</text>\n",
       "</g>\n",
       "<!-- 139756443763712&#45;&gt;139756443763544 -->\n",
       "<g id=\"edge34\" class=\"edge\">\n",
       "<title>139756443763712&#45;&gt;139756443763544</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M645.5,-424.7787C645.5,-417.6134 645.5,-407.9517 645.5,-399.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"649.0001,-399.1732 645.5,-389.1732 642.0001,-399.1732 649.0001,-399.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763880 -->\n",
       "<g id=\"node35\" class=\"node\">\n",
       "<title>139756443763880</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"692.5,-503 598.5,-503 598.5,-482 692.5,-482 692.5,-503\"/>\n",
       "<text text-anchor=\"middle\" x=\"645.5\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">ReluBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443763880&#45;&gt;139756443763712 -->\n",
       "<g id=\"edge35\" class=\"edge\">\n",
       "<title>139756443763880&#45;&gt;139756443763712</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M645.5,-481.7787C645.5,-474.6134 645.5,-464.9517 645.5,-456.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"649.0001,-456.1732 645.5,-446.1732 642.0001,-456.1732 649.0001,-456.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443763824 -->\n",
       "<g id=\"node36\" class=\"node\">\n",
       "<title>139756443763824</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"692.5,-560 600.5,-560 600.5,-539 692.5,-539 692.5,-560\"/>\n",
       "<text text-anchor=\"middle\" x=\"646.5\" y=\"-546.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 139756443763824&#45;&gt;139756443763880 -->\n",
       "<g id=\"edge36\" class=\"edge\">\n",
       "<title>139756443763824&#45;&gt;139756443763880</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M646.3119,-538.7787C646.1862,-531.6134 646.0167,-521.9517 645.8651,-513.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"649.3622,-513.1102 645.6872,-503.1732 642.3633,-513.2331 649.3622,-513.1102\"/>\n",
       "</g>\n",
       "<!-- 139756443764048&#45;&gt;139756443763824 -->\n",
       "<g id=\"edge37\" class=\"edge\">\n",
       "<title>139756443764048&#45;&gt;139756443763824</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M646.5,-595.7787C646.5,-588.6134 646.5,-578.9517 646.5,-570.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"650.0001,-570.1732 646.5,-560.1732 643.0001,-570.1732 650.0001,-570.1732\"/>\n",
       "</g>\n",
       "<!-- 139756443932936 -->\n",
       "<g id=\"node38\" class=\"node\">\n",
       "<title>139756443932936</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"712,-687 619,-687 619,-653 712,-653 712,-687\"/>\n",
       "<text text-anchor=\"middle\" x=\"665.5\" y=\"-673.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">convs.2.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"665.5\" y=\"-660.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (100, 1, 5, 100)</text>\n",
       "</g>\n",
       "<!-- 139756443932936&#45;&gt;139756443764048 -->\n",
       "<g id=\"edge39\" class=\"edge\">\n",
       "<title>139756443932936&#45;&gt;139756443764048</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M660.4083,-652.9832C658.0278,-645.0273 655.1728,-635.4855 652.6706,-627.1228\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"655.9718,-625.9457 649.752,-617.3687 649.2656,-627.9523 655.9718,-625.9457\"/>\n",
       "</g>\n",
       "<!-- 139756443764104 -->\n",
       "<g id=\"node39\" class=\"node\">\n",
       "<title>139756443764104</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"834,-617 743,-617 743,-596 834,-596 834,-617\"/>\n",
       "<text text-anchor=\"middle\" x=\"788.5\" y=\"-603.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">ViewBackward</text>\n",
       "</g>\n",
       "<!-- 139756443764104&#45;&gt;139756443763824 -->\n",
       "<g id=\"edge40\" class=\"edge\">\n",
       "<title>139756443764104&#45;&gt;139756443763824</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M762.1237,-595.9123C739.6072,-586.874 707.0212,-573.7937 682.1776,-563.8213\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"683.3192,-560.5081 672.7351,-560.031 680.7115,-567.0043 683.3192,-560.5081\"/>\n",
       "</g>\n",
       "<!-- 139756443932096 -->\n",
       "<g id=\"node40\" class=\"node\">\n",
       "<title>139756443932096</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"826,-687 751,-687 751,-653 826,-653 826,-687\"/>\n",
       "<text text-anchor=\"middle\" x=\"788.5\" y=\"-673.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">convs.2.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"788.5\" y=\"-660.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (100)</text>\n",
       "</g>\n",
       "<!-- 139756443932096&#45;&gt;139756443764104 -->\n",
       "<g id=\"edge41\" class=\"edge\">\n",
       "<title>139756443932096&#45;&gt;139756443764104</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M788.5,-652.9832C788.5,-645.1157 788.5,-635.6973 788.5,-627.4019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"792.0001,-627.3686 788.5,-617.3687 785.0001,-627.3687 792.0001,-627.3686\"/>\n",
       "</g>\n",
       "<!-- 139756444320488 -->\n",
       "<g id=\"node41\" class=\"node\">\n",
       "<title>139756444320488</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"604,-84.5 531,-84.5 531,-63.5 604,-63.5 604,-84.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"567.5\" y=\"-70.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">TBackward</text>\n",
       "</g>\n",
       "<!-- 139756444320488&#45;&gt;139756444322952 -->\n",
       "<g id=\"edge42\" class=\"edge\">\n",
       "<title>139756444320488&#45;&gt;139756444322952</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M547.0799,-63.3715C527.3745,-53.115 497.3292,-37.4767 475.0209,-25.8654\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"476.473,-22.6755 465.9867,-21.1631 473.2411,-28.8848 476.473,-22.6755\"/>\n",
       "</g>\n",
       "<!-- 139756444322392 -->\n",
       "<g id=\"node42\" class=\"node\">\n",
       "<title>139756444322392</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"598,-161 537,-161 537,-127 598,-127 598,-161\"/>\n",
       "<text text-anchor=\"middle\" x=\"567.5\" y=\"-147.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">fc.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"567.5\" y=\"-134.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1, 300)</text>\n",
       "</g>\n",
       "<!-- 139756444322392&#45;&gt;139756444320488 -->\n",
       "<g id=\"edge43\" class=\"edge\">\n",
       "<title>139756444322392&#45;&gt;139756444320488</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M567.5,-126.6966C567.5,-117.0634 567.5,-105.003 567.5,-94.8518\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"571.0001,-94.7912 567.5,-84.7913 564.0001,-94.7913 571.0001,-94.7912\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f1b95380160>"
      ]
     },
     "execution_count": 47,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = next(iter(train_iterator)).text\n",
    "make_dot(model(tmp), params=dict(model.named_parameters()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dai9-XzQvg2T"
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 545
    },
    "colab_type": "code",
    "id": "w4ZVy_1_p6px",
    "outputId": "f9aa5198-9815-49e1-fd03-100fb8c24f94"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.640 | Train Acc: 62.36%\n",
      "\t Val. Loss: 0.477 |  Val. Acc: 79.22%\n",
      "Epoch: 02 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.416 | Train Acc: 81.01%\n",
      "\t Val. Loss: 0.348 |  Val. Acc: 85.18%\n",
      "Epoch: 03 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.295 | Train Acc: 87.67%\n",
      "\t Val. Loss: 0.318 |  Val. Acc: 86.68%\n",
      "Epoch: 04 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.219 | Train Acc: 91.28%\n",
      "\t Val. Loss: 0.301 |  Val. Acc: 87.42%\n",
      "Epoch: 05 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.160 | Train Acc: 94.21%\n",
      "\t Val. Loss: 0.305 |  Val. Acc: 87.74%\n",
      "Epoch: 06 | Epoch Time: 0m 33s\n",
      "\tTrain Loss: 0.110 | Train Acc: 96.06%\n",
      "\t Val. Loss: 0.323 |  Val. Acc: 87.44%\n",
      "Epoch: 07 | Epoch Time: 0m 33s\n",
      "\tTrain Loss: 0.080 | Train Acc: 97.29%\n",
      "\t Val. Loss: 0.346 |  Val. Acc: 87.72%\n",
      "Epoch: 08 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.057 | Train Acc: 98.11%\n",
      "\t Val. Loss: 0.382 |  Val. Acc: 87.18%\n",
      "Epoch: 09 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.044 | Train Acc: 98.52%\n",
      "\t Val. Loss: 0.403 |  Val. Acc: 87.38%\n",
      "Epoch: 10 | Epoch Time: 0m 34s\n",
      "\tTrain Loss: 0.033 | Train Acc: 99.02%\n",
      "\t Val. Loss: 0.444 |  Val. Acc: 86.84%\n"
     ]
    }
   ],
   "source": [
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "criterion = criterion.to(device)\n",
    "\n",
    "N_EPOCHS = 10\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    start_time = time.time()    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)    \n",
    "    end_time = time.time()\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "\n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'TextCNN_train.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "ksHfWNpZyhtx",
    "outputId": "f07bf9b1-9001-49e1-ea3a-77445455c33a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 80,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('TextCNN_train.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R1FGc3Kpyp5m"
   },
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We adpoted the best model to run on the test dataset. The TextCNN can have around 86% accuracy rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTest Loss: 0.333 | Test Acc: 85.94%\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "print(f'\\tTest Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F8UvnnNq6wso"
   },
   "source": [
    "### 3.3 RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use Bi-directional LSTM as our model. Compared with a standard RNN, LSTMs overcome this by having an extra recurrent state called a cell, $c$ - which can be thought of as the \"memory\" of the LSTM - and the use use multiple gates which control the flow of information into and out of the memory. \n",
    "\n",
    "As well as having an RNN processing the words in the sentence from the first to the last (a forward RNN), we have a second RNN processing the words in the sentence from the last to the first (a backward RNN). At time step $t$, the forward RNN is processing word $x_t$, and the backward RNN is processing word $x_{T-t+1}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BiLSTM](https://raw.githubusercontent.com/wangys96/6010S-NLP/master/pic/textrnn.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A-pHzyuw62oN"
   },
   "source": [
    "#### Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wk9PJDEy66AV"
   },
   "outputs": [],
   "source": [
    "class RNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim,\n",
    "               n_layers, bidirectional, dropout, pad_idx):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size,embedding_dim,padding_idx=pad_idx)\n",
    "\n",
    "        self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers=n_layers,\n",
    "                           bidirectional=bidirectional, dropout=dropout)\n",
    "\n",
    "        self.fc = nn.Linear(hidden_dim*2, output_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "    \n",
    "    \n",
    "    def forward(self, text):\n",
    "        # [sent len, batch size, emb dim]\n",
    "        embedded = self.dropout(self.embedding(text)) \n",
    "\n",
    "        # output = [sent len, batch size, hid dim * num directions]\n",
    "        # hidden = [num layers * num directions, batch size, hid dim]\n",
    "        # cell = [num layers * num directions, batch size, hid dim]\n",
    "        output, (hidden, cell) = self.rnn(embedded)\n",
    "\n",
    "        # concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:])\n",
    "        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1))\n",
    "    \n",
    "        return self.fc(hidden)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aidzGSBqFb6E"
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM,\n",
    "            N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)\n",
    "model.embedding.weight.data.copy_(TEXT.vocab.vectors)\n",
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "YNIYWewVFn3v",
    "outputId": "cd0d82b0-1ed4-4c64-c9c1-be3ef1938ab2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Parameters:  4810857\n"
     ]
    }
   ],
   "source": [
    "print(\"Network Parameters: \",count_parameters(model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 376
    },
    "colab_type": "code",
    "id": "semW1STvW-TZ",
    "outputId": "e70649f8-b27c-4c7d-c643-c03f107eda64"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"864pt\" height=\"252pt\"\n",
       " viewBox=\"0.00 0.00 864.00 252.06\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(.3872 .3872) rotate(0) translate(4 647)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-647 2227.5,-647 2227.5,4 -4,4\"/>\n",
       "<!-- 140035750973280 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140035750973280</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"#000000\" points=\"1169,-21 1065,-21 1065,0 1169,0 1169,-21\"/>\n",
       "<text text-anchor=\"middle\" x=\"1117\" y=\"-7.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddmmBackward</text>\n",
       "</g>\n",
       "<!-- 140035750973336 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140035750973336</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1043,-91 989,-91 989,-57 1043,-57 1043,-91\"/>\n",
       "<text text-anchor=\"middle\" x=\"1016\" y=\"-77.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">fc.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"1016\" y=\"-64.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 140035750973336&#45;&gt;140035750973280 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140035750973336&#45;&gt;140035750973280</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1043.0661,-56.9832C1058.0669,-47.552 1076.6121,-35.8924 1091.3762,-26.61\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1093.4119,-29.4645 1100.0148,-21.1788 1089.686,-23.5384 1093.4119,-29.4645\"/>\n",
       "</g>\n",
       "<!-- 140035750973168 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140035750973168</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1172.5,-84.5 1061.5,-84.5 1061.5,-63.5 1172.5,-63.5 1172.5,-84.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1117\" y=\"-70.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SqueezeBackward0</text>\n",
       "</g>\n",
       "<!-- 140035750973168&#45;&gt;140035750973280 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140035750973168&#45;&gt;140035750973280</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1117,-63.2281C1117,-54.5091 1117,-41.9699 1117,-31.3068\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1120.5001,-31.1128 1117,-21.1128 1113.5001,-31.1129 1120.5001,-31.1128\"/>\n",
       "</g>\n",
       "<!-- 140035750970984 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140035750970984</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1182,-154.5 1048,-154.5 1048,-133.5 1182,-133.5 1182,-154.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1115\" y=\"-140.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">FusedDropoutBackward</text>\n",
       "</g>\n",
       "<!-- 140035750970984&#45;&gt;140035750973168 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140035750970984&#45;&gt;140035750973168</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1115.3038,-133.3685C1115.5945,-123.1925 1116.0411,-107.5606 1116.4031,-94.8912\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1119.9063,-94.8275 1116.6934,-84.7315 1112.9091,-94.6275 1119.9063,-94.8275\"/>\n",
       "</g>\n",
       "<!-- 140035750970872 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140035750970872</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1156,-218 1074,-218 1074,-197 1156,-197 1156,-218\"/>\n",
       "<text text-anchor=\"middle\" x=\"1115\" y=\"-204.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">CatBackward</text>\n",
       "</g>\n",
       "<!-- 140035750970872&#45;&gt;140035750970984 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140035750970872&#45;&gt;140035750970984</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1115,-196.7281C1115,-188.0091 1115,-175.4699 1115,-164.8068\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1118.5001,-164.6128 1115,-154.6128 1111.5001,-164.6129 1118.5001,-164.6128\"/>\n",
       "</g>\n",
       "<!-- 140035750970648 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140035750970648</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1106.5,-275 1017.5,-275 1017.5,-254 1106.5,-254 1106.5,-275\"/>\n",
       "<text text-anchor=\"middle\" x=\"1062\" y=\"-261.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SliceBackward</text>\n",
       "</g>\n",
       "<!-- 140035750970648&#45;&gt;140035750970872 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140035750970648&#45;&gt;140035750970872</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1071.9689,-253.7787C1079.2976,-245.8969 1089.435,-234.9944 1098.0295,-225.7512\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1100.8295,-227.8799 1105.0758,-218.1732 1095.7032,-223.1133 1100.8295,-227.8799\"/>\n",
       "</g>\n",
       "<!-- 140035750969808 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140035750969808</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1105.5,-332 1016.5,-332 1016.5,-311 1105.5,-311 1105.5,-332\"/>\n",
       "<text text-anchor=\"middle\" x=\"1061\" y=\"-318.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SliceBackward</text>\n",
       "</g>\n",
       "<!-- 140035750969808&#45;&gt;140035750970648 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140035750969808&#45;&gt;140035750970648</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1061.1881,-310.7787C1061.3138,-303.6134 1061.4833,-293.9517 1061.6349,-285.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1065.1367,-285.2331 1061.8128,-275.1732 1058.1378,-285.1102 1065.1367,-285.2331\"/>\n",
       "</g>\n",
       "<!-- 140035750969864 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140035750969864</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1106,-389 1012,-389 1012,-368 1106,-368 1106,-389\"/>\n",
       "<text text-anchor=\"middle\" x=\"1059\" y=\"-375.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SelectBackward</text>\n",
       "</g>\n",
       "<!-- 140035750969864&#45;&gt;140035750969808 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140035750969864&#45;&gt;140035750969808</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1059.3762,-367.7787C1059.6276,-360.6134 1059.9666,-350.9517 1060.2698,-342.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1063.7726,-342.2898 1060.6255,-332.1732 1056.7769,-342.0443 1063.7726,-342.2898\"/>\n",
       "</g>\n",
       "<!-- 140035750969920 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140035750969920</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1174,-446 1056,-446 1056,-425 1174,-425 1174,-446\"/>\n",
       "<text text-anchor=\"middle\" x=\"1115\" y=\"-432.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">CudnnRnnBackward</text>\n",
       "</g>\n",
       "<!-- 140035750969920&#45;&gt;140035750969864 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140035750969920&#45;&gt;140035750969864</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1104.4668,-424.7787C1096.645,-416.8173 1085.7954,-405.7739 1076.6564,-396.4717\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1078.9909,-393.8537 1069.4859,-389.1732 1073.9975,-398.7595 1078.9909,-393.8537\"/>\n",
       "</g>\n",
       "<!-- 140035750969416 -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>140035750969416</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1218,-389 1124,-389 1124,-368 1218,-368 1218,-389\"/>\n",
       "<text text-anchor=\"middle\" x=\"1171\" y=\"-375.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SelectBackward</text>\n",
       "</g>\n",
       "<!-- 140035750969920&#45;&gt;140035750969416 -->\n",
       "<g id=\"edge31\" class=\"edge\">\n",
       "<title>140035750969920&#45;&gt;140035750969416</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1125.5332,-424.7787C1133.355,-416.8173 1144.2046,-405.7739 1153.3436,-396.4717\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1156.0025,-398.7595 1160.5141,-389.1732 1151.0091,-393.8537 1156.0025,-398.7595\"/>\n",
       "</g>\n",
       "<!-- 140035750970032 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140035750970032</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"134,-509.5 0,-509.5 0,-488.5 134,-488.5 134,-509.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"67\" y=\"-495.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">FusedDropoutBackward</text>\n",
       "</g>\n",
       "<!-- 140035750970032&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140035750970032&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M108.1827,-488.4321C119.4057,-485.9243 131.6105,-483.5346 143,-482 317.6805,-458.4637 853.3593,-442.4028 1045.4998,-437.2677\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1045.8485,-440.7597 1055.752,-436.9953 1045.6626,-433.7622 1045.8485,-440.7597\"/>\n",
       "</g>\n",
       "<!-- 140035751085896 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140035751085896</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"127,-573 7,-573 7,-552 127,-552 127,-573\"/>\n",
       "<text text-anchor=\"middle\" x=\"67\" y=\"-559.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">EmbeddingBackward</text>\n",
       "</g>\n",
       "<!-- 140035751085896&#45;&gt;140035750970032 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140035751085896&#45;&gt;140035750970032</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M67,-551.7281C67,-543.0091 67,-530.4699 67,-519.8068\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"70.5001,-519.6128 67,-509.6128 63.5001,-519.6129 70.5001,-519.6128\"/>\n",
       "</g>\n",
       "<!-- 140035751086008 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140035751086008</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"119,-643 15,-643 15,-609 119,-609 119,-643\"/>\n",
       "<text text-anchor=\"middle\" x=\"67\" y=\"-629.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">embedding.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"67\" y=\"-616.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (25002, 100)</text>\n",
       "</g>\n",
       "<!-- 140035751086008&#45;&gt;140035751085896 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140035751086008&#45;&gt;140035751085896</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M67,-608.9832C67,-601.1157 67,-591.6973 67,-583.4019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"70.5001,-583.3686 67,-573.3687 63.5001,-583.3687 70.5001,-583.3686\"/>\n",
       "</g>\n",
       "<!-- 140035750969584 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140035750969584</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"250,-516 152,-516 152,-482 250,-482 250,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"201\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_ih_l0</text>\n",
       "<text text-anchor=\"middle\" x=\"201\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 100)</text>\n",
       "</g>\n",
       "<!-- 140035750969584&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140035750969584&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M250.1791,-483.7463C253.1479,-483.0872 256.1041,-482.4962 259,-482 409.6592,-456.1859 869.6465,-441.86 1045.5717,-437.2059\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1045.8375,-440.7002 1055.7424,-436.9392 1045.654,-433.7026 1045.8375,-440.7002\"/>\n",
       "</g>\n",
       "<!-- 140035750969528 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140035750969528</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"369.5,-516 268.5,-516 268.5,-482 369.5,-482 369.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"319\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_hh_l0</text>\n",
       "<text text-anchor=\"middle\" x=\"319\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 256)</text>\n",
       "</g>\n",
       "<!-- 140035750969528&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140035750969528&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M369.5648,-483.8551C372.7418,-483.158 375.9053,-482.5299 379,-482 505.9574,-460.2597 888.0049,-443.9686 1045.7312,-437.9909\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1045.9297,-441.486 1055.7909,-437.6121 1045.6662,-434.491 1045.9297,-441.486\"/>\n",
       "</g>\n",
       "<!-- 140035750972104 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140035750972104</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"472,-516 388,-516 388,-482 472,-482 472,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"430\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_ih_l0</text>\n",
       "<text text-anchor=\"middle\" x=\"430\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035750972104&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140035750972104&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M472.0821,-484.0764C475.0782,-483.2911 478.0715,-482.586 481,-482 587.3698,-460.7165 904.2224,-444.7007 1045.5794,-438.4126\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1046.0202,-441.8967 1055.8561,-437.9589 1045.7114,-434.9035 1046.0202,-441.8967\"/>\n",
       "</g>\n",
       "<!-- 140035750971264 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>140035750971264</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"577.5,-516 490.5,-516 490.5,-482 577.5,-482 577.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"534\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_hh_l0</text>\n",
       "<text text-anchor=\"middle\" x=\"534\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035750971264&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>140035750971264&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M577.784,-483.9074C580.5471,-483.1973 583.3015,-482.5521 586,-482 671.9865,-464.4064 922.6857,-447.2615 1045.5275,-439.6297\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1045.8128,-443.1189 1055.578,-439.0089 1045.3811,-436.1322 1045.8128,-443.1189\"/>\n",
       "</g>\n",
       "<!-- 140035750972608 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>140035750972608</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"732.5,-516 595.5,-516 595.5,-482 732.5,-482 732.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"664\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_ih_l0_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"664\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 100)</text>\n",
       "</g>\n",
       "<!-- 140035750972608&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>140035750972608&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M732.8375,-483.6429C735.9295,-483.0648 738.9945,-482.5138 742,-482 847.3619,-463.9889 970.5954,-449.9896 1045.6981,-442.2418\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1046.3775,-445.6907 1055.9691,-441.1903 1045.6646,-438.7271 1046.3775,-445.6907\"/>\n",
       "</g>\n",
       "<!-- 140035750972216 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>140035750972216</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"891,-516 751,-516 751,-482 891,-482 891,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"821\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_hh_l0_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"821\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 256)</text>\n",
       "</g>\n",
       "<!-- 140035750972216&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>140035750972216&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M891.0468,-483.8708C941.47,-472.9801 1008.7162,-458.4559 1056.3222,-448.1736\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1057.1339,-451.5791 1066.1696,-446.0467 1055.6561,-444.7368 1057.1339,-451.5791\"/>\n",
       "</g>\n",
       "<!-- 140035750971040 -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>140035750971040</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1033,-516 909,-516 909,-482 1033,-482 1033,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"971\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_ih_l0_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"971\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035750971040&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>140035750971040&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1009.5894,-481.9832C1031.9853,-472.1072 1059.923,-459.7874 1081.4087,-450.3128\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1083.0458,-453.4162 1090.7835,-446.1788 1080.2214,-447.0113 1083.0458,-453.4162\"/>\n",
       "</g>\n",
       "<!-- 140035750972720 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>140035750972720</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1178.5,-516 1051.5,-516 1051.5,-482 1178.5,-482 1178.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1115\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_hh_l0_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"1115\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035750972720&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>140035750972720&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1115,-481.9832C1115,-474.1157 1115,-464.6973 1115,-456.4019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1118.5001,-456.3686 1115,-446.3687 1111.5001,-456.3687 1118.5001,-456.3686\"/>\n",
       "</g>\n",
       "<!-- 140035750971712 -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>140035750971712</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1295,-516 1197,-516 1197,-482 1295,-482 1295,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1246\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_ih_l1</text>\n",
       "<text text-anchor=\"middle\" x=\"1246\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 512)</text>\n",
       "</g>\n",
       "<!-- 140035750971712&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>140035750971712&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1210.8944,-481.9832C1190.7038,-472.1961 1165.5623,-460.0092 1146.0884,-450.5696\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1147.5556,-447.3913 1137.0303,-446.1788 1144.5022,-453.6903 1147.5556,-447.3913\"/>\n",
       "</g>\n",
       "<!-- 140035750972944 -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>140035750972944</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1414.5,-516 1313.5,-516 1313.5,-482 1414.5,-482 1414.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1364\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_hh_l1</text>\n",
       "<text text-anchor=\"middle\" x=\"1364\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 256)</text>\n",
       "</g>\n",
       "<!-- 140035750972944&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>140035750972944&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1313.2951,-484.4995C1310.1529,-483.6425 1307.0353,-482.8031 1304,-482 1258.8694,-470.0586 1207.4055,-457.4633 1169.5879,-448.3977\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1170.3305,-444.9767 1159.7907,-446.0549 1168.7024,-451.7847 1170.3305,-444.9767\"/>\n",
       "</g>\n",
       "<!-- 140035750972776 -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>140035750972776</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1517,-516 1433,-516 1433,-482 1517,-482 1517,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1475\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_ih_l1</text>\n",
       "<text text-anchor=\"middle\" x=\"1475\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035750972776&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>140035750972776&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1432.8618,-484.3315C1429.8791,-483.4854 1426.9041,-482.697 1424,-482 1342.7989,-462.5107 1247.5295,-449.6467 1184.4684,-442.487\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1184.5709,-438.9767 1174.244,-441.3441 1183.7932,-445.9334 1184.5709,-438.9767\"/>\n",
       "</g>\n",
       "<!-- 140035750972664 -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>140035750972664</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1622.5,-516 1535.5,-516 1535.5,-482 1622.5,-482 1622.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1579\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_hh_l1</text>\n",
       "<text text-anchor=\"middle\" x=\"1579\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035750972664&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>140035750972664&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1535.2215,-484.2216C1532.1189,-483.4017 1529.0233,-482.6492 1526,-482 1407.5005,-456.5539 1266.6019,-444.4486 1184.3489,-439.1537\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1184.3231,-435.6452 1174.1232,-438.5112 1183.884,-442.6315 1184.3231,-435.6452\"/>\n",
       "</g>\n",
       "<!-- 140035750971936 -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>140035750971936</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1777.5,-516 1640.5,-516 1640.5,-482 1777.5,-482 1777.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1709\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_ih_l1_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"1709\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 512)</text>\n",
       "</g>\n",
       "<!-- 140035750971936&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>140035750971936&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1640.2351,-483.3624C1637.4588,-482.8756 1634.7058,-482.4186 1632,-482 1472.8711,-457.3805 1284.0194,-444.4755 1184.4538,-438.9258\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1184.4008,-435.4177 1174.2239,-438.3641 1184.0169,-442.4072 1184.4008,-435.4177\"/>\n",
       "</g>\n",
       "<!-- 140035750972888 -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>140035750972888</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1936,-516 1796,-516 1796,-482 1936,-482 1936,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"1866\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.weight_hh_l1_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"1866\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024, 256)</text>\n",
       "</g>\n",
       "<!-- 140035750972888&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge25\" class=\"edge\">\n",
       "<title>140035750972888&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1795.882,-483.3676C1792.8872,-482.872 1789.9171,-482.4126 1787,-482 1568.4423,-451.0841 1306.863,-440.4789 1184.5122,-437.0404\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1184.2655,-433.5324 1174.1735,-436.7577 1184.0741,-440.5298 1184.2655,-433.5324\"/>\n",
       "</g>\n",
       "<!-- 140035751086456 -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>140035751086456</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"2078,-516 1954,-516 1954,-482 2078,-482 2078,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"2016\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_ih_l1_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"2016\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035751086456&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge26\" class=\"edge\">\n",
       "<title>140035751086456&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1953.7845,-483.4373C1950.8236,-482.9066 1947.8844,-482.4226 1945,-482 1798.938,-460.5984 1356.3015,-443.7076 1184.3496,-437.7848\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1184.2691,-434.2801 1174.1551,-437.4356 1184.0294,-441.276 1184.2691,-434.2801\"/>\n",
       "</g>\n",
       "<!-- 140035751086680 -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>140035751086680</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"2223.5,-516 2096.5,-516 2096.5,-482 2223.5,-482 2223.5,-516\"/>\n",
       "<text text-anchor=\"middle\" x=\"2160\" y=\"-502.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">rnn.bias_hh_l1_reverse</text>\n",
       "<text text-anchor=\"middle\" x=\"2160\" y=\"-489.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1024)</text>\n",
       "</g>\n",
       "<!-- 140035751086680&#45;&gt;140035750969920 -->\n",
       "<g id=\"edge27\" class=\"edge\">\n",
       "<title>140035751086680&#45;&gt;140035750969920</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M2096.4258,-483.481C2093.2481,-482.9298 2090.0932,-482.4305 2087,-482 1912.424,-457.7009 1376.6794,-442.1206 1184.5109,-437.1894\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1184.3432,-433.6841 1174.2572,-436.928 1184.1647,-440.6818 1184.3432,-433.6841\"/>\n",
       "</g>\n",
       "<!-- 140035750973056 -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>140035750973056</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1214.5,-275 1125.5,-275 1125.5,-254 1214.5,-254 1214.5,-275\"/>\n",
       "<text text-anchor=\"middle\" x=\"1170\" y=\"-261.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SliceBackward</text>\n",
       "</g>\n",
       "<!-- 140035750973056&#45;&gt;140035750970872 -->\n",
       "<g id=\"edge28\" class=\"edge\">\n",
       "<title>140035750973056&#45;&gt;140035750970872</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1159.6549,-253.7787C1151.9728,-245.8173 1141.3169,-234.7739 1132.3411,-225.4717\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1134.7611,-222.9391 1125.2987,-218.1732 1129.7237,-227.7997 1134.7611,-222.9391\"/>\n",
       "</g>\n",
       "<!-- 140035750969472 -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>140035750969472</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1214.5,-332 1125.5,-332 1125.5,-311 1214.5,-311 1214.5,-332\"/>\n",
       "<text text-anchor=\"middle\" x=\"1170\" y=\"-318.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SliceBackward</text>\n",
       "</g>\n",
       "<!-- 140035750969472&#45;&gt;140035750973056 -->\n",
       "<g id=\"edge29\" class=\"edge\">\n",
       "<title>140035750969472&#45;&gt;140035750973056</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1170,-310.7787C1170,-303.6134 1170,-293.9517 1170,-285.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1173.5001,-285.1732 1170,-275.1732 1166.5001,-285.1732 1173.5001,-285.1732\"/>\n",
       "</g>\n",
       "<!-- 140035750969416&#45;&gt;140035750969472 -->\n",
       "<g id=\"edge30\" class=\"edge\">\n",
       "<title>140035750969416&#45;&gt;140035750969472</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1170.8119,-367.7787C1170.6862,-360.6134 1170.5167,-350.9517 1170.3651,-342.3097\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1173.8622,-342.1102 1170.1872,-332.1732 1166.8633,-342.2331 1173.8622,-342.1102\"/>\n",
       "</g>\n",
       "<!-- 140035750971824 -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>140035750971824</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"1266.5,-84.5 1193.5,-84.5 1193.5,-63.5 1266.5,-63.5 1266.5,-84.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1230\" y=\"-70.9\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">TBackward</text>\n",
       "</g>\n",
       "<!-- 140035750971824&#45;&gt;140035750973280 -->\n",
       "<g id=\"edge32\" class=\"edge\">\n",
       "<title>140035750971824&#45;&gt;140035750973280</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1211.0863,-63.3715C1192.9975,-53.2066 1165.5019,-37.7555 1144.8983,-26.1774\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1146.4078,-23.0109 1135.9754,-21.1631 1142.9785,-29.1134 1146.4078,-23.0109\"/>\n",
       "</g>\n",
       "<!-- 140035750970480 -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>140035750970480</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"1261.5,-161 1200.5,-161 1200.5,-127 1261.5,-127 1261.5,-161\"/>\n",
       "<text text-anchor=\"middle\" x=\"1231\" y=\"-147.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">fc.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"1231\" y=\"-134.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1, 512)</text>\n",
       "</g>\n",
       "<!-- 140035750970480&#45;&gt;140035750971824 -->\n",
       "<g id=\"edge33\" class=\"edge\">\n",
       "<title>140035750970480&#45;&gt;140035750971824</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M1230.7528,-126.6966C1230.6152,-117.0634 1230.4429,-105.003 1230.2979,-94.8518\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"1233.7967,-94.7402 1230.1542,-84.7913 1226.7975,-94.8403 1233.7967,-94.7402\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f5c9d3149e8>"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = next(iter(train_iterator)).text\n",
    "make_dot(model(tmp), params=dict(model.named_parameters()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8AkjjqAoHx0v"
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 580
    },
    "colab_type": "code",
    "id": "dTpgGoLcaydJ",
    "outputId": "f9b3f5b0-8afb-42b5-f441-99767359bc6e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch: 01 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.594 | Train Acc: 70.69%\n",
      "\t Val. Loss: 0.493 |  Val. Acc: 78.31%\n",
      "Epoch: 02 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.502 | Train Acc: 82.04%\n",
      "\t Val. Loss: 0.373 |  Val. Acc: 84.53%\n",
      "Epoch: 03 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.440 | Train Acc: 85.97%\n",
      "\t Val. Loss: 0.363 |  Val. Acc: 86.17%\n",
      "Epoch: 04 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.391 | Train Acc: 88.27%\n",
      "\t Val. Loss: 0.342 |  Val. Acc: 87.73%\n",
      "Epoch: 05 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.357 | Train Acc: 89.50%\n",
      "\t Val. Loss: 0.350 |  Val. Acc: 88.20%\n",
      "Epoch: 06 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.325 | Train Acc: 90.49%\n",
      "\t Val. Loss: 0.360 |  Val. Acc: 88.60%\n",
      "Epoch: 07 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.299 | Train Acc: 91.28%\n",
      "\t Val. Loss: 0.376 |  Val. Acc: 88.88%\n",
      "Epoch: 08 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.276 | Train Acc: 92.13%\n",
      "\t Val. Loss: 0.382 |  Val. Acc: 89.25%\n",
      "Epoch: 09 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.256 | Train Acc: 92.91%\n",
      "\t Val. Loss: 0.396 |  Val. Acc: 89.45%\n",
      "Epoch: 10 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.239 | Train Acc: 93.41%\n",
      "\t Val. Loss: 0.410 |  Val. Acc: 89.60%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "criterion = criterion.to(device)\n",
    "\n",
    "N_EPOCHS = 10\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    start_time = time.time()\n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'RNN_train.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a2C_l8VQWA9f"
   },
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "lTXL1zHMbKMf",
    "outputId": "e93b7d82-145b-4fe4-b113-46642356d60d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.381 | Test Acc: 86.31%\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U0uKvh4PbvyN"
   },
   "source": [
    "### 3.4 BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Transformer is a kind of model architecture relying entirely on an attention mechanism to draw global dependencies between input and output. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transformers](https://raw.githubusercontent.com/wangys96/6010S-NLP/master/pic/transformers.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **B**idirectional **E**ncoder **R**epresentations from **T**ransformers, is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BERT](https://raw.githubusercontent.com/wangys96/6010S-NLP/master/pic/bert.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use a pre-trained BERT-base model with a simple GRU and linear layer to perform classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6GzKcLYCcpXX"
   },
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, BertModel\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9BNV3dtMIwFZ"
   },
   "source": [
    "#### Tokenize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BERT has its own vocabulary and trained tokenizer, we import and initialize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84,
     "referenced_widgets": [
      "3a67237bb8714caaae9f8acebcc42584",
      "446a235dbfff406a83936fe4e4414aad",
      "8502d1ed8063471e827fcf4a7d9dd1f3",
      "73e9d323d6f9470c916f9cc41203fa94",
      "30f607e368db4b93877c282e741f4696",
      "a9f34a2e152547d5a1ab9783018c3df1",
      "5a407cf88cc447828a839b60071f141e",
      "dd02901dfdc04512bad6d20291883435"
     ]
    },
    "colab_type": "code",
    "id": "ijzDRV0yvHfr",
    "outputId": "9bf72e93-9c89-4589-9a85-11f4bfbdc74b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a67237bb8714caaae9f8acebcc42584",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=231508.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30522"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "len(tokenizer.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A6GlB63_vS0L"
   },
   "outputs": [],
   "source": [
    "cls_token=tokenizer.cls_token\n",
    "sep_token=tokenizer.sep_token\n",
    "pad_token=tokenizer.pad_token\n",
    "unk_token=tokenizer.unk_token\n",
    "cls_token_id=tokenizer.cls_token_id\n",
    "sep_token_id=tokenizer.sep_token_id\n",
    "pad_token_id=tokenizer.pad_token_id\n",
    "unk_token_id=tokenizer.unk_token_id\n",
    "max_input_length = tokenizer.max_model_input_sizes['bert-base-uncased']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-V4kJhdnvehq"
   },
   "outputs": [],
   "source": [
    "def tokenize_and_cut(sentence):\n",
    "    return tokenizer.tokenize(sentence)[:max_input_length-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TULZ2Op_vpiJ"
   },
   "outputs": [],
   "source": [
    "TEXT=data.Field(batch_first=True,use_vocab=False,tokenize=tokenize_and_cut,\n",
    "                preprocessing=tokenizer.convert_tokens_to_ids,\n",
    "                init_token=cls_token_id,\n",
    "                eos_token=sep_token_id,\n",
    "                pad_token=pad_token_id,\n",
    "                unk_token=unk_token_id)\n",
    "\n",
    "LABEL=data.LabelField(dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gIE9wKIlI1Zj"
   },
   "source": [
    "#### Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "id": "rkQPCto_vtxw",
    "outputId": "1cc52054-5709-4395-8f5d-ea91bf27b723"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading aclImdb_v1.tar.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "aclImdb_v1.tar.gz: 100%|██████████| 84.1M/84.1M [00:08<00:00, 9.36MB/s]\n"
     ]
    }
   ],
   "source": [
    "train_data, test_data =datasets.IMDB.splits(TEXT,LABEL)\n",
    "train_data,valid_data=train_data.split(split_ratio=0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LUV5Xt3yv4tM"
   },
   "outputs": [],
   "source": [
    "LABEL.build_vocab(train_data)\n",
    "BATCH_SIZE = 128\n",
    "train_iterator, valid_iterator, test_iterator=data.BucketIterator.splits(\n",
    "    (train_data,valid_data,test_data),\n",
    "    batch_size=BATCH_SIZE,\n",
    "    device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R3dj-em-I5NN"
   },
   "source": [
    "#### Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 116,
     "referenced_widgets": [
      "18605b0de2194792b735795658c760a0",
      "7fe45da824a24f299f95340ec4313836",
      "e7461517bd8047aa8d78a11f34e77922",
      "bd0f14351b274895a333ca8bb2b6d50b",
      "1888088bbe0a42139778d5c160ec0f68",
      "7445526bcbd3465faf044f986ee5938b",
      "3e0f1ff2d74048839bb69cafd256a12d",
      "13d939e34529427a8d31d6edb84ce2f2",
      "39b4125b97e04d7faf61de3493e8af6c",
      "44788a6ce74e400395fda435f2c97854",
      "6597eaeee3904a4da060e246f0ff53b9",
      "f071db692da248c0ac5d80df3546ee9c",
      "ade5fa1757c94843ac0b1543041e0428",
      "5d2ac0cbd74940db89977f08b4ec348d",
      "a3b41f4e803b4c3f8b6b24cad36cf9ee",
      "4dfe0b0cd5db403c9a21c80ae4ad4816"
     ]
    },
    "colab_type": "code",
    "id": "geVnDLRPwAsJ",
    "outputId": "4b6b0dcf-dbcf-4b35-d78e-3dcacc00c8ca"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18605b0de2194792b735795658c760a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39b4125b97e04d7faf61de3493e8af6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=440473133.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from transformers import BertModel\n",
    "bert=BertModel.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pXFeDzpjwH6S"
   },
   "outputs": [],
   "source": [
    "class BERTGRUSentiment(nn.Module):\n",
    "    def __init__(self,bert:BertModel,\n",
    "                 hidden_dim:int,\n",
    "                 output_dim:int,\n",
    "                 n_layers:int,\n",
    "                 bidirectional:bool,\n",
    "                 dropout:float):\n",
    "        super(BERTGRUSentiment, self).__init__()\n",
    "\n",
    "        self.bert=bert\n",
    "        embedding_dim=bert.config.to_dict()['hidden_size']\n",
    "\n",
    "        self.rnn=nn.GRU(embedding_dim,hidden_dim,n_layers,\n",
    "                        bidirectional=bidirectional,\n",
    "                        batch_first=True,\n",
    "                        dropout=0 if n_layers<2 else dropout)\n",
    "\n",
    "        self.fc=nn.Linear(hidden_dim*2 if bidirectional else hidden_dim,output_dim)\n",
    "\n",
    "        self.dropout=nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self,text):\n",
    "        with torch.no_grad():\n",
    "            embedding=self.bert(text)[0]\n",
    "            #embeddiing:(batch,seq,embedding_dim)\n",
    "\n",
    "        _,hidden=self.rnn(embedding)\n",
    "        #(bi*num_layers,batch,hidden_size)\n",
    "\n",
    "        hidden=self.dropout(torch.cat((hidden[-1,:,:],hidden[-2,:,:]),dim=1) if self.rnn.bidirectional else hidden[-1,:,:]) \n",
    "        \n",
    "        return self.fc(hidden)\n",
    "        #（batch,output_dim）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "33ZuRszFwL1d"
   },
   "outputs": [],
   "source": [
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.25\n",
    "\n",
    "model = BERTGRUSentiment(bert,\n",
    "                         HIDDEN_DIM,\n",
    "                         OUTPUT_DIM,\n",
    "                         N_LAYERS,\n",
    "                         BIDIRECTIONAL,\n",
    "                         DROPOUT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4tG-m4cXwMUu"
   },
   "outputs": [],
   "source": [
    "for name, parameters in model.named_parameters():\n",
    "    if name.startswith('bert'):\n",
    "        parameters.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we freeze the BERT network, only 2.7m parameters are trainable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "d8uZCmKLw69u",
    "outputId": "3318886f-0098-4142-b9ad-ba8e8f48b615"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 2,759,169 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fyzQqBrqI8nN"
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "b4kenZ8lw98f"
   },
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
    "    correct = rounded_preds.eq(y).float() \n",
    "    acc = correct.sum() / len(correct)\n",
    "    return acc\n",
    "\n",
    "def train(model:nn.Module, iterator:data.BucketIterator, \n",
    "          optimizer:optim.Adam, criterion:nn.BCEWithLogitsLoss):\n",
    "    epoch_loss = 0.\n",
    "    epoch_acc = 0.\n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        preds=model(batch.text).squeeze(-1)\n",
    "        loss=criterion(preds,batch.label)\n",
    "        acc=binary_accuracy(preds,batch.label)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        epoch_loss+=loss.item()\n",
    "        epoch_acc+=acc.item()\n",
    "    return epoch_loss/len(iterator),epoch_acc/len(iterator)\n",
    "\n",
    "def evaluate(model: nn.Module, iterator: data.BucketIterator,\n",
    "           criterion: nn.BCEWithLogitsLoss):\n",
    "    epoch_loss = 0.\n",
    "    epoch_acc = 0.\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for batch in iterator:\n",
    "            preds = model(batch.text).squeeze(-1)\n",
    "            loss = criterion(preds, batch.label)\n",
    "            acc = binary_accuracy(preds, batch.label)\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RCdUiZz5xUrf"
   },
   "outputs": [],
   "source": [
    "criterion=nn.BCEWithLogitsLoss()\n",
    "criterion=criterion.to(device)\n",
    "model=model.to(device)\n",
    "optimizer=optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17
    },
    "colab_type": "code",
    "id": "hEzObUmKxFay",
    "outputId": "febdb32b-9946-402f-fc1b-d650df75e675"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 16m 57s\n",
      "\tTrain Loss: 0.510 | Train Acc: 73.98%\n",
      "Epoch: 02 | Epoch Time: 17m 7s\n",
      "\tTrain Loss: 0.278 | Train Acc: 88.52%\n",
      "Epoch: 03 | Epoch Time: 17m 9s\n",
      "\tTrain Loss: 0.237 | Train Acc: 90.56%\n",
      "Epoch: 04 | Epoch Time: 17m 11s\n",
      "\tTrain Loss: 0.216 | Train Acc: 91.63%\n",
      "Epoch: 05 | Epoch Time: 17m 10s\n",
      "\tTrain Loss: 0.182 | Train Acc: 93.15%\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 5\n",
    "best_valid_loss = float('inf')\n",
    "for epoch in range(N_EPOCHS):    \n",
    "    start_time = time.time()    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)        \n",
    "    end_time = time.time()        \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "        \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'BERT_train.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "colab_type": "code",
    "id": "xUAg9CUqVspQ",
    "outputId": "ed649e04-62e1-43e5-f9f2-38f24b31003f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 17m 5s\n",
      "\tTrain Loss: 0.159 | Train Acc: 93.94%\n",
      "\t Val. Loss: 0.212 |  Val. Acc: 91.94%\n",
      "Epoch: 02 | Epoch Time: 17m 6s\n",
      "\tTrain Loss: 0.133 | Train Acc: 94.73%\n",
      "\t Val. Loss: 0.243 |  Val. Acc: 92.29%\n",
      "Epoch: 03 | Epoch Time: 17m 9s\n",
      "\tTrain Loss: 0.104 | Train Acc: 96.12%\n",
      "\t Val. Loss: 0.267 |  Val. Acc: 91.35%\n",
      "Epoch: 04 | Epoch Time: 17m 8s\n",
      "\tTrain Loss: 0.087 | Train Acc: 96.93%\n",
      "\t Val. Loss: 0.278 |  Val. Acc: 91.41%\n",
      "Epoch: 05 | Epoch Time: 17m 9s\n",
      "\tTrain Loss: 0.072 | Train Acc: 97.37%\n",
      "\t Val. Loss: 0.334 |  Val. Acc: 91.61%\n"
     ]
    }
   ],
   "source": [
    "best_valid_loss = float('inf')\n",
    "for epoch in range(N_EPOCHS):    \n",
    "    start_time = time.time()    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)        \n",
    "    end_time = time.time()        \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "        \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'BERT_train1.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Hfw8DtiNI--g"
   },
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "DAeiu57OGip4",
    "outputId": "8a32e833-33d0-48ef-fa63-a34d6298e834"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.198 | Test Acc: 92.15%\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('BERT_train.pt'))\n",
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performance on test dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| model |   train accuracy   | train loss  |   test accuracy    |  test loss  |\n",
    "| :--:  | :----------------: |  :-------:  |  :---------------: |  :--------: |\n",
    "| CNN   |       94.21%       |    0.160    |        85.94%      |    0.333    | \n",
    "| LSTM  |       93.41%       |    0.239    |        86.31%      |    0.381    | \n",
    "| BERT  |       97.37%       |    0.072    |        92.15%      |    0.198    | \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The state-of-the-art model BERT outperform other model more then 5% on test set. In other words, the pre-trained encoder can better understand the correlation and sentiment of the sentences. Although it require huge computation power to train even we freeze the BERT net, we believe in the future the transformers model series will continuesly grow larger and perform better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[\\[1\\] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation.](https://nlp.stanford.edu/pubs/glove.pdf)   \n",
    "[\\[2\\] Kim, Y. 2014. Convolutional neural networks for sentence classification.](https://arxiv.org/abs/1408.5882)  \n",
    "[\\[3\\] Vaswani,A.,Shazeer,N.,Parmar,N., Uszkoreit,J.,Jones,L.,&Gomez,A.N.,etal. 2017. Attention is all you need.](https://arxiv.org/abs/1706.03762)  \n",
    "[\\[4\\] DEVLIN, Jacob, et al. Bert: Pre-training of deep bidirectional transformers for language understanding.](https://arxiv.org/abs/1810.04805)\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "gxXHgExBvOYA",
    "dai9-XzQvg2T",
    "R1FGc3Kpyp5m",
    "A-pHzyuw62oN",
    "8AkjjqAoHx0v",
    "a2C_l8VQWA9f"
   ],
   "name": "6010ML.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "13d939e34529427a8d31d6edb84ce2f2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "18605b0de2194792b735795658c760a0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e7461517bd8047aa8d78a11f34e77922",
       "IPY_MODEL_bd0f14351b274895a333ca8bb2b6d50b"
      ],
      "layout": "IPY_MODEL_7fe45da824a24f299f95340ec4313836"
     }
    },
    "1888088bbe0a42139778d5c160ec0f68": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "30f607e368db4b93877c282e741f4696": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "39b4125b97e04d7faf61de3493e8af6c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6597eaeee3904a4da060e246f0ff53b9",
       "IPY_MODEL_f071db692da248c0ac5d80df3546ee9c"
      ],
      "layout": "IPY_MODEL_44788a6ce74e400395fda435f2c97854"
     }
    },
    "3a67237bb8714caaae9f8acebcc42584": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8502d1ed8063471e827fcf4a7d9dd1f3",
       "IPY_MODEL_73e9d323d6f9470c916f9cc41203fa94"
      ],
      "layout": "IPY_MODEL_446a235dbfff406a83936fe4e4414aad"
     }
    },
    "3e0f1ff2d74048839bb69cafd256a12d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "446a235dbfff406a83936fe4e4414aad": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "44788a6ce74e400395fda435f2c97854": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4dfe0b0cd5db403c9a21c80ae4ad4816": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5a407cf88cc447828a839b60071f141e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5d2ac0cbd74940db89977f08b4ec348d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6597eaeee3904a4da060e246f0ff53b9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5d2ac0cbd74940db89977f08b4ec348d",
      "max": 440473133,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ade5fa1757c94843ac0b1543041e0428",
      "value": 440473133
     }
    },
    "73e9d323d6f9470c916f9cc41203fa94": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd02901dfdc04512bad6d20291883435",
      "placeholder": "​",
      "style": "IPY_MODEL_5a407cf88cc447828a839b60071f141e",
      "value": " 232k/232k [00:00&lt;00:00, 247kB/s]"
     }
    },
    "7445526bcbd3465faf044f986ee5938b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7fe45da824a24f299f95340ec4313836": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8502d1ed8063471e827fcf4a7d9dd1f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a9f34a2e152547d5a1ab9783018c3df1",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_30f607e368db4b93877c282e741f4696",
      "value": 231508
     }
    },
    "a3b41f4e803b4c3f8b6b24cad36cf9ee": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a9f34a2e152547d5a1ab9783018c3df1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ade5fa1757c94843ac0b1543041e0428": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "bd0f14351b274895a333ca8bb2b6d50b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_13d939e34529427a8d31d6edb84ce2f2",
      "placeholder": "​",
      "style": "IPY_MODEL_3e0f1ff2d74048839bb69cafd256a12d",
      "value": " 433/433 [00:00&lt;00:00, 620B/s]"
     }
    },
    "dd02901dfdc04512bad6d20291883435": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e7461517bd8047aa8d78a11f34e77922": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7445526bcbd3465faf044f986ee5938b",
      "max": 433,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1888088bbe0a42139778d5c160ec0f68",
      "value": 433
     }
    },
    "f071db692da248c0ac5d80df3546ee9c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4dfe0b0cd5db403c9a21c80ae4ad4816",
      "placeholder": "​",
      "style": "IPY_MODEL_a3b41f4e803b4c3f8b6b24cad36cf9ee",
      "value": " 440M/440M [02:10&lt;00:00, 3.36MB/s]"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}