Train_a_GPT_2_Text_Generating_Model_w_GPU.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Train a GPT-2 Text-Generating Model w/ GPU",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/meisi/Torc-NN-test/blob/master/Train_a_GPT_2_Text_Generating_Model_w_GPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "H7LoMj4GA4n_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#  Train a GPT-2 Text-Generating Model w/ GPU\n",
        "\n",
        "by [Max Woolf](http://minimaxir.com)\n",
        "\n",
        "*Last updated: April 18, 2019*\n",
        "\n",
        "Retrain an advanced text generating neural network on any text dataset **for free on a GPU using Collaboratory** using `gpt-2-simple`!\n",
        "\n",
        "For more about `gpt-2-simple`, you can visit [this GitHub repository](https://github.com/minimaxir/gpt-2-simple).\n",
        "\n",
        "\n",
        "To get started:\n",
        "\n",
        "1. Copy this notebook to your Google Drive to keep it and save your changes. (File -> Save a Copy in Drive)\n",
        "2. Make sure you're running the notebook in Google Chrome.\n",
        "3. Run the cells below:\n"
      ]
    },
    {
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install -q gpt_2_simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0wXB05bPDYxS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Downloading GPT-2\n",
        "\n",
        "If you're retraining a model on new text, you need to download GPT-2 first. The next cell downloads it from Google Cloud Storage and saves it in `/models/117M`.\n",
        "\n",
        "This model isn't saved in the Colaboratory VM; you'll have to redownload it if you want to retrain it at a later time."
      ]
    },
    {
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab_type": "code",
        "outputId": "8101b8d1-12ca-4b61-96ea-e5bd5aae2667",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        }
      },
      "cell_type": "code",
      "source": [
        "gpt2.download_gpt2()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.00kit [00:00, 262kit/s]                                                      \n",
            "Fetching encoder.json: 1.04Mit [00:00, 50.9Mit/s]                                                   \n",
            "Fetching hparams.json: 1.00kit [00:00, 209kit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:08, 58.1Mit/s]                                  \n",
            "Fetching model.ckpt.index: 6.00kit [00:00, 1.29Mit/s]                                               \n",
            "Fetching model.ckpt.meta: 472kit [00:00, 33.8Mit/s]                                                 \n",
            "Fetching vocab.bpe: 457kit [00:00, 34.7Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "N8KXuKWzQSsN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Mounting Google Drive\n",
        "\n",
        "The best way to get input text to-be-trained into the Colaboratory VM, and to get the trained model *out* of the VM, is to route it through Google Drive *first*.\n",
        "\n",
        "Running this cell (which will only work in Colaboratory) will mount your personal Google Drive in the VM, which later cells can use to get data in/out. (it will ask for an auth code; that auth is not saved anywhere)"
      ]
    },
    {
      "metadata": {
        "id": "puq4iC6vUAHc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "gpt2.mount_gdrive()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BT__brhBCvJu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Uploading a Text File to be Trained to Colaboratory\n",
        "\n",
        "In the Colaboratory Notebook sidebar on the left of the screen, select *Files*. From there you can upload files:\n",
        "\n",
        "![alt text](https://i.imgur.com/TGcZT4h.png)\n",
        "\n",
        "Upload **any smaller text file**  (<10 MB) and update the file name in the cell below, then run the cell."
      ]
    },
    {
      "metadata": {
        "id": "6OFnPCLADfll",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "file_name = \"shakespeare.txt\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HeeSKtNWUedE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If your text file is larger than 10MB, it is recommended to upload that file to Google Drive first, then copy that file from Google Drive to the Colaboratory VM."
      ]
    },
    {
      "metadata": {
        "id": "-Z6okFD8VKtS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "gpt2.copy_file_from_gdrive(file_name)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LdpZQXknFNY3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Finetune GPT-2\n",
        "\n",
        "The next cell will start the actual finetuning. It creates a persistent TensorFlow session which stores the training config, then runs the training to the number of `steps`. (to have the finetuning run indefinitely, set `steps = -1`)\n",
        "\n",
        "The model checkpoints will be saved in `/checkpoint/117M` by default. The checkpoints are saved every 1000 steps (can be changed) and when the cell is stopped.\n",
        "\n",
        "The training might time out after 4ish hours; make sure you end training and save the results so you don't lose them!\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files."
      ]
    },
    {
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab_type": "code",
        "outputId": "c39105fd-41ad-4881-a31e-1064440b2291",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 11472
        }
      },
      "cell_type": "code",
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.finetune(sess,\n",
        "              dataset=file_name,\n",
        "              steps=1000,\n",
        "              restore_from='fresh',   # change to 'latest' to resume training\n",
        "              print_every=10,   # how many steps between printing progress\n",
        "              sample_every=200,   # how many steps to print a demo sample\n",
        "              save_every=500   # how many steps between saving checkpoint\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:51: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:53: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.random.categorical instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "Loading checkpoint models/117M/model.ckpt\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "INFO:tensorflow:Restoring parameters from models/117M/model.ckpt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/1 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading dataset...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r100%|██████████| 1/1 [00:02<00:00,  2.16s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "dataset has 338025 tokens\n",
            "Training...\n",
            "[10 | 29.68] loss=3.63 avg=3.63\n",
            "[20 | 52.84] loss=3.42 avg=3.52\n",
            "[30 | 77.01] loss=3.31 avg=3.45\n",
            "[40 | 100.62] loss=3.39 avg=3.44\n",
            "[50 | 123.83] loss=3.20 avg=3.39\n",
            "[60 | 147.08] loss=3.51 avg=3.41\n",
            "[70 | 170.72] loss=3.08 avg=3.36\n",
            "[80 | 194.23] loss=3.07 avg=3.32\n",
            "[90 | 217.66] loss=3.27 avg=3.32\n",
            "[100 | 241.03] loss=3.09 avg=3.29\n",
            "[110 | 264.48] loss=3.09 avg=3.27\n",
            "[120 | 288.01] loss=2.98 avg=3.25\n",
            "[130 | 311.54] loss=3.21 avg=3.24\n",
            "[140 | 335.04] loss=3.27 avg=3.25\n",
            "[150 | 358.55] loss=3.11 avg=3.24\n",
            "[160 | 382.05] loss=3.04 avg=3.22\n",
            "[170 | 405.53] loss=2.98 avg=3.21\n",
            "[180 | 429.09] loss=3.01 avg=3.20\n",
            "[190 | 452.56] loss=2.77 avg=3.17\n",
            "======== SAMPLE 1 ========\n",
            "S. BUCKINGHAM:\n",
            "You'll be there, sir;\n",
            "He that lives not to sit, you must be his\n",
            "mistress; and you must be his partner.\n",
            "\n",
            "DUKE OF YORK:\n",
            "How now, brother:\n",
            "Your brother is come to bed, with a fever.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Brother, have you no reason to be ill?\n",
            "\n",
            "CALIBAN:\n",
            "Not a day goes by without one that comes\n",
            "Not in a cold, but in a hot, hot hour.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Well, brother! you need not say so:\n",
            "It is an old habit to say,\n",
            "And you may say it more, that you think away; and\n",
            "Which, being ill, goes to a fever and then a fever,\n",
            "And thus you fall sick: therefore, be patient, your brother\n",
            "Is well: your brother is one-and-twenty; your sister\n",
            "Will be so soon, being four: your brother lives but one\n",
            "Two days. Now, brother; what, did you say you'll be sick?\n",
            "\n",
            "CALIBAN:\n",
            "If I had said so, I'll know your brother\n",
            "Is well; if, brother, you had said so, I would have\n",
            "Believed you dead.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "What, do you hear my brother talk of dying?\n",
            "\n",
            "CALIBAN:\n",
            "I speak of heaping on himself.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "How now, brother! what, have you not been ill?\n",
            "\n",
            "CALIBAN:\n",
            "All my life have I been well.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Come, brother, I may well believe but that: but I am old, weary,\n",
            "And, in many a sick state, ailing; and my brother\n",
            "Does not like me ill either, having lived: therefore, be patient.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Is not my brother?\n",
            "\n",
            "CALIBAN:\n",
            "God send his light.\n",
            "\n",
            "DUISTY:\n",
            "Come, brother, my soul is full of cold; I am going to\n",
            "Stands upon the Tower, hoping my brother\n",
            "Will come.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "No, I am afraid he shall not; but God knows his\n",
            "fathers not: but my brother is an honest man; for\n",
            "he's a soldier in the war, and a good man: therefore, follow him;\n",
            "he's not going to stand idly by.\n",
            "\n",
            "KING RICHARD III:\n",
            "Bid him to my chamber.\n",
            "\n",
            "BUCKINGHAM:\n",
            "God grant their blessings!\n",
            "\n",
            "BUCKINGHAM:\n",
            "Sirrah, we look in the holy light: we have no need\n",
            "To ask any thing of a minister; nor shall we hear them,\n",
            "but that, if so, we are likely to find reasons\n",
            "Why some may say they see them.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "What would it be, my lord,\n",
            "That we should make us hear our gods?\n",
            "\n",
            "KING RICHARD III:\n",
            "My lord, it is very like,\n",
            "That we should think it strange that we should\n",
            "Speak in the name of God. What is done, what's done?\n",
            "\n",
            "CALIBAN:\n",
            "And that it is done, God forbid; for it is impossible\n",
            "To say whether he hath said, he is coming\n",
            "I'll be his minister, I will give my report,\n",
            "And pray him kindly for good. Therefore, brother Clifford\n",
            "Is gone. O, brother, this is a dangerous time!\n",
            "\n",
            "KING RICHARD III:\n",
            "Nay, and that it was no dangerous time: for, when God\n",
            "First spoke with King Henry of France, he\n",
            "sent the next messenger; and that is the meaning\n",
            "Of his journey unto our aid. Therefore, sirrah, and pray thee,\n",
            "This is comfort I lack you can make!\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Prithee, pray thy brother\n",
            "For his death. This is for all men: go hence:\n",
            "Take from us all that foul thing we have stolen,\n",
            "And leave not this untaken: the king is come.\n",
            "\n",
            "KING RICHARD III:\n",
            "What says my brother?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Away, my lord.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "He's coming?\n",
            "\n",
            "KING RICHARD III:\n",
            "No.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Come, sirrah: I have done a good deed, being\n",
            "The king: and thou hast done, I will yield a ransom of\n",
            "My brother, and of thy brother, of mine. But I, your lord,\n",
            "Will yield to thy demands; for I will\n",
            "\n",
            "[200 | 488.64] loss=2.71 avg=3.15\n",
            "[210 | 512.12] loss=2.73 avg=3.12\n",
            "[220 | 535.57] loss=2.95 avg=3.12\n",
            "[230 | 558.99] loss=2.62 avg=3.09\n",
            "[240 | 582.46] loss=2.58 avg=3.07\n",
            "[250 | 605.91] loss=2.98 avg=3.06\n",
            "[260 | 629.34] loss=2.71 avg=3.05\n",
            "[270 | 652.78] loss=2.87 avg=3.04\n",
            "[280 | 676.23] loss=2.87 avg=3.03\n",
            "[290 | 699.66] loss=2.44 avg=3.01\n",
            "[300 | 723.12] loss=2.32 avg=2.98\n",
            "[310 | 746.60] loss=2.82 avg=2.98\n",
            "[320 | 770.08] loss=2.64 avg=2.97\n",
            "[330 | 793.50] loss=2.34 avg=2.94\n",
            "[340 | 817.02] loss=2.68 avg=2.93\n",
            "[350 | 840.52] loss=2.26 avg=2.91\n",
            "[360 | 863.99] loss=2.31 avg=2.89\n",
            "[370 | 887.52] loss=2.37 avg=2.88\n",
            "[380 | 911.07] loss=2.63 avg=2.87\n",
            "[390 | 934.58] loss=2.33 avg=2.85\n",
            "======== SAMPLE 1 ========\n",
            "'s love and good will to those that love him.\n",
            "Here's this gentleman,\n",
            "Where I can direct you with all the business,\n",
            "What I shall command, upon your request;\n",
            "Unless you advise me for a suit of peace,\n",
            "Wherein your suit must the like be;\n",
            "If not, your suit by exile be for debt,\n",
            "Or imprisonment in prison.\n",
            "\n",
            "LADY ANCHER:\n",
            "If you will know my mind, my father\n",
            "And all the officers of state, my father's love,\n",
            "That I have in this state of spirit and good will\n",
            "To give you all I ask, I vow to follow you\n",
            "Unto this very present hour.\n",
            "\n",
            "LADY ANCHER:\n",
            "I have done nothing dishonour'd: yet God does make good\n",
            "The devil's advocate to sin: but then\n",
            "My spirit and my disposition shall straight make him speak.\n",
            "If you'll tell me aught I would not say;\n",
            "But I would yield my heart to you for your will,\n",
            "And leave you in the discretion.\n",
            "\n",
            "BUSHY:\n",
            "My mind, my lord, is as good free for worship\n",
            "As any sweet jewel that makes it fruitful.\n",
            "You are not free for worship, were you not\n",
            "The prince I command? and ever your will\n",
            "And no amendment your will.\n",
            "\n",
            "BUSHY:\n",
            "How now, sir! what is the matter, sir?\n",
            "\n",
            "HENRY BOLINGBROKE:\n",
            "The matter is this;\n",
            "It was my father that suggested me to the world,\n",
            "Unless the devil make perfect master of me.\n",
            "\n",
            "BUSHY:\n",
            "Is that so?\n",
            "\n",
            "HENRY BOLINGBROKE:\n",
            "No, not at all; and yet, sir, in the name of God,\n",
            "I must confess that I love and do swear the truth,\n",
            "As I love every gentleman in this land,\n",
            "And in that true faith I'll make my mind known;\n",
            "When, if it please you at all, you know,\n",
            "I am to the king my father's person;\n",
            "If not, then by your authority I'll go\n",
            "Where no man would find you. If your suit prevails,\n",
            "Go, for I mean no wrong; and, for your suit\n",
            "As mine, as yours, as yours must.\n",
            "\n",
            "BUSHY:\n",
            "Here is the royal seal.\n",
            "\n",
            "First Lord:\n",
            "You are a strange fellow.\n",
            "\n",
            "Second Lord:\n",
            "A most unsuspected gentleman.\n",
            "\n",
            "First Lord:\n",
            "And, as your suit would entail much trouble,\n",
            "That you should use this means to entreat and demand\n",
            "What you have not received, that you are not free\n",
            "To do this harm in all likelihood, which is much,\n",
            "Much greater than what this world will bear.\n",
            "\n",
            "BUSHY:\n",
            "What, you mean that in this a man shall be free\n",
            "To go where nobody shall come?\n",
            "\n",
            "Third Lord:\n",
            "But what shall I say?\n",
            "\n",
            "BUSHY:\n",
            "Well, as my suit would entail much trouble,\n",
            "And that it will be much worse than what you\n",
            "Are now to order.\n",
            "\n",
            "First Lord:\n",
            "No, not now, not till you be resolved:\n",
            "Your resolution shall have weightier than now.\n",
            "\n",
            "BUSHY:\n",
            "What shall I say?\n",
            "\n",
            "First Lord:\n",
            "No, not now, not till you be resolved.\n",
            "\n",
            "BUSHY:\n",
            "Then I cannot say: you will be much worse\n",
            "In harm to perform your alteration\n",
            "On myself now that I have resolve'd my suit,\n",
            "Or that the other suit now before the king\n",
            "Is, as yet, as dreadful and unpitied,\n",
            "As you now stand before the prince.\n",
            "\n",
            "First Lord:\n",
            "You shall have that.\n",
            "\n",
            "BUSHY:\n",
            "But then you shall know the limit of your suit,\n",
            "As I am then resolved to carry it.\n",
            "\n",
            "First Lord:\n",
            "I might answer.\n",
            "\n",
            "BUSHY:\n",
            "Then, if it please your will, you shall be free.\n",
            "\n",
            "First Lord:\n",
            "I can scarcely make my mind.\n",
            "\n",
            "BUSHY:\n",
            "But I will answer, if you please to leave me.\n",
            "\n",
            "Second Lord:\n",
            "I will not.\n",
            "\n",
            "BUSHY:\n",
            "You shall have my thanks: though you must be afraid,\n",
            "You shall have my kisses: as I am wont\n",
            "Since I got your leave, you shall.\n",
            "\n",
            "First Lord:\n",
            "And you shall.\n",
            "\n",
            "BUSHY:\n",
            "Ay, and you shall be too; for it follows\n",
            "That you are wont to be; and in time my fancy\n",
            "Shall withers, and I die a coward, though I\n",
            "Keep on this life being proud.\n",
            "\n",
            "First Lord:\n",
            "What is your will?\n",
            "\n",
            "BUSHY:\n",
            "What is your will; not\n",
            "\n",
            "[400 | 969.52] loss=2.22 avg=2.83\n",
            "[410 | 993.00] loss=1.98 avg=2.81\n",
            "[420 | 1016.40] loss=1.81 avg=2.78\n",
            "[430 | 1039.98] loss=1.75 avg=2.75\n",
            "[440 | 1063.62] loss=1.95 avg=2.73\n",
            "[450 | 1087.08] loss=2.05 avg=2.71\n",
            "[460 | 1110.52] loss=2.02 avg=2.69\n",
            "[470 | 1134.02] loss=1.57 avg=2.66\n",
            "[480 | 1157.58] loss=1.72 avg=2.63\n",
            "[490 | 1181.15] loss=1.83 avg=2.61\n",
            "Saving checkpoint/run1/model-500\n",
            "[500 | 1208.81] loss=1.67 avg=2.59\n",
            "[510 | 1232.37] loss=1.40 avg=2.56\n",
            "[520 | 1255.97] loss=2.22 avg=2.55\n",
            "[530 | 1279.47] loss=1.71 avg=2.53\n",
            "[540 | 1302.96] loss=1.68 avg=2.51\n",
            "[550 | 1326.40] loss=1.66 avg=2.49\n",
            "[560 | 1349.87] loss=2.04 avg=2.48\n",
            "[570 | 1373.39] loss=1.85 avg=2.47\n",
            "[580 | 1396.87] loss=1.56 avg=2.45\n",
            "[590 | 1420.35] loss=1.40 avg=2.42\n",
            "======== SAMPLE 1 ========\n",
            " poor for that, to be revenged on him, which's as little\n",
            "As to be undone.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "The very thought of it made me think\n",
            "To serve as the patron of my study.\n",
            "Though the poor spirit of this noble duke, poor\n",
            "Servile for false imprisonment, may seem\n",
            "A devilish spirit to our human eyes, I'll confess;\n",
            "And I'll confess his good will yielded no measure\n",
            "To his jealousies; that, he being a subject,\n",
            "Was as free from jealousies as an actor;\n",
            "Who thus was free from all conflicts, he was never\n",
            "Seeking alliance with any power. He loved\n",
            "The people unthankfulness or contempt, which\n",
            "He publicly abused; which power, as it were,\n",
            "With some unwillingness, publicly\n",
            "Cried vengeance, and in some degrees, bitterly\n",
            "Declared all faults of the people faults,\n",
            "Whose base and base valour yielded no measure.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "A noble father, a noble father! it were folly\n",
            "To hang our king thus rashly our countryman,\n",
            "When he had a more ripening heart; whose tears\n",
            "Would cover our nation's history, show me\n",
            "His valour in a subject's face. Would\n",
            "Our casque troublous dream, his rough reverie,\n",
            "Or poor brother's tears, show me this prince?\n",
            "\n",
            "Servile Forbear:\n",
            "Indeed, thou liest, if we had our own.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "We will find him now.\n",
            "\n",
            "Servile Forbear:\n",
            "As it were incense to grope his majesty\n",
            "And make him grope you, so it were a show\n",
            "Of what he was, wert thou not deceived\n",
            "By what he said.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "Nay, I warrant him,\n",
            "We will find him.\n",
            "\n",
            "VOLUMNIA:\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "We shall find him.\n",
            "\n",
            "VOLUMNIA:\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "We'll find him.\n",
            "\n",
            "VOLUMNIA:\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "We have no men.\n",
            "\n",
            "VOLUMNIA:\n",
            "We have devils!\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "No, only devils in princes.\n",
            "\n",
            "VOLUMNIA:\n",
            "You'll swear your free worship?\n",
            "\n",
            "VOLUMNIA:\n",
            "I'll swear it, if a devil touch me.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "You will swear it then, if a little devil touch\n",
            "You?\n",
            "\n",
            "VOLUMNIA:\n",
            "I'll swear it hereafter, if a little devil rain rain\n",
            "On me: but as yet, I do not swear it.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "You will swear it then you'll temper your friend.\n",
            "\n",
            "VOLUMNIA:\n",
            "I say to thee, I will abide his sight\n",
            "As long as his fancy lasts him: but if it last,\n",
            "He will renew it with new and striking wear and tears.\n",
            "Why 'twere well you grumble, my lord:\n",
            "If any of you be of Roman blood, here's one\n",
            "He'll beat to it, the Earl of Wiltshire.\n",
            "\n",
            "DUKE VINCENTIO:\n",
            "Who? that? I humbly thank you.\n",
            "I pray you, good faith do not shun me:\n",
            "I am bound to you; and, adieu, good faith,\n",
            "Go see Lord Angelo here with Lord Montague.\n",
            "\n",
            "LADY CAPULET:\n",
            "Angelo, good night; pray.\n",
            "\n",
            "CAPULET:\n",
            "O, bethink thee, sir, of the night to night\n",
            "So beseem them of the day of unrest!\n",
            "\n",
            "JULIET:\n",
            "Good night, good night; thou canst not wear out thy breath\n",
            "Forcing a tedious story to the heart,\n",
            "But bending it again to the purpose,\n",
            "With new and changing accents.\n",
            "\n",
            "Angelo:\n",
            "I cannot tell;\n",
            "I have heard the poor duke lave in his coat\n",
            "And say the duke lave, in the patrimony of his breath,\n",
            "That the Earl of Wiltshire hath in his coat\n",
            "Some noble duke of the Roman blood;\n",
            "Which may be sworn against all likelihood,\n",
            "Either by Angelo himself or some other great guest\n",
            "Than the duke lave himself.\n",
            "\n",
            "Angelo:\n",
            "No, by my troth, the duke duke lave,\n",
            "Who dares me; and I am sworn his pride\n",
            "With all confidence rooted in truth:\n",
            "To me he bears the coat of arms\n",
            "That I have wearisomely wearisomely wear:\n",
            "But where I aim most am I\n",
            "\n",
            "[600 | 1455.34] loss=1.13 avg=2.39\n",
            "[610 | 1478.80] loss=2.04 avg=2.39\n",
            "[620 | 1502.26] loss=1.05 avg=2.36\n",
            "[630 | 1525.68] loss=1.19 avg=2.33\n",
            "[640 | 1549.18] loss=1.48 avg=2.31\n",
            "[650 | 1572.76] loss=1.57 avg=2.30\n",
            "[660 | 1596.24] loss=1.56 avg=2.28\n",
            "[670 | 1619.73] loss=1.54 avg=2.27\n",
            "[680 | 1643.27] loss=1.27 avg=2.25\n",
            "[690 | 1666.85] loss=1.16 avg=2.23\n",
            "[700 | 1690.37] loss=1.04 avg=2.20\n",
            "[710 | 1713.93] loss=0.66 avg=2.17\n",
            "[720 | 1737.44] loss=0.68 avg=2.14\n",
            "[730 | 1761.00] loss=0.89 avg=2.12\n",
            "[740 | 1784.54] loss=1.22 avg=2.10\n",
            "[750 | 1808.13] loss=1.44 avg=2.09\n",
            "[760 | 1831.70] loss=1.05 avg=2.07\n",
            "[770 | 1855.20] loss=0.86 avg=2.05\n",
            "[780 | 1878.72] loss=1.01 avg=2.03\n",
            "[790 | 1902.23] loss=0.77 avg=2.01\n",
            "======== SAMPLE 1 ========\n",
            "IO\n",
            "You think the nobility may still enjoy the peace of Naples\n",
            "Like savages, who are wont to hunt one another's\n",
            "till they had hunted all, if only for sport,\n",
            "and not for war?\n",
            "\n",
            "BAGOT:\n",
            "The nobility will, they think, but for the peace.\n",
            "\n",
            "FRIAR PETER:\n",
            "That's true, but the nobility will not enjoy\n",
            "The peace because it conflicts with their own\n",
            "Or because it does not like the other's which\n",
            "they cherish; for the nobility would not,\n",
            "forsweary and unfeeling with uncertain times,\n",
            "fear their members make untimely injunctions\n",
            "No longer to prevail with the nobility,\n",
            "Nor unrest their souls with doubtful times. What\n",
            "vice, good Lord Ratcliff? advise you, as wisely\n",
            "As I do, Lord Fermat for Naples advise me\n",
            "For me to Mantua and to Padua help him,\n",
            "And not to Padua because he is but Hereford's son.\n",
            "\n",
            "BAGOT:\n",
            "\n",
            "FRIAR PETER:\n",
            "\n",
            "BAGOT:\n",
            "\n",
            "FRIAR PETER:\n",
            "\n",
            "PAGE:\n",
            "\n",
            "GONZALO:\n",
            "How now, my Gremio! What's the news with you?\n",
            "\n",
            "FLORIZEL:\n",
            "Great tidings; good news, great evils.\n",
            "\n",
            "ISABELLA:\n",
            "And bad, good news?\n",
            "\n",
            "GONZALO:\n",
            "The contrary is truly, great Friar Ratcliff,\n",
            "How our brother's banishment hither comes:\n",
            "How shall we together overcome our differences,\n",
            "And seal the brother from off-foot what is to come\n",
            "A common doom and lasting\n",
            "Your brother's wife shall be yours and not a villain\n",
            "The fee is ten and twenty shillings a head.\n",
            "\n",
            "ISABELLA:\n",
            "Banish him hence and spare not.\n",
            "\n",
            "FLORIZEL:\n",
            "O, sir, he is banished hither;\n",
            "Which to his country and to himself shall come.\n",
            "\n",
            "ISABELLA:\n",
            "O, banish him too!\n",
            "I'll tell you how.\n",
            "\n",
            "FLORIZEL:\n",
            "By my gosh, my good Lord of Naples!\n",
            "I think this is a matter for a noble lord.\n",
            "\n",
            "ESCALUS:\n",
            "My lord, the matter for your royal grace!\n",
            "I have heard say, my lord, the wars which I\n",
            "have been to perform have been most of bloody\n",
            "methinks there be some cause against Naples.\n",
            "\n",
            "MARIANA:\n",
            "O my good lord, forbear this talk!\n",
            "In sign of truth, I will tell you something\n",
            "That shall be considered with some measure of truth;\n",
            "Who hath been, is, and is held, by himself,\n",
            "Most certainly to the end of the world his kindred,\n",
            "That haply he may thrive and prosper in himself.\n",
            "The matter I have to tell you, your noble brother,\n",
            "Is marvellous man; and from my distaff eyes,\n",
            "I see no other such a man alive.\n",
            "\n",
            "LUCIO:\n",
            "Marvellous man!\n",
            "\n",
            "MARIANA:\n",
            "O my dear lord, forbear this talk!\n",
            "\n",
            "LUCIO:\n",
            "If my glories be not so repugnant to you, sir,\n",
            "You must mar a saint; for there is nothing repugnant\n",
            "About him, being both a saint and villain.\n",
            "\n",
            "MARIANA:\n",
            "He that abhors being a saint!\n",
            "\n",
            "ESCALUS:\n",
            "Such a one as you! O my lord,\n",
            "This is a man for your highness to shame.\n",
            "There is a woman by the name of Isabel,\n",
            "Ere much pleased in herself, envied in herself;\n",
            "I have her consent to be her reproof.\n",
            "I see, my lord, that she is one that\n",
            "You would not want in herself; indeed, in thee;\n",
            "More, that you would with her would be guilty of\n",
            "Abstinence.\n",
            "\n",
            "MARIANA:\n",
            "Would I were possessed with child by myself;\n",
            "And that were enough the shame of!\n",
            "\n",
            "LUCIO:\n",
            "Would you with child by yourself?\n",
            "\n",
            "MARIANA:\n",
            "Sir, by consent.\n",
            "\n",
            "ISABELLA:\n",
            "Thy consent is very piteous to you:\n",
            "You marry her for this? No? I would your mercy\n",
            "With all faults I have,--\n",
            "\n",
            "LUCIO:\n",
            "Marry, consent would be murder.\n",
            "\n",
            "MARIANA:\n",
            "As for you,\n",
            "You look'd of death ere you had a cause to look\n",
            "Back, and some impediment to get\n",
            "The mark of what you look for; and so would I\n",
            "Neglect the nuptial haste that often\n",
            "Flow'd my way with desperate looks. What, are we dead?\n",
            "\n",
            "\n",
            "[800 | 1937.19] loss=0.57 avg=1.98\n",
            "[810 | 1960.71] loss=0.69 avg=1.96\n",
            "[820 | 1984.23] loss=0.80 avg=1.94\n",
            "[830 | 2007.70] loss=0.93 avg=1.92\n",
            "[840 | 2031.21] loss=0.87 avg=1.90\n",
            "[850 | 2054.73] loss=0.47 avg=1.88\n",
            "[860 | 2078.25] loss=0.48 avg=1.85\n",
            "[870 | 2101.78] loss=0.54 avg=1.83\n",
            "[880 | 2125.33] loss=0.65 avg=1.81\n",
            "[890 | 2148.86] loss=0.58 avg=1.79\n",
            "[900 | 2172.37] loss=0.54 avg=1.77\n",
            "[910 | 2195.87] loss=0.43 avg=1.74\n",
            "[920 | 2219.31] loss=0.49 avg=1.72\n",
            "[930 | 2242.75] loss=0.63 avg=1.71\n",
            "[940 | 2266.25] loss=0.50 avg=1.69\n",
            "[950 | 2289.77] loss=0.43 avg=1.67\n",
            "[960 | 2313.35] loss=0.35 avg=1.64\n",
            "[970 | 2336.91] loss=0.37 avg=1.62\n",
            "[980 | 2360.51] loss=0.37 avg=1.60\n",
            "[990 | 2384.13] loss=0.52 avg=1.59\n",
            "Saving checkpoint/run1/model-1000\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "IXSuTNERaw6K",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "After the model is trained, you can copy the checkpoint folder to your own Google Drive.\n",
        "\n",
        "If you want to download it to your personal computer, it's strongly recommended you copy it there first, then download from Google Drive."
      ]
    },
    {
      "metadata": {
        "id": "VHdTL8NDbAh3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "gpt2.copy_checkpoint_to_gdrive()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qQJgV_b4bmzd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You're done! Feel free to go to the **Generate Text From The Trained Model** section to generate text based on your retrained model."
      ]
    },
    {
      "metadata": {
        "id": "pel-uBULXO2L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load a Trained Model Checkpoint\n",
        "\n",
        "Running the next cell will copy the `checkpoint` folder from your Google Drive into the Colaboratory VM."
      ]
    },
    {
      "metadata": {
        "id": "DCcx5u7sbPTD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "gpt2.copy_checkpoint_from_gdrive()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RTa6zf3e_9gV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The next cell will allow you to load the retrained model checkpoint + metadata necessary to generate text.\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files."
      ]
    },
    {
      "metadata": {
        "id": "-fxL77nvAMAX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "gpt2.load_gpt2(sess)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ClJwpF_ACONp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Generate Text From The Trained Model\n",
        "\n",
        "After you've trained the model or loaded a retrained model from checkpoint, you can now generate text. `generate` generates a single text with a temperature of `0.7` and a max length of `1023` tokens."
      ]
    },
    {
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab_type": "code",
        "outputId": "0e11dcf7-fbb0-43a7-db4c-a8333f27b8d7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2176
        }
      },
      "cell_type": "code",
      "source": [
        "gpt2.generate(sess)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I have sat here all day\n",
            "Unto this hand, as if it\n",
            "Did some thing nigh me, whoo!\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "What hand?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "My uncle Clarence' greatest kindness.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "My liege, I had rather lie in wait on thee.\n",
            "I promised thy duke should come and apprehend\n",
            "Thy wife and young daughter.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "What, do you mean, to apprehend my daughter?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "To answer the duke.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, what then? 'tis time thou hadst come.\n",
            "Uncle, you have made a sainted father of you.\n",
            "Dare you break your fast before the king's seal\n",
            "And besiege the buried foes with a holy vow?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Why, how now, uncle! hast thou forsook thy wife\n",
            "For divines; for she's the mother of all\n",
            "Abstinence; that she may enter\n",
            "Of these most holy nuns?\n",
            "Or blasphemers, perjured liars, and true\n",
            "Of Christ so manifest?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Have you forsook your wife\n",
            "For any of these liars; for few,\n",
            "When they shall come to speak for them, but they\n",
            "Must heartily confess themselves true.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Why, what, have I left her?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Uncle, you have made good work; but she's changed\n",
            "And is no more hereafter to grace the air\n",
            "Than thou hast ever been a girl.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Uncle, thou hast made good supper; but I am\n",
            "Disgusted with thee. Go not home.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, fie, fie, fie! Why, fie, fie!\n",
            "Thou hast made me requite my former hours of nap,\n",
            "In ordering the day, of all; fie, fie!\n",
            "Thou hast put on my gown, which I have untied up,\n",
            "With a little hoop; and, not to be seen,\n",
            "I have got a pair of bloody shoes,\n",
            "In thine own part, with bloody palms.\n",
            "Fie, fie! fie, fie! I am tired, fie!\n",
            "I have done enough; I am done with thee!\n",
            "Farewell, fie! farewell; I am done.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Prithee, peace enjoin you.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "How fares my uncle Clarence?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "O dear my liege,\n",
            "How is it that thou dost depart so soon?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "With signs of thine that thou wilt not show.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, what of my general?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Are you so resolved that I am not pleased?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "No, no, my good Clarence; nor of our state.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, so?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, then, am I resolved that thou hast\n",
            "Misgave me my due and wronged me; and that\n",
            "Thou hast plotted against me with devilish plots\n",
            "To raze my treasure and take my daughter from me.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Why, so.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Then know you for truth that I have\n",
            "Infer'd my daughter from thee? and thou hast plotted\n",
            "Against me with devilish plots to raze\n",
            "My treasure and take from me a daughter?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "With what malice didst thou first betray\n",
            "My dear son Mowbray? and then did\n",
            "Thou wrong my dear wife and daughter?\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, then know you for truth that I am\n",
            "In earnest peril; and, knowing my meaning,\n",
            "Have plotted against me with devilish plots\n",
            "To raze my treasure and take from me\n",
            "A daughter; and having laid thee on the high ground,\n",
            "Have drawn thy sword more than ever sword of truth.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "I have drawn it,\n",
            "Or sought it; and, if thou slay me,\n",
            "I'll give thee life as quickly as I can.\n",
            "And tell how this ends, that the last\n",
            "Doth not live to\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "oF4-PqF0Fl7R",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If you're creating an API based on your model and need to pass the generated text elsewhere, you can do `text = gpt2.generate(sess, return_as_list=True)[0]`\n",
        "\n",
        "You can also pass in a `prefix` to the generate function to force the text to start with a given character sequence (good if you add an indicator when the text starts).\n",
        "\n",
        "You can also generate multiple texts at a time by specifing `nsamples`. Unique to GPT-2, you can pass a `batch_size` to generate multiple samples in parallel, giving a massive speedup."
      ]
    },
    {
      "metadata": {
        "id": "8DKMc0fiej4N",
        "colab_type": "code",
        "outputId": "ec1fc622-c3fe-43e5-9312-f26cec6e52fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3679
        }
      },
      "cell_type": "code",
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              prefix=\"LORD\",\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "LORD FITZWATER:\n",
            "Saw'st thou not, sir?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Madam, didst thou not mean, as some do,\n",
            "To cram others with beggary?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Ha, ha! what should you say? why didst thou say no?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "Hath this woman in this world hungrier than man?\n",
            "Think'st thou that all these woes can be avoided?\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "As many lives she hath prevented as man.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "Why, hark! the Tower is brought down.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "The King of Naples comes apace.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "It shall be so, I hope.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "The Tower comes down.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "The Naples comes apace.\n",
            "\n",
            "QUEEN ELIZABETH:\n",
            "It shall be so, I hope.\n",
            "\n",
            "DUCHESS OF YORK:\n",
            "\n",
            "====================\n",
            "\n",
            "LORDY:\n",
            "Every man that's gone hath died\n",
            "And that's the end of all.\n",
            "\n",
            "GLOUCESTER:\n",
            "Your lordship must be revenged, Hermione.\n",
            "\n",
            "LADY ANNE:\n",
            "And she shall be the man that I dreamt of.\n",
            "\n",
            "GLOUCESTER:\n",
            "I cannot sleep: never a waking man shall\n",
            "Go to any sleepder than to sleep.\n",
            "\n",
            "LADY ANNE:\n",
            "It shall be such a dream, I'll bear witness to it,\n",
            "That I may believe what I say.\n",
            "\n",
            "GLOUCESTER:\n",
            "All tongues to hell: never man to man,\n",
            "Let light hear me?\n",
            "\n",
            "LADY ANNE:\n",
            "No, I'll not bear witness to it.\n",
            "\n",
            "GLOUCESTER:\n",
            "I said you had heard it, but could not put\n",
            "You in words.\n",
            "\n",
            "LADY ANNE:\n",
            "It is a strange tale, the tale doth it:\n",
            "Hermione's great care is felling on wild Bolingbroke,\n",
            "And Bolingbroke is the king.\n",
            "\n",
            "GLOUCESTER:\n",
            "I could not put mine hand\n",
            "====================\n",
            "\n",
            "LORD FITZWATER:\n",
            "On pain of death, will he live to see this day\n",
            "His country requite itself again.\n",
            "\n",
            "DUKE OF YORK:\n",
            "What country?\n",
            "\n",
            "HENRY BOLINGBROKE:\n",
            "Nor none but himself,\n",
            "That is, most certainly, from this day on\n",
            "My proudest unknown man shall be your king.\n",
            "\n",
            "DUKE OF AUMERLE:\n",
            "Of all men I am the one who shall lose.\n",
            "\n",
            "HENRY BOLINGBROKE:\n",
            "Take thou the crown, and crown it well\n",
            "With all the proudest that have touch'd it.\n",
            "\n",
            "DUKE OF YORK:\n",
            "I'll have the crown too, and set it on me.\n",
            "\n",
            "HENRY BOLINGBROKE:\n",
            "Now York shall stand for his wish.\n",
            "\n",
            "DUKE OF YORK:\n",
            "Stand you like a leader, like a great king\n",
            "And will you yield all obedience to him?\n",
            "\n",
            "HENRY BOLINGBROKE:\n",
            "I'll be king, I vow thee.\n",
            "\n",
            "DUKE OF YORK:\n",
            "And shall stand for that aim so well\n",
            "In all\n",
            "====================\n",
            "\n",
            "LORD FITZWATER:\n",
            "Sir, your company is witness to that.\n",
            "\n",
            "Servant:\n",
            "The duke shall have a chamber-maid,\n",
            "And not an officer of the people, to take\n",
            "The absolute duke of his city.\n",
            "\n",
            "GLOUCESTER:\n",
            "That's as many rogues as there are.\n",
            "\n",
            "Servant:\n",
            "Nine!\n",
            "\n",
            "Servant:\n",
            "Nine rogues!\n",
            "\n",
            "Servant:\n",
            "Nine!\n",
            "\n",
            "GLOUCESTER:\n",
            "A foul witch! Undone, go with her:\n",
            "This jade shall be queen of the dead.\n",
            "\n",
            "Servant:\n",
            "I'll not be queen.\n",
            "\n",
            "Servant:\n",
            "By any other name\n",
            "Will destroy Richard's crown,\n",
            "And London enter in a new sense.\n",
            "\n",
            "Lord:\n",
            "What is the king's name?\n",
            "\n",
            "Lord:\n",
            "Richard's name; 'tis the name of the duke slain\n",
            "In your famous battle.\n",
            "\n",
            "Servant:\n",
            "Nine!\n",
            "\n",
            "Lord:\n",
            "Nine!\n",
            "\n",
            "Servant:\n",
            "Nine!\n",
            "\n",
            "GLOUCESTER:\n",
            "The king!\n",
            "\n",
            "Servant:\n",
            "Nine!\n",
            "\n",
            "Lord:\n",
            "\n",
            "====================\n",
            "\n",
            "LORD STANLEY:\n",
            "Away with her!\n",
            "\n",
            "CATESBY:\n",
            "What counsel pluck, you, or me, or any of your party?\n",
            "\n",
            "WARWICK:\n",
            "Fear me not: I'll tell her she hath made a wrong.\n",
            "\n",
            "CATESBY:\n",
            "I shall feel it, sir, when I see it.\n",
            "\n",
            "WARWICK:\n",
            "Now, how shall we avoid her?\n",
            "\n",
            "KING HENRY VI:\n",
            "By living by, I'll hate myself and my tongue,\n",
            "And never bid farewell my body.\n",
            "\n",
            "CATESBY:\n",
            "'Tis a disgrace to live by,\n",
            "And I will hate myself as well as any\n",
            "Whom fortune hath held dear dear.\n",
            "\n",
            "WARWICK:\n",
            "What is the matter, my lord?\n",
            "\n",
            "NORTHUMBERLAND:\n",
            "Get you hence, gentle queen, by way of reprieve:\n",
            "The Earl of Hereford, with all the rest of Spain,\n",
            "Staying in London till the war be done,\n",
            "And Goverrus with the rest of England,\n",
            "To prevent this loss of life.\n",
            "\n",
            "CATESBY:\n",
            "This is the business.\n",
            "\n",
            "NORTHUM\n",
            "====================\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zjjEN2Tafhl2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For bulk generation, you can generate text to a file and sort out the samples locally on your computer. The next cell will generate a generated text file with a unique timestamp and then download it.\n",
        "\n",
        "NB: For Colaboratory's K80 GPU, I recommend capping the `batch_size` at 20."
      ]
    },
    {
      "metadata": {
        "id": "Fa6p6arifSL0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n",
        "\n",
        "gpt2.generate_to_file(sess,\n",
        "                      destination_path=gen_file,\n",
        "                      length=500,\n",
        "                      temperature=0.7,\n",
        "                      nsamples=100,\n",
        "                      batch_size=20\n",
        "                      )\n",
        "\n",
        "files.download(gen_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "92Zjtsb_Dgj-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Etcetera"
      ]
    },
    {
      "metadata": {
        "id": "ig-KVgkCDCKD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If the notebook has errors (e.g. GPU Sync Fail), force-kill the Colaboratory virtual machine and restart it with the command below:"
      ]
    },
    {
      "metadata": {
        "id": "rIHiVP53FnsX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!kill -9 -1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wmTXWNUygS5E",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# LICENSE\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Max Woolf\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE."
      ]
    }
  ]
}