Exercise+5a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment Analysis of Movie Reviews\n",
    "===================================\n",
    "\n",
    "This tutorial will guide you through the implementation of a recurrent neural network to analyze movie reviews on IMDB and decide if they are positive or negative reviews.\n",
    "\n",
    "The IMDB dataset consists of 25,000 reviews, each with a binary label (1 = positive, 0 = negative). Here is an example review:\n",
    "\n",
    "> “Okay, sorry, but I loved this movie. I just love the whole 80’s genre of these kind of movies, because you don’t see many like this...” -~CupidGrl~\n",
    "\n",
    "The dataset contains a large vocabulary of words, and reviews have variable length ranging from tens to hundreds of words. We reduce the complexity of the dataset with several steps: \n",
    "1. Limit the vocabulary size to `vocab_size = 20000` words by replacing the less frequent words with a Out-Of-Vocab (OOV) character. \n",
    "2. Truncate each example to `max_len = 128` words. \n",
    "3. For reviews with less than max_len words, pad the review with whitespace. This equalizes the review lengths across examples.\n",
    "\n",
    "We have already done this preprocessing and saved the data in a pickle file: `imdb_data.pkl`. \n",
    "The needed file can be downloaded from [https://s3-us-west-1.amazonaws.com/nervana-course/imdb_data.pkl](https://s3-us-west-1.amazonaws.com/nervana-course/imdb_data.pkl) and placed in the `data` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle as pkl\n",
    "\n",
    "data = pkl.load(open('data/imdb_data.pkl', 'rb'), encoding='latin1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `data` dictionary contains four numpy arrays for the data:\n",
    "\n",
    "1. `data['X_train']` is an array with shape (20009, 128) for 20009 example reviews, each with up to 128 words.\n",
    "2. `data['Y_train']` is an array with shape (20009, 1) with a target label (positive=1, negative=0) for each review.\n",
    "3. `data['X_valid']` is an array with shape (4991, 128) for the 4991 examples in the test set.\n",
    "4. `data['Y_valid']` is an array with shape (4991, 1) for the 4991 examples in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(20009, 128)\n"
     ]
    }
   ],
   "source": [
    "print(data['X_train'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute backend\n",
    "---------------\n",
    "\n",
    "We first generate a backend to tell neon what hardware to run the model on. This is shared by all neon objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.backends import gen_backend\n",
    "\n",
    "be = gen_backend(backend='mkl', batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the model, we use neon's `ArrayIterator` object which will iterate over these numpy arrays, returning a minibatch of data with each call to pass to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.data import ArrayIterator\n",
    "import numpy as np\n",
    "data['Y_train'] = np.array(data['Y_train'], dtype=np.int32)\n",
    "data['Y_valid'] = np.array(data['Y_valid'], dtype=np.int32)\n",
    "\n",
    "train_set = ArrayIterator(data['X_train'], data['Y_train'], nclass=2)\n",
    "valid_set = ArrayIterator(data['X_valid'], data['Y_valid'], nclass=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model Specification\n",
    "-------------------\n",
    "\n",
    "\n",
    "\n",
    "For most of the layers, we randomly initialize the parameters either randomly uniform numbers or Xavier Glorot's initialization scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Uniform, GlorotUniform\n",
    "\n",
    "init_glorot = GlorotUniform()\n",
    "init_uniform = Uniform(-0.1/128, 0.1/128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network consists of sequential list of the following layers:\n",
    "\n",
    "1. `LookupTable` is a word embedding that maps from a sparse one-hot representation to dense word vectors. The embedding is learned from the data.\n",
    "2. `LSTM` is a recurrent layer with “long short-term memory” units. LSTM networks are good at learning temporal dependencies during training, and often perform better than standard RNN layers.\n",
    "3. `RecurrentSum` is a recurrent output layer that collapses over the time dimension of the LSTM by summing outputs from individual steps.\n",
    "4. `Dropout` performs regularization by silencing a random subset of the units during training.\n",
    "5. `Affine` is a fully connected layer for the binary classification of the outputs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-10-fc3beca488e0>, line 14)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-10-fc3beca488e0>\"\u001b[0;36m, line \u001b[0;32m14\u001b[0m\n\u001b[0;31m    ...\u001b[0m\n\u001b[0m      ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "from neon.layers import LSTM, Affine, Dropout, LookupTable, RecurrentSum\n",
    "from neon.transforms import Logistic, Tanh, Softmax\n",
    "from neon.models import Model\n",
    "\n",
    "layers = [\n",
    "    LookupTable(vocab_size=20000, embedding_dim=128, init=init_uniform),\n",
    "    LSTM(output_size=128, init=init_glorot, activation=Tanh(),\n",
    "         gate_activation=Logistic(), reset_cells=True),\n",
    "    \n",
    "    # Add a recurrent sum layer\n",
    "    ...\n",
    "    \n",
    "    # Add dropout (50%)\n",
    "    ...\n",
    "    \n",
    "    # Add an affine layer with two outputs using the Xavier Glorot's initialization scheme\n",
    "    ...\n",
    "]\n",
    "\n",
    "# create model object\n",
    "model = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cost, Optimizers, and Callbacks\n",
    "-------------------------------\n",
    "For training, we use the Adagrad optimizer and the Cross Entropy cost function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.optimizers import Adagrad\n",
    "from neon.transforms import CrossEntropyMulti\n",
    "from neon.layers import GeneralizedCost\n",
    "\n",
    "cost = GeneralizedCost(costfunc=CrossEntropyMulti(usebits=True))\n",
    "optimizer = Adagrad(learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks allow the model to report its progress during the course of training. Here we tell neon to save the model every epoch ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.callbacks import Callbacks\n",
    "\n",
    "model_file = 'imdb_lstm.pkl'\n",
    "callbacks = Callbacks(model, eval_set=valid_set, serialize=1, save_path=model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train model\n",
    "-----------\n",
    "\n",
    "To train the model, we call the `fit()` function and pass in the training set. Here we train for 2 epochs, meaning two complete passes through the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# call the model.fit() function\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy\n",
    "--------\n",
    "\n",
    "We can then measure the model's accuracy on both the training set but also the held-out validation set.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.transforms import Accuracy\n",
    "\n",
    "print(\"Test  Accuracy - {}\".format(100 * model.eval(valid_set, metric=Accuracy())))\n",
    "\n",
    "# similar to the above evaluation of the model's accuracy on the held-out validation set, \n",
    "# evaluate the model's accuracy on the training set, 'train_set':\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tuning\n",
    "--------\n",
    "\n",
    "Now that you have run an LSTM that can analyze the sentiment of IMDB movie reviews, your task is to tune the model architecture and hyperparameters to increase its accuracy. Make sure not to overfit the training dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}