Exercise+2a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore this ipython notebook, press `SHIFT+ENTER` to progress to the next cell. Feel free to make changes, enter code, and hack around. You can create new code cells by selecting `INSERT->Insert Cell Below`\n",
    "\n",
    "# MNIST Exercise (Part 2)\n",
    "\n",
    "MNIST is a computer vision dataset consisting of 70,000 images of handwritten digits. Each image has 28x28 pixels for a total of 784 features, and is associated with a digit between 0-9.\n",
    "\n",
    "<img src=\"http://corpocrat.com/wp-content/uploads/2014/10/figure_1.png\" width=200px>\n",
    "\n",
    "\n",
    "In the last exercise, you constructed a multi-layer perceptron (also called softmax regression) to recognize each image. In this exercise, you will tune the model's hyperparameters to decrease the misclassification error. Note that this exercise assumes some basic familiarity with python and machine learning.\n",
    "\n",
    "This tutorial is similar to the model specified in `examples/mnist_mlp.py`.\n",
    "\n",
    "## Setup\n",
    "\n",
    "This example works with Python 2.7. The urllib request method needs to be changed in the inference steps for Python 3.x.\n",
    "\n",
    "Your environment needs to have the following packages installed:\n",
    "- neon v2.0.0\n",
    "- matplotlib (for the inference)\n",
    "\n",
    "## Preamble\n",
    "The first step is to set up our compute backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.backends import gen_backend\n",
    "\n",
    "batch_size = 32\n",
    "\n",
    "be = gen_backend(batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The MNIST dataset can be found on Yann LeCunn’s website. We have included an easy function that downloads the MNIST dataset into your `~/nervana/data/` directory and loads it into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<neon.data.dataiterator.ArrayIterator at 0x7fcad5a07080>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from neon.data import MNIST\n",
    "\n",
    "mnist = MNIST(path='data/')\n",
    "train_set = mnist.train_iter\n",
    "valid_set = mnist.valid_iter\n",
    "valid_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During training, neon iterates over the training examples to compute the gradients. The `train_iter` and `valid_iter` handle sending data to the model for training and validation, respectively.\n",
    "\n",
    "For small datasets like MNIST, this step may seem trivial. However, for large datasets that cannot fit into memory (e.g. ImageNet or Sports-1M), the data has to be efficiently loaded and fed to the optimizer in batches. This requires more advanced iterators described in Loading data.\n",
    "\n",
    "## Model specification\n",
    "Training a deep learning model in Neon requires \n",
    "* Specifying the dataset\n",
    "* Building a model from a list of layers\n",
    "* Cost function\n",
    "* Learning rule. \n",
    "\n",
    "Here we guide you through each item in turn.\n",
    "\n",
    "### Initializing weights\n",
    "Neon supports many ways of initializing weight matrices. In this tutorial, we initialize the weights using a Gaussian distribution with zero mean and 0.01 standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<neon.initializers.initializer.Gaussian at 0x7fcadc6b4400>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from neon.initializers import Gaussian\n",
    "\n",
    "init_norm = Gaussian(loc=0.0, scale=0.01)\n",
    "init_norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model architecture\n",
    "The model is specified as a list of layers. For classifying MNIST images, we use a multi-layer perceptron with fully connected layers.\n",
    "\n",
    "- Affine (i.e. fully-connected) layer made up of hidden units and a rectified linear activation function, defined as Rectlin().\n",
    "- An output layer with 10 units to match the number of labels in the MNIST dataset. We use the Softmax() activation function to ensure the outputs sum to one and are within the range [0,1].\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[<neon.layers.layer.Linear at 0x7fcad5a07dd8>,\n",
       "  <neon.layers.layer.Activation at 0x7fcad5a07da0>],\n",
       " [<neon.layers.layer.Linear at 0x7fcad5a07eb8>,\n",
       "  <neon.layers.layer.Activation at 0x7fcad5a11208>]]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from neon.layers import Affine\n",
    "from neon.transforms import Rectlin, Softmax\n",
    "\n",
    "layers = []\n",
    "layers.append(Affine(nout=10, init=init_norm, activation=Rectlin()))\n",
    "layers.append(Affine(nout=10, init=init_norm,\n",
    "                     activation=Softmax()))\n",
    "layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize the weights in each layer with the init_norm defined previously. Neon supports many other layer types (convolutional, pooling, recurrent, etc.) that will be described in subsequent examples. We then construct the model via"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initialize model object\n",
    "from neon.models import Model\n",
    "\n",
    "mlp = Model(layers=layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Costs\n",
    "The cost function is wrapped within a GeneralizedCost layer, which handles the comparison of the outputs with the provided labels in the dataset. One common cost function which we use here is the cross entropy loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.layers import GeneralizedCost\n",
    "from neon.transforms import CrossEntropyMulti\n",
    "\n",
    "cost = GeneralizedCost(costfunc=CrossEntropyMulti())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning rules\n",
    "For learning, we use stochastic gradient descent with a learning rate of 0.1 and momentum coefficient of 0.9.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.optimizers import GradientDescentMomentum\n",
    "\n",
    "optimizer = GradientDescentMomentum(0.1, momentum_coef=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks\n",
    "Neon provides an API for calling operations during the model fit (see Callbacks). Here we set up the default callback, which is displaying a progress bar for each epoch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.callbacks.callbacks import Callbacks\n",
    "\n",
    "callbacks = Callbacks(mlp, eval_set=valid_set)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting it all together\n",
    "We are ready to put all the ingredients together and run our model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0   [Train |                    |    9/1875 batches, 2.30 cost, 0.20s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/nervananeon-2.6.0-py3.6.egg/neon/backends/nervanacpu.py:681: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n",
      "  array_output[numpy_ind.tolist()] = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0   [Train |████████████████████| 1875/1875 batches, 0.72 cost, 69.60s]\n",
      "Epoch 1   [Train |████████████████████| 1875/1875 batches, 0.45 cost, 81.10s]\n",
      "Epoch 2   [Train |████████████████████| 1875/1875 batches, 0.45 cost, 83.59s]\n",
      "Epoch 3   [Train |████████████████████| 1875/1875 batches, 0.35 cost, 76.50s]\n",
      "Epoch 4   [Train |████████████████████| 1875/1875 batches, 0.38 cost, 86.71s]\n",
      "Epoch 5   [Train |████████████████████| 1875/1875 batches, 0.33 cost, 84.71s]\n",
      "Epoch 6   [Train |████████████████████| 1875/1875 batches, 0.36 cost, 81.09s]\n",
      "Epoch 7   [Train |████████████████████| 1875/1875 batches, 0.31 cost, 82.20s]\n",
      "Epoch 8   [Train |████████████████████| 1875/1875 batches, 0.35 cost, 80.79s]\n",
      "Epoch 9   [Train |████████████████████| 1875/1875 batches, 0.28 cost, 110.60s]\n"
     ]
    }
   ],
   "source": [
    "mlp.fit(train_set, optimizer=optimizer, num_epochs=10, cost=cost,\n",
    "        callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the beginning of the fitting procedure, neon propagates train_set through the model to set the input and output shapes of each layer. Each layer has a `configure()` method that determines the appropriate layer shapes, and an `allocate()` method to set up the needed buffers for holding the forward propagation information.\n",
    "\n",
    "During the training, neon sends batches of the training data through the model, calling each layers’ `fprop()` and `bprop()` methods to compute the gradients and update the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the trained model\n",
    "Now that the model is successfully trained, we can use the trained model to classify a novel image, measure performance, and visualize the weights and training results.\n",
    "\n",
    "#### Get outputs\n",
    "Given a set of images such as those contained in the iterable `valid_set`, we can fetch the ouput of the final model layer via\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/nervananeon-2.6.0-py3.6.egg/neon/backends/nervanacpu.py:681: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n",
      "  array_output[numpy_ind.tolist()] = 1\n"
     ]
    }
   ],
   "source": [
    "results = mlp.get_outputs(valid_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable results is a numpy array with shape `(num_test_examples, num_outputs) = (10000,10)` with the model probabilities for each label.\n",
    "\n",
    "#### Performance\n",
    "Neon supports convenience functions for evaluating performance using custom metrics. Here we measure the misclassification rate on the held out test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/nervananeon-2.6.0-py3.6.egg/neon/backends/nervanacpu.py:681: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n",
      "  array_output[numpy_ind.tolist()] = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Misclassification error = 12.0%\n"
     ]
    }
   ],
   "source": [
    "from neon.transforms import Misclassification\n",
    "\n",
    "# evaluate the model on test_set using the misclassification metric\n",
    "error = mlp.eval(valid_set, metric=Misclassification())*100\n",
    "print('Misclassification error = %.1f%%' % error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning\n",
    "\n",
    "Now that you have run a multi-layer perceptron that can recognize MNIST images, your task is to tune the model architecture and hyperparameters, including the batch size and learning rate, to decrease the misclassification error. Aim to get a misclassification error of 3% or lower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference\n",
    "\n",
    "Now let's download a new digit image from the web, and use our trained model to recognize the digit. We first download the image and scale it to the 28x28 pixels that our model expects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "\n",
    "# scale to 28x28 pixels\n",
    "img = Image.open(\"data/digit.jpg\")\n",
    "img.thumbnail((28, 28))\n",
    "\n",
    "digit = np.asarray(img, dtype=np.float32)[:, :, 0]\n",
    "\n",
    "# reshape to a single feature vector\n",
    "digit = digit.reshape(784, 1)\n",
    "\n",
    "# store digit into a GPU tensor\n",
    "x_new = be.zeros((28*28, batch_size), dtype=np.float32)\n",
    "x_new[:, 0] = digit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model final layer was: [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "The most probable guess is digit: 4\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f8af60f3940>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADytJREFUeJzt3WuMVHWax/HfQ19EYLjoAEMcVhCJiBidtSUkbhAljrIh\nwXkxOL7YsNnJtC/Q7CRjsmpMMNlMYjY7M+uLzSSwkMEEnJnEC7wwI0bNuiarES8BR9wdg9yWtpuL\nXFqQpquffdGHSQ/2+Z+y63IKnu8nMVV1njpVD2X/+tTp/znnb+4uAPGMK7sBAOUg/EBQhB8IivAD\nQRF+ICjCDwRF+IGgCD8QFOEHgmpv5puNGzfO29vz37JSqTSxG+DyMzQ0JHe3ap5bU/jN7D5Jz0hq\nk/Qf7v508s3a2zVjxozc+unTp2tpBwivv7+/6ueO+Wu/mbVJ+ndJKyQtlPSgmS0c6+sBaK5a9vkX\nS/rU3fe6+4Ck30paVZ+2ADRaLeG/RtLBEY8PZcv+gpl1m9lOM9s5NDRUw9sBqKdawj/aHxW+dn6w\nu6939y537xo3jsEFoFXUksZDkmaPePxdSYdrawdAs9QS/nclzTezuWbWKelHkrbXpy0AjTbmoT53\nHzSzhyW9ouGhvk3u/sfUOpVKRadOncqtM9QHNI818zJebW1tPnHixNw64QdqV+1BPvwFDgiK8ANB\nEX4gKMIPBEX4gaAIPxBUU8/nlySzqkYhADQYW34gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBThB4Ii/EBQTT+lF/WXmglp8uTJyXXPnz+frJ85cyZZb+bVn1FfbPmBoAg/\nEBThB4Ii/EBQhB8IivADQRF+IKiaxvnNbJ+k05IqkgbdvaseTeGbmTFjRm7tySefTK67e/fuZH3r\n1q3JeiNnVu7o6EjWi45hGBgYyK319/cn141w/EI9DvK5y92P1uF1ADQRX/uBoGoNv0vaYWbvmVl3\nPRoC0By1fu2/w90Pm9kMSa+a2Sfu/ubIJ2S/FLqz+zW+HYB6qWnL7+6Hs9s+SS9KWjzKc9a7e5e7\ndxF+oHWMOfxmNtHMvnXhvqTvS/qoXo0BaKxavvbPlPRitjVvl7TV3f9Ql64ANNyYw+/ueyXdUsde\nkCN1vr4kLVq0KLd25513JtetVCrJemdnZ7LeSPPmzUvWH3/88WT9gw8+yK1t3LgxuW4jj19oFQz1\nAUERfiAowg8ERfiBoAg/EBThB4Li0t2XgKlTpybrK1euzK1NmzYtue6xY8eS9cHBwWS9FkVHfBb1\nvmTJkmS9vT3/x3vLli3JdRnqA3DZIvxAUIQfCIrwA0ERfiAowg8ERfiBoBjnbwGp8WhJuu2225L1\npUuX5taGhoaS6+7duzdZP3fuXLJei/HjxyfrCxYsqGn9CJffrgVbfiAowg8ERfiBoAg/EBThB4Ii\n/EBQhB8IinH+FjBnzpxk/ZFHHknWU5e4fumll5LrvvXWW8l6meP8N910U7JedD2A1DEMjfx3XSrY\n8gNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIXj/Ga2SdJKSX3uvihbdpWk30maI2mfpNXu/kXj2ry0\nFY1nr1ixIlm//fbbk/Xjx4/n1nbs2JFc9/PPP0/WG3lOfNHncuONN9b0+qlx/oGBgZpe+3JQzZb/\nN5Luu2jZY5Jec/f5kl7LHgO4hBSG393flHTxpmWVpM3Z/c2S7q9zXwAabKz7/DPdvUeSstsZ9WsJ\nQDM0/Nh+M+uW1J3db/TbAajSWLf8vWY2S5Ky2768J7r7enfvcvcuwg+0jrGGf7ukNdn9NZK21acd\nAM1SGH4ze07Sf0u6wcwOmdmPJT0t6R4z+5Oke7LHAC4hhfv87v5gTml5nXu5bBWdl7569epkfdy4\n9O/oDRs25NaKxvm/+uqrZL2ROjs7k/VJkyYl60Xn5O/atWvM60bAEX5AUIQfCIrwA0ERfiAowg8E\nRfiBoLh0d5VSRyemLp0tSevWrUvWb7nllmT9jTfeSNa3bcs/xurIkSPJdcs0ffr0ZH3mzJk1vf4X\nX+SfZc703Wz5gbAIPxAU4QeCIvxAUIQfCIrwA0ERfiAoxvmrdPXVV+fW1q5dm1z3rrvuStYHBweT\n9ddffz1ZP3XqVG6to6Mjue7Q0FCyXqlUkvVaFI3zT5kyJVk/e/Zsss5YfhpbfiAowg8ERfiBoAg/\nEBThB4Ii/EBQhB8IinH+KqXG+YvG8YsuQX3+/Plk/aGHHkrWb7755tzaK6+8klz3xIkTyfrJkyeT\n9Z6enmQ99W9ftmxZct0JEyYk68eOHUvWGedPY8sPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0EVjvOb\n2SZJKyX1ufuibNlTkn4i6cJF4Z9w95cb1WQrGBgYyK0VXRv/9OnTyfoVV1yRrM+fPz9Znz17dm5t\n1apVyXWLjjE4evRosr5v375kffLkybm1uXPnJtctutbA22+/nax/+eWXyXp01Wz5fyPpvlGW/8rd\nb83+u6yDD1yOCsPv7m9KOt6EXgA0US37/A+b2S4z22Rm0+rWEYCmGGv4fy1pnqRbJfVI+kXeE82s\n28x2mtlOjrUGWseYwu/uve5ecfchSRskLU48d727d7l7V2qySwDNNabwm9msEQ9/IOmj+rQDoFmq\nGep7TtIySd82s0OS1klaZma3SnJJ+ySlzzkF0HKsmfvhbW1tnjq/O3X9+bJdeeWVubXly5cn1126\ndGmyfv311yfrRee1p44TuO6665LrpsbhpeI5Bc6cOZOsp/5/F733/v37k/VHH300WX/55fwR6NRx\nG5c6d69q/5oj/ICgCD8QFOEHgiL8QFCEHwiK8ANBMdRXB52dncn6+PHja1q/6MjI1OsvWbIkue4N\nN9yQrBf9fBRd+vvuu+/Ord17773JdT/++ONk/YEHHkjWP/vss2T9csVQH4Akwg8ERfiBoAg/EBTh\nB4Ii/EBQhB8Iiim666Do9NAyTx/t7e1N1js6OpL1onH+otNyFyxYkFsr+lz6+vqS9bNnzybrSGPL\nDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc5/mWv0MQipS5pL0pQpU8b83p988kmyfu7cuWQdaWz5\ngaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCownF+M5st6VlJ35E0JGm9uz9jZldJ+p2kOZL2SVrt7l80\nrlW0oqlTpybrqenHi64lcPLkyWS9Uqkk60irZss/KOln7n6jpCWS1prZQkmPSXrN3edLei17DOAS\nURh+d+9x9/ez+6cl7ZF0jaRVkjZnT9ss6f5GNQmg/r7RPr+ZzZH0PUnvSJrp7j3S8C8ISTPq3RyA\nxqn62H4zmyTpeUk/dfdTRfPHjVivW1J3dn8sPQJogKq2/GbWoeHgb3H3F7LFvWY2K6vPkjTq1Rbd\nfb27d7l7F+EHWkdh+G04sRsl7XH3X44obZe0Jru/RtK2+rcHoFGq+dp/h6S/k7TbzD7Mlj0h6WlJ\nvzezH0s6IOmHjWkRZWpvT/+ILFy4MFm/9tprc2tFQ3V79+5N1jmltzaF4Xf3tyTlfV9fXt92ADQL\nR/gBQRF+ICjCDwRF+IGgCD8QFOEHguLS3Uhqa2tL1lOX5pbSp+2eOXMmue7BgweT9fPnzyfrSGPL\nDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc6PpKKx9AMHDiTrvb29ubX9+/fX9NpDQ0PJOtLY8gNB\nEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzI6loLH3Pnj3J+tatW3NrRdflP3LkSLKO2rDlB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgzN3TTzCbLelZSd+RNCRpvbs/Y2ZPSfqJpAuDsU+4+8up12pra/NJ\nkybl1k+dOlV952gJZnmztw8bP358bq1SqSTXLbqWQNHPblTunv6fkqnmIJ9BST9z9/fN7FuS3jOz\nV7Par9z9X8faJIDyFIbf3Xsk9WT3T5vZHknXNLoxAI31jfb5zWyOpO9Jeidb9LCZ7TKzTWY2LWed\nbjPbaWY7+ZoGtI7Cff4/P9FskqT/lPRzd3/BzGZKOirJJf2zpFnu/g+p12Cf//LDPn/rqXafv6ot\nv5l1SHpe0hZ3fyF7g153r7j7kKQNkhaPtVkAzVcYfhv+1b5R0h53/+WI5bNGPO0Hkj6qf3sAGqWa\nob6/kfRfknZreKhPkp6Q9KCkWzX8tX+fpIeyPw7mamtr8wkTJuTW+/v7q+0bQI5qv/ZXvc9fD4Qf\naLy67vMDuPwQfiAowg8ERfiBoAg/EBThB4Jq6qW729vbNX369Nx66lBQAMVOnDhR9XPZ8gNBEX4g\nKMIPBEX4gaAIPxAU4QeCIvxAUE09pdfMjkjaP2LRtzV8KbBW1Kq9tWpfEr2NVT17u9bd8w+mGaGp\n4f/amw9f1LOrtAYSWrW3Vu1LorexKqs3vvYDQRF+IKiyw7++5PdPadXeWrUvid7GqpTeSt3nB1Ce\nsrf8AEpSSvjN7D4z+x8z+9TMHiujhzxmts/MdpvZh2a2s+ReNplZn5l9NGLZVWb2qpn9KbsddZq0\nknp7ysz+L/vsPjSzvy2pt9lm9oaZ7TGzP5rZP2bLS/3sEn2V8rk1/Wu/mbVJ+l9J90g6JOldSQ+6\n+8dNbSSHme2T1OXupY8Jm9lSSf2SnnX3Rdmyf5F03N2fzn5xTnP3f2qR3p6S1F/2zM3ZhDKzRs4s\nLel+SX+vEj+7RF+rVcLnVsaWf7GkT919r7sPSPqtpFUl9NHy3P1NSccvWrxK0ubs/mYN//A0XU5v\nLcHde9z9/ez+aUkXZpYu9bNL9FWKMsJ/jaSDIx4fUmtN+e2SdpjZe2bWXXYzo5h5YWak7HZGyf1c\nrHDm5ma6aGbplvnsxjLjdb2VEf7RZhNppSGHO9z9ryWtkLQ2+3qL6vxa0jwNT+PWI+kXZTaTzSz9\nvKSfunvLTAE9Sl+lfG5lhP+QpNkjHn9X0uES+hiVux/ObvskvajWm32498IkqdltX8n9/Fkrzdw8\n2szSaoHPrpVmvC4j/O9Kmm9mc82sU9KPJG0voY+vMbOJ2R9iZGYTJX1frTf78HZJa7L7ayRtK7GX\nv9AqMzfnzSytkj+7VpvxupSDfLKhjH+T1CZpk7v/vOlNjMLMrtPw1l4avrLx1jJ7M7PnJC3T8Flf\nvZLWSXpJ0u8l/ZWkA5J+6O5N/8NbTm/L9A1nbm5Qb3kzS7+jEj+7es54XZd+OMIPiIkj/ICgCD8Q\nFOEHgiL8QFCEHwiK8ANBEX4gKMIPBPX/1m6zZ1Mh0ZcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8b04b24be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# forward pass through the model\n",
    "outputs = mlp.fprop(x_new)\n",
    "outputs = outputs.get()[:, 0]\n",
    "\n",
    "# examine the output of the model for this image\n",
    "print(\"Model final layer was: {}\".format(outputs))\n",
    "print(\"The most probable guess is digit: {}\".format(np.argmax(outputs)))\n",
    "plt.figure(2)\n",
    "plt.imshow(img)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}