tutorial/30.lin.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Linear models, loss functions, gradients, SGD\n",
    "(c) Deniz Yuret, 2019\n",
    "* Objectives: Define, train and visualize a simple model; understand gradients and SGD; learn to use the GPU.\n",
    "* Prerequisites: [Callable objects](https://docs.julialang.org/en/v1/manual/methods/#Function-like-objects-1), [Generator expressions](https://docs.julialang.org/en/v1/manual/arrays/#Generator-Expressions-1), [MNIST](20.mnist.ipynb), [Iterators](25.iterators.ipynb)\n",
    "* New functions: \n",
    "[mnistdata](https://github.com/denizyuret/Knet.jl/blob/master/data/mnist.jl),\n",
    "[accuracy](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.accuracy), \n",
    "[zeroone](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.zeroone), \n",
    "[nll](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.nll), \n",
    "[Param, @diff, value, params, grad](http://denizyuret.github.io/Knet.jl/latest/reference/#AutoGrad),\n",
    "[sgd](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.sgd),\n",
    "[progress, progress!](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.progress), \n",
    "[gpu](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.gpu), \n",
    "[KnetArray](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.KnetArray), \n",
    "[load](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.load), \n",
    "[save](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.save)\n",
    "\n",
    "\n",
    "<img src=\"https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/assets/tfdl_0401.png\" alt=\"A linear model\" width=300/> ([image source](https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/ch04.html))\n",
    "\n",
    "In Knet, a machine learning model is defined using plain Julia code. A typical model consists of a **prediction** and a **loss** function. The prediction function takes some input, returns the prediction of the model for that input. The loss function measures how bad the prediction is with respect to some desired output. We train a model by adjusting its parameters to reduce the loss.\n",
    "\n",
    "In this section we will implement a simple linear model to classify MNIST digits. The prediction function will return 10 scores for each of the possible labels 0..9 as a linear combination of the pixel values. The loss function will convert these scores to normalized probabilities and return the average -log probability of the correct answers. Minimizing this loss should maximize the scores assigned to correct answers by the model. We will make use of the loss gradient with respect to each parameter, which tells us the direction of the greatest loss increase. We will improve the model by moving the parameters in the opposite direction (using a GPU if available). We will visualize the model weights and performance over time. The final accuracy of about 92% is close to the limit of what we can achieve with this type of model. To improve further we must look beyond linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Set display width, load packages, import symbols\n",
    "ENV[\"COLUMNS\"]=72\n",
    "using Pkg; for p in (\"Knet\",\"AutoGrad\",\"IterTools\",\"Plots\",\"Images\",\"ImageMagick\"); haskey(Pkg.installed(),p) || Pkg.add(p); end\n",
    "using Statistics: mean\n",
    "using Base.Iterators: flatten\n",
    "using IterTools: ncycle, takenth\n",
    "import Random # seed!\n",
    "using Knet: Knet, AutoGrad, dir, Data, Param, @diff, value, params, grad, progress, progress!, gpu, KnetArray, load, save\n",
    "# The following are defined for instruction even though they are provided in Knet\n",
    "# using Knet: accuracy, zeroone, nll, sgd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Loading MNIST...\n",
      "└ @ Main /kuacc/users/dyuret/.julia/dev/Knet/data/mnist.jl:33\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "600-element Data{Tuple{Array{Float32,2},Array{UInt8,1}}}\n",
      "100-element Data{Tuple{Array{Float32,2},Array{UInt8,1}}}\n"
     ]
    }
   ],
   "source": [
    "# Load data (mnistdata basically replicates mnist.ipynb)\n",
    "include(Knet.dir(\"data\",\"mnist.jl\"))\n",
    "dtrn,dtst = mnistdata(xsize=(784,:),xtype=Array)\n",
    "println.(summary.((dtrn,dtst)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear([-0.00930764 -1.73228e-5 … 0.00286358 0.00338102; -0.0154502 -0.00492697 … -0.00578297 -0.0168118; … ; -0.0135301 -0.0130993 … -0.0100258 -0.0010083; -0.0224418 0.00547202 … -0.00505807 0.00594477], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In Julia we define a new datatype using `struct`:\n",
    "struct Linear; w; b; end\n",
    "\n",
    "# The new struct comes with a default constructor:\n",
    "model = Linear(0.01 * randn(10,784), zeros(10))\n",
    "\n",
    "# We can define other constructors with different inputs:\n",
    "Linear(i::Int,o::Int,scale=0.01) = Linear(scale * randn(o,i), zeros(o))\n",
    "\n",
    "# This one allows instances to be defined using input and output sizes:\n",
    "model = Linear(784,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# We turn Linear instances into callable objects for prediction:\n",
    "(m::Linear)(x) = m.w * x .+ m.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"784×100 Array{Float32,2}\", \"100-element Array{UInt8,1}\")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,y = first(dtst) # The first minibatch from the test set\n",
    "summary.((x,y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×100 LinearAlgebra.Adjoint{Int64,Array{Int64,1}}:\n",
       " 7  2  1  10  4  1  4  9  5  9  …  1  3  6  9  3  1  4  1  7  6  9"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Int.(y)' # correct answers are given as an array of integers (remember we use 10 for 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10×100 Array{Float64,2}:\n",
       " -0.00410484   0.021517    0.0135085  …  -0.0142226   -0.0172761  \n",
       " -0.0240913    0.0335927  -0.0706286      0.0791179    0.0736431  \n",
       "  0.0932138   -0.057978   -0.0536457      0.105226    -0.0364376  \n",
       " -0.0585087    0.164165    0.0140825     -0.061517    -0.000969126\n",
       "  0.00974735   0.105798   -0.0116878     -0.00833981   0.109366   \n",
       "  0.0321055    0.0523192   0.107068   …   0.127174     0.134652   \n",
       " -0.11468     -0.0669171  -0.107339      -0.0710706   -0.16536    \n",
       " -0.0751655   -0.108038    0.0747432     -0.0961626   -0.0103486  \n",
       "  0.117249     0.0954578   0.074709       0.0766856    0.294205   \n",
       " -0.0574604    0.0391589  -0.0471648      0.0195734   -0.0645414  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ypred = model(x)  # Predictions on the first minibatch: a 10x100 score matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can calculate the accuracy of our model for the first minibatch\n",
    "accuracy(model,x,y) = mean(y' .== map(i->i[1], findmax(Array(model(x)),dims=1)[2]))\n",
    "accuracy(model,x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10329999999999999"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can calculate the accuracy of our model for the whole test set\n",
    "accuracy(model,data) = mean(accuracy(model,x,y) for (x,y) in data)\n",
    "accuracy(model,dtst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8967"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ZeroOne loss (or error) is defined as 1 - accuracy\n",
    "zeroone(x...) = 1 - accuracy(x...)\n",
    "zeroone(model,dtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nll (generic function with 1 method)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For classification we use negative log likelihood loss (aka cross entropy, softmax loss, NLL)\n",
    "# This is the average -log probability assigned to correct answers by the model\n",
    "function nll(scores, y)\n",
    "    expscores = exp.(scores)\n",
    "    probabilities = expscores ./ sum(expscores, dims=1)\n",
    "    answerprobs = (probabilities[y[i],i] for i in 1:length(y))\n",
    "    mean(-log.(answerprobs))\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.308938079912147"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model(x) gives predictions, let model(x,y) give the loss\n",
    "(m::Linear)(x, y) = nll(m(x), y)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.308938079912147"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also use the Knet nll implementation for efficiency\n",
    "(m::Linear)(x, y) = Knet.nll(m(x), y)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# If the input is a dataset compute average loss:\n",
    "(m::Linear)(data::Data) = mean(m(x,y) for (x,y) in data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.3096749652904793"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is per-instance average negative log likelihood for the whole test set\n",
    "model(dtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus question:** What is special about the loss value 2.3?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Calculating the gradient using AutoGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "Usage:\n",
       "\n",
       "\\begin{verbatim}\n",
       "x = Param([1,2,3])          # user declares parameters with `Param`\n",
       "x => P([1,2,3])             # `Param` is just a struct wrapping a value\n",
       "value(x) => [1,2,3]         # `value` returns the thing wrapped\n",
       "sum(x .* x) => 14           # Params act like regular values\n",
       "y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\n",
       "y => T(14)                  # you get another struct\n",
       "value(y) => 14              # which carries the same result\n",
       "params(y) => [x]            # and the Params that it depends on \n",
       "grad(y,x) => [2,4,6]        # and the gradients for all Params\n",
       "\\end{verbatim}\n",
       "\\texttt{Param(x)} returns a struct that acts like \\texttt{x} but marks it as a parameter you want to compute gradients with respect to.\n",
       "\n",
       "\\texttt{@diff expr} evaluates an expression and returns a struct that contains the result (which should be a scalar) and gradient information.\n",
       "\n",
       "\\texttt{grad(y, x)} returns the gradient of \\texttt{y} (output by @diff) with respect to any parameter \\texttt{x::Param}, or  \\texttt{nothing} if the gradient is 0.\n",
       "\n",
       "\\texttt{value(x)} returns the value associated with \\texttt{x} if \\texttt{x} is a \\texttt{Param} or the output of \\texttt{@diff}, otherwise returns \\texttt{x}.\n",
       "\n",
       "\\texttt{params(x)} returns an iterator of Params found by a recursive search of object \\texttt{x}.\n",
       "\n",
       "Alternative usage:\n",
       "\n",
       "\\begin{verbatim}\n",
       "x = [1 2 3]\n",
       "f(x) = sum(x .* x)\n",
       "f(x) => 14\n",
       "grad(f)(x) => [2 4 6]\n",
       "gradloss(f)(x) => ([2 4 6], 14)\n",
       "\\end{verbatim}\n",
       "Given a scalar valued function \\texttt{f}, \\texttt{grad(f,argnum=1)} returns another function \\texttt{g} which takes the same inputs as \\texttt{f} and returns the gradient of the output with respect to the argnum'th argument. \\texttt{gradloss} is similar except the resulting function also returns f's output.\n",
       "\n"
      ],
      "text/markdown": [
       "Usage:\n",
       "\n",
       "```\n",
       "x = Param([1,2,3])          # user declares parameters with `Param`\n",
       "x => P([1,2,3])             # `Param` is just a struct wrapping a value\n",
       "value(x) => [1,2,3]         # `value` returns the thing wrapped\n",
       "sum(x .* x) => 14           # Params act like regular values\n",
       "y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\n",
       "y => T(14)                  # you get another struct\n",
       "value(y) => 14              # which carries the same result\n",
       "params(y) => [x]            # and the Params that it depends on \n",
       "grad(y,x) => [2,4,6]        # and the gradients for all Params\n",
       "```\n",
       "\n",
       "`Param(x)` returns a struct that acts like `x` but marks it as a parameter you want to compute gradients with respect to.\n",
       "\n",
       "`@diff expr` evaluates an expression and returns a struct that contains the result (which should be a scalar) and gradient information.\n",
       "\n",
       "`grad(y, x)` returns the gradient of `y` (output by @diff) with respect to any parameter `x::Param`, or  `nothing` if the gradient is 0.\n",
       "\n",
       "`value(x)` returns the value associated with `x` if `x` is a `Param` or the output of `@diff`, otherwise returns `x`.\n",
       "\n",
       "`params(x)` returns an iterator of Params found by a recursive search of object `x`.\n",
       "\n",
       "Alternative usage:\n",
       "\n",
       "```\n",
       "x = [1 2 3]\n",
       "f(x) = sum(x .* x)\n",
       "f(x) => 14\n",
       "grad(f)(x) => [2 4 6]\n",
       "gradloss(f)(x) => ([2 4 6], 14)\n",
       "```\n",
       "\n",
       "Given a scalar valued function `f`, `grad(f,argnum=1)` returns another function `g` which takes the same inputs as `f` and returns the gradient of the output with respect to the argnum'th argument. `gradloss` is similar except the resulting function also returns f's output.\n"
      ],
      "text/plain": [
       "  Usage:\n",
       "\n",
       "\u001b[36m  x = Param([1,2,3])          # user declares parameters with `Param`\u001b[39m\n",
       "\u001b[36m  x => P([1,2,3])             # `Param` is just a struct wrapping a value\u001b[39m\n",
       "\u001b[36m  value(x) => [1,2,3]         # `value` returns the thing wrapped\u001b[39m\n",
       "\u001b[36m  sum(x .* x) => 14           # Params act like regular values\u001b[39m\n",
       "\u001b[36m  y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\u001b[39m\n",
       "\u001b[36m  y => T(14)                  # you get another struct\u001b[39m\n",
       "\u001b[36m  value(y) => 14              # which carries the same result\u001b[39m\n",
       "\u001b[36m  params(y) => [x]            # and the Params that it depends on \u001b[39m\n",
       "\u001b[36m  grad(y,x) => [2,4,6]        # and the gradients for all Params\u001b[39m\n",
       "\n",
       "  \u001b[36mParam(x)\u001b[39m returns a struct that acts like \u001b[36mx\u001b[39m but marks it as a\n",
       "  parameter you want to compute gradients with respect to.\n",
       "\n",
       "  \u001b[36m@diff expr\u001b[39m evaluates an expression and returns a struct that\n",
       "  contains the result (which should be a scalar) and gradient\n",
       "  information.\n",
       "\n",
       "  \u001b[36mgrad(y, x)\u001b[39m returns the gradient of \u001b[36my\u001b[39m (output by @diff) with respect\n",
       "  to any parameter \u001b[36mx::Param\u001b[39m, or \u001b[36mnothing\u001b[39m if the gradient is 0.\n",
       "\n",
       "  \u001b[36mvalue(x)\u001b[39m returns the value associated with \u001b[36mx\u001b[39m if \u001b[36mx\u001b[39m is a \u001b[36mParam\u001b[39m or the\n",
       "  output of \u001b[36m@diff\u001b[39m, otherwise returns \u001b[36mx\u001b[39m.\n",
       "\n",
       "  \u001b[36mparams(x)\u001b[39m returns an iterator of Params found by a recursive search\n",
       "  of object \u001b[36mx\u001b[39m.\n",
       "\n",
       "  Alternative usage:\n",
       "\n",
       "\u001b[36m  x = [1 2 3]\u001b[39m\n",
       "\u001b[36m  f(x) = sum(x .* x)\u001b[39m\n",
       "\u001b[36m  f(x) => 14\u001b[39m\n",
       "\u001b[36m  grad(f)(x) => [2 4 6]\u001b[39m\n",
       "\u001b[36m  gradloss(f)(x) => ([2 4 6], 14)\u001b[39m\n",
       "\n",
       "  Given a scalar valued function \u001b[36mf\u001b[39m, \u001b[36mgrad(f,argnum=1)\u001b[39m returns another\n",
       "  function \u001b[36mg\u001b[39m which takes the same inputs as \u001b[36mf\u001b[39m and returns the gradient\n",
       "  of the output with respect to the argnum'th argument. \u001b[36mgradloss\u001b[39m is\n",
       "  similar except the resulting function also returns f's output."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@doc AutoGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Redefine the constructor to use Param's so we can compute gradients\n",
    "Linear(i::Int,o::Int,scale=0.01) = \n",
    "    Linear(Param(scale * randn(o,i)), Param(zeros(o)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Set random seed for replicability\n",
    "Random.seed!(9);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(P(Array{Float64,2}(10,784)), P(Array{Float64,1}(10)))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use a larger scale to get a large initial loss\n",
    "model = Linear(784,10,1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.10423456298375"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can still do predictions and calculate loss:\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T(19.10423456298375)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And we can do the same loss calculation also computing gradients:\n",
    "J = @diff model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.10423456298375"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the actual loss value from J:\n",
    "value(J)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Param,1}:\n",
       " P(Array{Float64,1}(10))    \n",
       " P(Array{Float64,2}(10,784))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# params(J) returns an iterator of Params J depends on (i.e. model.b, model.w):\n",
    "params(J) |> collect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10×784 Array{Float64,2}:\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the gradient of a parameter from J:\n",
    "∇w = grad(J,model.w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "∇b = grad(J, model.b) = [-0.139954, -0.064541, -0.109522, -0.1275, -0.059184, -0.0980703, -0.102617, 0.0133898, -0.104578, 0.792576]\n"
     ]
    }
   ],
   "source": [
    "# Note that each gradient has the same size and shape as the corresponding parameter:\n",
    "@show ∇b = grad(J,model.b);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking the gradient using numerical approximation\n",
    "\n",
    "What does ∇b represent?\n",
    "\n",
    "∇b[10] = 0.79 means if I increase b[10] by ϵ, loss will increase by 0.79ϵ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value(model.b) = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "19.10423456298375"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loss for the first minibatch with the original parameters\n",
    "@show value(model.b)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To numerically check the gradient let's increase the last entry of b by +0.1.\n",
    "model.b[10] = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value(model.b) = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "19.183620170313954"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We see that the loss moves by ≈ +0.79*0.1 as expected.\n",
    "@show value(model.b)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reset the change.\n",
    "model.b[10] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking the gradient using manual implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Without AutoGrad we would have to define the gradients manually:\n",
    "function nllgrad(model,x,y)\n",
    "    scores = model(x)\n",
    "    expscores = exp.(scores)\n",
    "    probabilities = expscores ./ sum(expscores, dims=1)\n",
    "    for i in 1:length(y); probabilities[y[i],i] -= 1; end\n",
    "    dJds = probabilities / length(y)\n",
    "    dJdw = dJds * x'\n",
    "    dJdb = vec(sum(dJds,dims=2))\n",
    "    dJdw,dJdb\n",
    "end;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0], [-0.139954, -0.064541, -0.109522, -0.1275, -0.059184, -0.0980703, -0.102617, 0.0133898, -0.104578, 0.792576])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇w2,∇b2 = nllgrad(model,x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇w2 ≈ ∇w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇b2 ≈ ∇b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training with Stochastic Gradient Descent (SGD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sgdupdate! (generic function with 1 method)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is a single SGD update:\n",
    "function sgdupdate!(func, args; lr=0.1)\n",
    "    fval = @diff func(args...)\n",
    "    for param in params(fval)\n",
    "        ∇param = grad(fval, param)\n",
    "        param .-= lr * ∇param\n",
    "    end\n",
    "    return value(fval)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sgd (generic function with 1 method)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We define SGD for a dataset as an iterator so that:\n",
    "# 1. We can monitor and report the training loss\n",
    "# 2. We can take snapshots of the model during training\n",
    "# 3. We can pause/terminate training when necessary\n",
    "sgd(func, data; lr=0.1) = \n",
    "    (sgdupdate!(func, args; lr=lr) for args in data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 2.313187215962106\n",
      "3.03e-01  100.00%┣███████████████████┫ 6000/6000 [00:11/00:11, 544.04i/s]\n",
      "model(dtst) = 0.28065356838012656\n"
     ]
    }
   ],
   "source": [
    "# Let's train a model for 10 epochs to compare training speed on cpu vs gpu.\n",
    "# progress!(itr) displays a progress bar when wrapped around an iterator like this:\n",
    "# 2.94e-01  100.00%┣████████████████████┫ 6000/6000 [00:10/00:10, 592.96/s] 2.31->0.28\n",
    "model = Linear(784,10)\n",
    "@show model(dtst)\n",
    "progress!(sgd(model, ncycle(dtrn,10)))\n",
    "@show model(dtst);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 2.3035536f0\n",
      "3.03e-01  100.00%┣██████████████████┫ 6000/6000 [00:05/00:05, 1248.44i/s]\n",
      "model(dtst) = 0.28049186f0\n"
     ]
    }
   ],
   "source": [
    "# The training would go a lot faster on a GPU:\n",
    "# 2.94e-01  100.00%┣███████████████████┫ 6000/6000 [00:02/00:02, 2653.45/s]  2.31->0.28\n",
    "# To work on a GPU, all we have to do is convert Arrays to KnetArrays:\n",
    "if gpu() >= 0  # gpu() returns a device id >= 0 if there is a GPU, -1 otherwise\n",
    "    atype = KnetArray{Float32}  # KnetArrays are stored and operated in the GPU\n",
    "    dtrn,dtst = mnistdata(xsize=(784,:),xtype=atype)\n",
    "    Linear(i::Int,o::Int,scale=0.01) = \n",
    "        Linear(Param(atype(scale * randn(o,i))), \n",
    "               Param(atype(zeros(o))))\n",
    "\n",
    "    model = Linear(784,10)\n",
    "    @show model(dtst)\n",
    "    progress!(sgd(model,ncycle(dtrn,10)))\n",
    "    @show model(dtst)\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Recording progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainresults (generic function with 1 method)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function trainresults(file, model)\n",
    "    if (print(\"Train from scratch? (~77s) \"); readline()[1]=='y')\n",
    "        # We will train 100 epochs (the following returns an iterator, does not start training)\n",
    "        training = sgd(model, ncycle(dtrn,100))\n",
    "        # We will snapshot model and train/test loss and errors\n",
    "        snapshot() = (deepcopy(model),model(dtrn),model(dtst),zeroone(model,dtrn),zeroone(model,dtst))\n",
    "        # Snapshot results once every epoch (still an iterator)\n",
    "        snapshots = (snapshot() for x in takenth(progress(training),length(dtrn)))\n",
    "        # Run the snapshot/training iterator, reshape and save results as a 5x100 array\n",
    "        lin = reshape(collect(flatten(snapshots)),(5,:))\n",
    "        # Knet.save and Knet.load can be used to store models in files\n",
    "        Knet.save(file,\"results\",lin)\n",
    "    else\n",
    "        isfile(file) || download(\"http://people.csail.mit.edu/deniz/models/tutorial/$file\", file)\n",
    "        lin = Knet.load(file,\"results\")    \n",
    "    end\n",
    "    return lin\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train from scratch? (~77s) stdin> y\n",
      "2.29e-01  100.00%┣████████████████▉┫ 60000/60000 [01:07/01:07, 893.73i/s]\n"
     ]
    }
   ],
   "source": [
    "# 2.43e-01  100.00%┣████████████████▉┫ 60000/60000 [00:44/00:44, 1349.13/s]\n",
    "lin = trainresults(\"lin113.jld2\",Linear(784,10));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear model shows underfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "using Plots; default(fmt = :png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3deXxU9b3/8c/3zD6TlYSETTYREdECWqS0ikvD7W2rgFyxFIO1oF61WvuTroK1WrSgbW+1iqjUh1ttaIvaW2+LuBZaraUqKFJF9iUQsk9mJrOc8/39cUJMQ4AAyUwy5/V88Mhj5nvOnPnMYWbe8/2eTWmtBQAApzIyXQAAAJlEEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjdXsQhsPhBQsWTJ8+feHCheFw+NAZtm3b9uUvf7m7ywAAoEPdHoQVFRWlpaUVFRUlJSUrVqxoN7WpqWnJkiXxeLy7ywAAoEPdHoRr166dOnWq1+udOnXqmjVr2k6yLGvJkiVf/epXu7sGAAAOx93dT1BTU1NaWioipaWltbW1bSc988wzgwYNOvfcc4/w8M2bN994441er7ddeyAQeOqpp0zTVEoZBls6M0BrbZqm293tbyF0KJlMejyeTFfhUKlUyuVyKaUyXYgTWZaltXa5XJ2c3+12H/V/qtu/xbTWdhFaa8uyWtvffvvtd955Z/HixUd++P3339+vX78vfelL7dpdLpdpmuFwWCmVk5PT5WXjqBKJRFNTU58+fTJdiENVVVX169cv01U4VH19fTAY9Pl8mS7EiSKRiGmaeXl5nZy/Mz9Zuj0Ii4qKqqqqBg0aVF1dXVxc3Nr+9ttvr1+//gtf+IJ9t6ys7Oc///mYMWPaPdwwjLFjx1522WUdLjyZTCql/H5/NxWPIzAMI5lMsvIzxefzsfIzxV75BGFGmKZpmmbXvvm7fVBx4sSJq1at0lqvWrVq0qRJIrJ+/XoRmTdv3uqDRGT16tWHpiAAAN2t24OwvLx869ats2bN2r59++zZs0Vk/vz53f2kAAB0UrcPjebk5CxatKhti90FPHILAADpwf6WAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEfLqiD82369/EPr6PMBAHBQVgVhXUJ+t40gBAAcg6wKwpNzZUs400UAAHqVrArCYblqZ5NO0ScEAHRaVgWhzyX9AmpnRGe6EABAr5FVQSgiI/Lk48ZMFwEA6D26/XqEaXZyntrSqGWgynQhAHBsrr766nXr1mW6ip7rggsu+NnPftYdS87SIASA3mbz5s033HDDWWedlelCeqLXXnvt9ddf76aFZ1sQjsiTv+3PdBEAcFxGjBgxbty4TFfRE+3YsaP7gjDbthHSIwQAHJOsC8JctSWsSUIAQCdlWxDmeCTfI3s5ggIA0DnZFoQiMiJfcQQFAKCTsjAI7dHRTFcBAOgdsjEI2V8GANBpWRiEnFwGANB5WRWEVnPUrKuiRwgA6LysCsLmD96qf/6REXlqcwNBCADdrqysLNMldIGsCkJPyaBU1e4+PnEZUt2c6WoAoJf7zne+k+kS0iGrgtBdMihVvVe0HpGnPmZ0FABOzDvvvJPpEtIhq4JQef1GMNesO8BmQgA4QbfddpuIXHvttWVlZS+//PKMGTPKyspeffXVa6655tJLL/3973/fOmdjY+PixYtnzpx5+eWXL168uLGxUUT+/Oc/z5w589JLL3322WcPvdujZNtJt90lg5JVu0bkFbHjKIDe7v06vWCdlZ7nGlMoPz7b1bbljjvuKCsrW7ZsWVlZ2caNG++9995rrrmmqqpq2bJl77777oIFC2bMmGHPuXTpUrfb/eSTT4rIAw88sGzZsm9/+9sPPfTQz3/+c4/Hc999902fPr3d3fS8qE7KtiC0NxOe3G/cK3vpEQLo3foF1NdGpunqqsW+Iz3RnDlzCgoKROTiiy9WSo0bNy6RSLRO/fvf/758+XKfzyciV1111TXXXCMiZ5555vLly8vKyu6+++5D7/Yo2RaE7pKTUlW7Tx6pHvkwTT+jAKCbFPtl2pAesQHLTkERCQaDHc6g1Cc5apqmiPzoRz9at27diy+++MILLyxZsqTd3TTU3Hk9YhV3IXfJoGTVbnaWAYAukUqljjrPhAkTHnvssUQiEY/HH3vssXPOOUdErrjiiv79+19xxRWbN28+9G6Pkn09wkGpql39gxJNSWNS8jyZLggAeq0JEybMmTPnqLNdf/31Dz744OzZs5VSZ5111nXXXScil19++U033eRyueyR0nZ3e5SsC8LCEisa1vHY8FzPlkY9rihNw+sAkH0WLVrU9u7q1avb3bb/5uXlfe9732v32EsuueSSSy453N0eJduGRkUpd1H/1IG9HEEBAOiMrAtCEXfJScmqXZx6GwDQGVkYhC1HUNAjBAB0QhYGodsOwlyCEABwdNkYhKX2ERQMjQIAji4bg7BkUOrAnsEhOdCsY0c/AAYAcMyy4wJMtiwMQsMfMnx+Ha4dkqO2NTE6CgDHqcPLMGXftZmyMAjl4GH1J+cJmwkB4Lh1eBmm7Ls2U7YdUG9zl5yUrNo9Iu8MNhMCwPFpvQzT9OnTf/WrX6VSqfLycjsFr732WnuexsbGpUuX/vOf/1RKjR8//rrrrsvLy/vzn//cOv/06dPb3c3kSzqMbA3CQamq3SNGqn810CME0Fulqvc2/eX59DyXu3hAznlT27a0XoZp2rRprVdQWrJkSeu1maT3X4DJlp1B6CkZFP/w7bM/yzUoAPRiyuNzFw9Iz3O5CooPN+kIV1Dq7RdgsmVnELpLBqX27zqnr6pu1psb9Cn5nHEUQO/jyi9q10vLiCNfQalXX4DJlqU7y/TpZ4brVCpxyWDjf3cyOgoAxymVSh16BaXWazP19gsw2bKzRyiG4erTL1W9d+qQIT9Zb/6/M7Iz7wGgW9mXYfrKV77S9gpKba/N1NsvwGTL0iBsOePorgvHDP3qq3p/TEoDmS4IAHqb1sswtb2CUttrM/X2CzDZsrarZF+q3ueSKYOMF3axywwAoGPZHISpqt0iMnWIen4HmwkBAB3L2iD0lJ6U2r9bRL50kvF6pRXhpKMAgI5kbRC29gjzvXJOX7VqN6OjAIAOZG0QGsFccbnMcJ2ITB1iMDoKAOhQ1gahiLhLTrI7hdOGqhd2Win6hACAQ2RzEHoOjo4OCKqT89Rf9tEpBAC0l81B2LqZUFpGR+kSAgDay+Yg9A4eGf/4Pfv2tKHquR2aLiEAoJ2sPbOMiPhGnGmG65KV2z39h44uUH6XvFujxxVxAm4APdSKFSv++c9/ZrqKnuj999/vvoVncxCKUsGzLoj+89X8L18lLUfWW+OKXJkuCwA6sHTp0kcffXTv3r2ZLqQn6tOnz+TJk7tp4VkdhCLBT3++eun38794pRjGZcOMma+YPxgr3mweDwbQW5122mk//elPM12FE2V5Jnj6DTZyC+NbNojIp/uqU/NlOZfqBQC0keVBKCKhT18U/cfL9u1FZ7t+/I4V5XRrAICDsj8Ig2ddEHvvDR2PichZxWpCX/XQJjqFAIAW2R+ERk6Bd9jo2Htv2Hd/fLaxeIMZTma2KABAT5H9QSgiwU9fFF3XMjp6eqH6/ADjvo10CgEAIg4JwsCYzyR2bTbrq+27t483/ud9szae2aIAAD2CI4JQebyBMyZF337NvntKvpo6xPjZe2ZGiwIA9AiOCEKxR0ffWt1697ZxxtJN1v5YBisCAPQITglC3/AxOplI7tli3x2co2aPMH6ynk4hADidU4JQlAqefWHkrZdaG34w1vWbLdZars0EAM7mmCAUCU38j+i6l1PVLefx6xeQX53n/uqrZnVzZusCAGSSg4LQVViSW/aVut/8XA5ejuk/T1KzR6g5r6csuoUA4FQOCkIRyZ08XZtm5O+rWlvuPMvVlJTFGzisEAAcyllBKEoVzryp4X9/ZTbU2A1uQ1Zc5P7lRuuVvfQKAcCJuj0Iw+HwggULpk+fvnDhwnA43HbSW2+9NXfu3GnTps2dO3fdunXdXYnN039ozue+XLfivtaWfgF58nxX+WvmPo6mAADn6fYgrKioKC0traioKCkpWbFiRWu7aZp33XXXDTfcsHLlyiuvvPLee+/t7kpa5U75qlmzL7Z+bWvLhQPUVSNV+WupBEOkAOAw3R6Ea9eunTp1qtfrnTp16po1a1rbTdP8/ve/P27cuObmZo/HEwqFuruSVsrlLpz1rfqVS63oJz3UH53lyvOo/3rJJAsBwFG6/Qr1NTU1paWlIlJaWlpbW9va7vV6zznnnFgsNnXqVBH5n//5nw4fbprms88+u3379nbtPp9vwYIFTU1NSimtj33zXp+BrtMmHFhxf3DGja1tyyfINW96vvR/yV9/Lul3HfMinSaRSDQ1NXm93kwX4lBNTU3ttjUgbZqamizLSiQSmS7EiaLRqGmaSqlOzh8MBl2uo3yhd3sQaq3tirXWltW+txUIBP7whz88++yzDz744AMPPNDhEkKhUHFxcbtGj8ej2jiOwoJTZjctvz2+6kn/F+bYLW4lj3wm9d9/93z1r95ffzYZ6PZ107udyMrHiWPlZxBv/gw61pXfmTm7/cu+qKioqqpq0KBB1dXVbfOssrLyj3/849VXXx0IBP7zP//zmWee6fDhLpdrypQpN998c4dT7ZTNyck5rtJygjcurn7w++ZLv86fdk1r61MXyVWvm1e84XquzE0WHkEikdBaH+/Kx4kKhUKs/ExJJBKhUMjn82W6ECdSSpmm2bVv/m7fRjhx4sRVq1ZprVetWjVp0iQRWb9+vYgUFRW98MIL7733ntb6tddeGzFiRHdXcijDHyq+7q74lvcbnn+ktdGl5LHJrr4BdcnqVAMjHwCQ7bo9CMvLy7du3Tpr1qzt27fPnj1bRObPny8iXq/3Rz/60YMPPnjppZe++uqr3/rWt7q7kg4ZgZzi6++Kf7yh4Q+Ptja6lDw+2XVagfr086kNtRxfCADZ7Lj2NEmjm2++eejQoYcbGg2HwycwNPoJK9J44IHvBk6fmPfFOdJmQPmZLdaNfzOXnOP6+kiHnXmgExKJRDgcLioqynQhDlVZWdm/f/9MV+FQtbW1DI1mSiQSMU0zLy+vC5fJ97uIiBHK63v9T5o/Xl/98EKzsa61fdbJxtqL3T/dYF27lsMqACA7EYQtjJz8khvv9Y04s+qn32je+GZr+6gC9cZU94FmOe+Pqc0NPbr3DAA4DgRhG4aRe9HMoq8vrH92We3T9+hEy/WZ8jzy+8+7Zp9sfPZ/U4vetZJ0DQEgixCE7XmHjCqZ/0uxrKqf3dS8qeUMqErkxtONdy91v1Ojxz+beqOKriEAZAmCsAOGP9Sn/Lt5X7qq/rmHD/zyO4nt/7LbBwTV7y5y3TbeuOxl84a/mbXxzJYJAOgCBOFhBc74TL/vLcv53MW1T/7kwIPfT+7dZrdfNsx4f4ZbiZz62+Sid61IKrNlAgBOCEF4REoFxp5b+r2H/aedfeDB71c/cnvzpnWidYFXfjnJtW6ae2eTPrkiuXi9FTczXSoA4LgQhEenPN7cC2b0v/3J0KcvanzxmX2Lvh5+eYUVaRySo5Z9zvXnL7j/ss867XepZf+yYvQOAaC34WSanaXcnsDYcwNjz03s+FfT2j/u+/HXfSPHBcee+6nTJ7zwH/6/7teL11s//GfyutNc1482+vozXS4AoHMIwmPmHTKqz5BRVrQp9t7fIv9YXVfxC9+p48ePPfe5887+qDnws/etU3+bvHy4cePpxugCTk4PAD0dQXicjGBO6JwpoXOmWNGm2PtvRP7xUu1vft5nwPB7R42/c8JZD4VPnvInPTRH5p1qXDbcCLGaAaCn4hv6RBnBnNCEstCEMp1MJLZtbP7wbfnD/VfXVV0/bPQ2z2m/efu0H7wxYuoI/9dOMc4poYMIAD0OQdhllMfrGznON3Jc/sVzzXBdYuvGkds++N6ux/5f5bYDu4f99a+nPBEccfKpI6aMGzKmmNUOAD0F38jdwpVbGPjU5wKf+pyI6GSi766PRuz4sGrL+sia3/v/vP+V4GDd/+SThg0ZcvJQd78hrrw+ma4XAJyLIOx2yuP1DR/jGz4m9wIRESvR3PDexxs3bftgw44hb7w5Kr7Dr1K+/oO9fQe6W/4NcBcPUL5ApgsHAEcgCNPN8PrPOmvMWWeNEZH36/Rvd+hXNte7qndMie+bsHfv8G2veer2pKr3Kl/Q3afUXdTPVdTP3afUVdDX1afEVVBs+EOZfgUAkFUIwkwaU6jGFKpbx/bZH+vz4h7r4d169R6rqI+aMlmV5dVP9FT5w/vNmn2JXZvN9/5m1h1I1VUppVwFfV0FxUZeH1d+kSu30FXY15VTYOQWuvIKlZcDGAHg2BCEPUJpQMpHGOUjxNKud2v06j36/p35X92fNyL/lPP7q/NPVZNKjVKfiIjVHDHrDpgNNVZjbaqhOnVgT3zzerOp3grXmY21IsqVV2jkFBihfFdOnpFTYOQWuIK5Riiv5V8w1wh15ZWdAaC3Iwh7FkPJ+GI1vlh991NG0pJ/HNCvVupfbrSueNUcFFKTStVnSwOTSoeM7D+0w4frRLPZUGM1NZiRRqupwQrXmXUHknu2WJGwFW20ImEr0mBFm4xgrhHMMYK5RjDXCOSoQMgI5BjBnJbb/qDhDyp/yPAHlS9oBHPSuw4AIK0Iwp7LY8ikUjWpVN061jC1vFer1+7XL+7Rt79thZN6Ql81oa+aUGJM6Ktaz+imvH5334HSd+CRlqu1FQ1b0bAVa7IiYSvWpGMRK9ZkRZtSNfusWJNujlrNUd0ctWIRHY9ZsSblCxj+oPIFDF9ABUKG16+8fu31J5WrMa9AeX1GIEd5vMrjM4I54vYoj8/wB5Xbo3xB5fEqjzcNqwsAjg9B2Du4lIwtUmOL1DdGi4jsj8lbB6y3Duj73jffOqBzPcruR44vUuOLZUDwiEfuK2UPk3b+2a3mqI7HdHPUSjTrWJMVb9aJ5mQknGqoE6XsBNXJuE4mrGiTpJI6GbdiEW2mdDyqE3GdShr+kLhchj8kbrfh9SuPT9wewxcQl8vwBcTlNnwBMVzKF1Aul/L6lcutPD7l8bbMIEoFQiJiP0S5PPbWUMMfFIMTxwM4IQRhr1QakIsHGxcPFhHRIlsb9ds1+u1qff9G650aLSKfKlKf6qPO7KPO7KNGFyrviYWF4Q+KPyj5RW0bE4lEKhzOKyo63KPasmJNYppWPKpTSZ2I62RcUkmrOSaWacWjYppWPCaWqZtjVtzU9dXaTNnJKmbKijeL1ro5IiJWc1QsS6cSOhkXESsWEa2Vy618fhFRXr9yeUSJEcgREVGG8gdFRLlc9uEoynC1tBycJCJGICRKiYhyf9J5NQI5dqO43Iavpcdt53frbeU5eNvtVR7fwZVlGAeXDKBXIAh7PSVycp46OU9dNqylZW9Ub6iV9TX6z7v1kg3W1rAemqNOL1RjCtXphTK6UI3IO9FoPFZ2Mhk5+d2xcG2mdLxZRHQips2UaLFiTSIi2tLN0X+bwUzpeExE9MFJcjBNRUQn4zqVbGmMhluWbiex/UTJuLTOkPjkdmswi4hYlnVwySKivD51MDtFRPmCynAdbqq0ieeWhoMZ35bhD4lhWNFoXbBlZnXIA1vm7HD7rjpsVCu35wg7Hit/UB2x/608XnEffRhcGS7lP4bDZA1fQIzj/6Zq/aGDLJFoPpH3Q4cIwiw0IKgGBOULg1o+/ElLPmzQG+v0+3X6mS2ysc7aGdGDQ2p0oRqVLyPz1akFamS+KvIdeak9l3K5lf2N3/P267FHhj+5G49qyzzcVGkTzy0NByO8Las5IpYVa2jw5rf8sNCHPLClNdrUUU1Wqnpvx9WmkjrR/uk+mdoc1ZZ1uKny7z8UjjSbZerm2FFna2XFY2Id/6U+W3/odK1DVrfYA/td/kQ9gqWt5kimizjI68+9Zankd+WvaoIw+3mMlgMWLz/YkrRkc6PeVKf/1SAv79VLN1kfNWiXklPy1an5akSeOjlPRuSpEXmqsNemYw+hvD7lbbMSuy6qGysrQ/37d9XScExqa2tDoZDP928fDx2PadM83EN6N0P1nFN5RCIRs6vXM0HoRB5DRheodpdLrIrJhw16c6Pe0qif2yEfN1ofN2iPIcNy1fA8NTxXhueq4XlqaI6clJPukVWg51O+ACOwvRRBiBYlASkJqHP7/dtn+UCzbG3UW8N6a1jerNLPbLG2N8neqC7xqyE5aqAvMLLIHJyjBueowSEZkqMCvKEA9DZ8b+FI+vqlr1+1u5JiypI9Ub25NvlhTapa+9/Yryu2WjubZGeTzvXIoJA6KUcNyZFBITUwKINz1ICgDAwpf5ZuPQHQ2xGEOGZuQ4bkqP5eGRdKFhX92yDp/pjsjujdEb2jSXZH9Poa2RWxdkdkb1Tne6V/QA0KyYCQGhhUA4LSP6gGhqRfQJUGxGBQCUCGEIToSqUBKQ2os4o7iLX9MamMajsU90b1WwekMmrtjUplVNfEpa9f9Q9K/6CUBtSAoJTYf/2qJCADgirHc+jyAKBrEIRIEzsjxxaJSPuYTFmyP6YrY7IvKvtjem9UPqzXr+6VqmarKiZ7o9rSLUnZ16/6+qXlRkBKA6rEL30DqtgnbvbfAXBcCEJkntuQgSE1sGX37A56k9GU7Ivp/TE5ENP7Y7IvJlvC+u8HpDJqVTfLgWZ9oFkKvVLsV30DUuxTxX4pCUixXxX5pNiviv1S7Jdiv8qjZwngEAQheoGgW4bnquG50mFMiogWqW6W6mZt/61qlupm2dmk366WmrgdllLTrJtNKfJJkV8V+aSPTxX5xU7KPj7p45M+vk9usPsr4Bx83JEN1MEdXA/e61jCkppmqYlr+29tXKqbpSqm/1UvtXGpjVu1cfuGFmnJxUKfFPpUobf9jQKvFHil0CcFXhXkYwT0ZnyC4SBeQ/oHpX/wKHkpItGU1MV1bVzqEgdvxKUuoT9skPqE1MWt+oTUx6UuoesTYlpS4JMCryrwSr5XCryq0NdyI98rB/+pfK/keaTAqwp8R3xuAOlFEAIdCLol6D7SZsu24qbUJ6Q+oevj0pCU+riuT0hdXBoSek9UGhLSkNANCWlISGNSGhK6Li65Hsn3qjyP5HklzyP5XlXgk1yP5HlUrkdyPVLg++S2PXO+l4NMgG5BEAInyudq2Sf2YMPR88pOxMaENCalMSGNSV0Xl3BSGhN6d0TCSTs+rXBS7H+NSV0fl4Bbcj2S41EFXvHp/D6hVI5H5XkkzyM5HhXyiH075JGQW+V7JccjIbeE3PRBgSMhCIEMyPNInkfJJ+cx7lRORVISTkpTUjckZNu+qC/PH0namSpNSb0vKpuT0pCQSEpHU9KQkHBSIimJpnRd3O7jtvQyg24JeaTAqwIuCbil0CdBtwq4JN8rIbcE3JLnUSGP+F2S75WgW/ldUnD0yysBvRVBCPQaIbeE3CIBJSIDksn+/Y/h2Mloys5R3ZiQaEqiKalP6GhKYqbUxyWS0lUJ+bhRIimJpaQxaTUlpdmUxoREUrrZlIaEBNzid0mBV/lddqaK3y05bpXrEa9L8r3id0nApfK84jUkzysBl/K7pMAnHkNyPRJ0K58h9E3RAxGEgCPYPcKDO9baji2SYilpNqUuoZtNiaWkISHNpkRSOpyUuCmNCYmZui6htzdJwpRwUqIpK25JfVwSljQlJZLSCUvq4+IyJNfTEpN5XnGrlrDMcSs7a3M94jakwKvchuR5xOdq2WTrM1oniaGkwKtchnBsKE4cQQigUwJuexy1XXwecwcvZUk4KTFT2z3OpCUNCUlYEklpO2vDSUlZUpfQKUs2JyVu2l1YK25JY0JMLfUJMbU0JLS9KI8hOR7xGRI8GKVBt/hcEnIrryE5npYuqduQPI9yKcnziku1tOR6lFu1zGP/tR9lLwcOQRACSCu3IYU+KewgQY9z0DRhSSQpzabEzJYojaYkbkpTSietllhtTIppSWNSm1p2NImpW1rCSSulW+ZpSkrSaum52kvwuyTgbolYOykNJfleSSYDRQHDZZh2mtq5a48M27MpkQKfiLRErz2D15CQPcmrRFpyFz0BQQigd/Ma4m25VvyJ9lbbsWPVjtiEKZGUmFoaExIOR2OG4fG47B5qJCUJs2XcOGlJU1IsLQ0JEZGGhGWJRJKSsFoCW4vUJ7RIS+7a6SgihV4lIiGPeI2W7qmIFHiVkpbuaWu42qPErcPCdhjbY8siLf1dtyG5HhGRHLfyGC35jcMhCAGgY/ZosIi0y9Ta2lQo5PP5uqBD12E62o0iUpfQIi3d09ZwtfcKtseNRVq6tklLmlKWSMvosZ3HIi3dYtOSxmTLK7KHfO2IbU3ckFu8LpGDedzabvdl5WDXVqmW/YdbZ7C39crBABaRAq8o9UnHt1dsxyUIASBjWruzfdpvfLV18T62dqaKSH1Ca2mTl0lJWiIHo7e1vXX+hoS2RCwtWxrl32ew4pY9g1haRKQuLiKfRHtrBrdmp9eQkFuJtIw8S5u4tTu7bVta57EHqEVEma6vDTPzunS1EIQA4BT2zsMih+701Kq7Dm9pzc64JdGUFhF792NpE7fRlLZjtbXFHp22Z7ZDOtwsuquLJAgBAN3OY0ihr/Xe8cdwJNJsmrqLimrBTksAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaAQhAMDRCEIAgKMRhAAARyMIAQCORhACAByNIAQAOBpBCABwNIIQAOBoBCEAwNEIQgCAoxGEAABHIwgBAI7W2SC0LKupqcm+nUgkqqurtdbdVhUAAGnSqSDcvHnz7Nmzly1bJiIffPDBzJkzZ82adfXVV+/du7ebywMAoHt1KggfeuihYcOGXX/99SLyyCOPjB8//re//W2fPn0eeuihbi4PAIDu1akg/Oijj84777xAIBAOh99///3/+q//KigouPDCCzds2NDd9QEA0K06FYRerzeZTIrI+vXr/X7/yJEjRcQwDKVU91YHAEA361QQnnnmma4ZZ/MAABH4SURBVGvWrKmpqXn22WcnTpzodrsjkcjq1atHjRrV3fUBANCtOhWE11xzTXV19Ve+8pWdO3eWl5eLyH//939XV1dfe+213VweAADdy92Zmfr37798+fLq6ur8/Hyv1ysi999/f35+PkOjAIDerrPHEWqtA4GAnYKJRCKVSnVnVQAApAnHEQIAHI3jCAEAjsZxhAAAR+M4QgCAo3X7cYThcHjBggXTp09fuHBhOBxuO2nNmjXz5s2bNm3at771rd27dx/nKwAA4AR0+3GEFRUVpaWlFRUVJSUlK1asaG2vrKy855575s+fX1FR8ZnPfOaee+457tcAAMBx6/bjCNeuXXvHHXd4vd6pU6fedtttc+fOtdsrKysvvPBCu085ZcqU3/zmNx0+3DTNtWvX2k/alsfjmT17diwWExHD4KqKGZBIJGKxWDQazXQhDsXKz6BYLKaUMk0z04U4UTQatSzL7e5UeImI3+8/akZ0dllKqb59+yYSiX379hUWFhYUFHTygTU1NaWlpSJSWlpaW1vb2j5+/Pjx48eLiGmajz/++Pnnn9/hw7XWO3bsePPNN9u1+/3+mTNnJpNJpZS9/RJpljwo04U4FCs/g+yVz0/wjEilUqZpdv7N7/P5jjpPZ4Pw9ddff/rpp7dt22bfHTp0aHl5+XnnnXfUB2qt7Y6j1tqyrHZT161b9+ijj5599tlXXXVVx/W53bNnz7755ps7nGrvsJOTk9PJV4EulEgkDMPIz8/PdCEOFY1GWfmZYppmKBTqzDcsupzb7TZNMy8vryuX2ZmZXn311bvvvnv27Nk//OEPi4uLa2pqXnzxxTvvvHPBggWTJ08+8mOLioqqqqoGDRpUXV1dXFzc2q61fuSRRzZt2rRgwYJBgwad0IsAAOB4daprX1FRMX369CuvvHLgwIE+n2/AgAFf+9rXpk2bVlFRcdTHTpw4cdWqVVrrVatWTZo0SUTWr18vIhs2bHjjjTfuvPPOoqKiWCxmb+0DACDNOhWEe/bsGT16dLvG008/fdeuXUd9bHl5+datW2fNmrV9+/bZs2eLyPz580Vk/fr1u3fvnj59+iUHHXvxAACcqE4NjQ4cOPCDDz5oNwq6adOmzgxp5uTkLFq0qG3L6tWrRWTOnDlz5sw5llIBAOh6neoRXn755StXrnziiScqKysTiURlZeUTTzyxcuXKyy+/vLvrAwCgW3WqR3jBBRcYhvHUU089+eSTdsuQIUNuvfXWwx3zAABAb9HZwycmT548efLkeDxeV1dXWFjIfsMAgOxwbAeE+ny+fv362Sn40ksvlZWVdU9VAACkCWdGAAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjHenwiZ07dx5hak1NTVcXAwBAuh0pCFsvogsAQLY6UhDaJwUFACCLsY0QAOBoBCEAwNEIQgCAoxGEAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaAQhAMDRCEIAgKMRhAAARyMIAQCORhACAByNIAQAOBpBCABwNIIQAOBoBCEAwNEIQgCAoxGEAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaAQhAMDRCEIAgKMRhAAARyMIAQCORhACAByNIAQAOBpBCABwNIIQAOBoBCEAwNEIQgCAoxGEAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaGkKwnA4vGDBgunTpy9cuDAcDtuNpmleddVV6SkAAIAOpSkIKyoqSktLKyoqSkpKVqxYISIrV6785je/uXv37vQUAABAh9zpeZq1a9fecccdXq936tSpt91229y5c4cPHz5gwICFCxce+YFa648++ujll19u1+5yuT7zmc/E43GllMfj6bbCcViJRCKRSMTj8UwX4lCs/AyKx+Nud5q+PNFOPB43TbPzb36v16uUOvI8afq/rKmpKS0tFZHS0tLa2loRGTt2bGcemEql/vSnP7377rvt2gOBwG9/+9toNKqUOuqLRHdIJBKRSMTn82W6EIeKRqORSCTTVThUNBoVkVQqlelCnCgajZqm6XK5Ojm/2+0+6sxpCkKttR1XWmvLsjr/QI/H881vfvPmm28+3FSlVE5OTtdUiWORSCQ8Hk+fPn0yXYhDxeNxVn4GhUIhfgVmhM/nM00zLy+vC5eZpm2ERUVFVVVVIlJdXV1cXJyeJwUA4KjSFIQTJ05ctWqV1nrVqlWTJk0SkfXr16fnqQEAOII0BWF5efnWrVtnzZq1ffv22bNni8j8+fPT89QAABxBmrYR5uTkLFq0qG3L6tWrW/8CAJApnFkGAOBoBCEAwNEIQgCAoxGEAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaAQhAMDRCEIAgKMRhAAARyMIAQCORhACAByNIAQAOBpBCABwNIIQAOBoBCEAwNEIQgCAoxGEAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaAQhAMDRCEIAgKMRhAAARyMIAQCORhACAByNIAQAOBpBCABwNIIQAOBoBCEAwNEIQgCAoxGEAABHIwgBAI5GEAIAHI0gBAA4GkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjkYQAgAcjSAEADgaQQgAcDSCEADgaAQhAMDRCEIAgKMRhAAARyMIAQCORhACAByNIAQAOBpBCABwNIIQAOBoBCEAwNEIQgCAo/XuIFy+fHlFRUWmq3Cod9555wc/+EGmq3CuGTNmmKaZ6Soc6vbbb//73/+e6Soc6rnnnlu6dGnXLtPdtYtLs507dxYUFGS6CodqaGj44IMPMl2Fc/3jH/+wLMvlcmW6ECf617/+VVtbm+kqHKqysnLbtm1du8x0BGE4HF68ePHGjRvHjBnzne98Jzc398jtAACkTTqGRisqKkpLSysqKkpKSlasWHHUdgAA0iYdPcK1a9fecccdXq936tSpt91229y5c4/c3s6ePXvefffddo2GYRQXFzc1NRmGsXfv3u59AehITU1NIpFg5WeK1rqystLt7t1bN3qpeDxeW1vLmz8jGhsbo9Fo51d+SUnJUT8mSmt9woUdxcUXX/y73/3O5/PF4/HLLrvsD3/4w5Hb23ruuedmzJhxaLtSqqioKJlMKqX4LsgIy7KSyaTP58t0IQ4Vi8UCgUCmq3CoeDzudrvZQJsRqVRKa+3xeDo5/0svvXTGGWcceZ50RIjWWill37As66jtbU2bNo1d4wAA3Scd2wiLioqqqqpEpLq6uri4+KjtAACkTTqCcOLEiatWrdJar1q1atKkSSKyfv36DtsBAEizdARheXn51q1bZ82atX379tmzZ4vI/PnzO2wHACDN0rGzDAAAPVav3N+SI/EzYs2aNY8//nh1dfWwYcNuueWWQYMGichNN920adMme4YvfelLN998c0ZrzFqHrmc+BelRVlbWruW0007jPZ8epmnOmzfvsccek46+9rvwI9Are4SPPvpoLBa79tprly1bFgwGD3cAIrpQZWXltddeu2TJkmHDhj3//PN//etff/GLX2itZ8yY8eijj9r78btcLq/Xm+lKs1CH65lPQXrEYrHW2xUVFclk8k9/+hPv+TRYuXLlK6+88uGHH65evVo6+trvwo9Arzzp9tq1a6dOnWofib9mzZpMl+MIlZWVF1544ahRo3w+35QpU3bt2iUiNTU1pmneeuutM2fOvPvuuyORSKbLzE4drmc+BekROGjfvn0ffPDB1KlTec+nx/Dhw6+44orWu4e+4bvwI9Arg7Cmpqa0tFRESktLOfVteowfP94eAjJN8/HHHz///PNFpLa29pRTTrnllluefvrpUCj04IMPZrjKLNXheuZTkE7JZPKnP/3pDTfcUF9fz3s+PcaOHTtx4sTWu4e+4bvwI9ArtxF25kh8dId169Y9+uijZ5999lVXXSUiI0eOvPfee+1J8+bNmzdvXkary1odrmc+Ben0u9/9btSoUUOGDBER3vMZcegbvgs/Ar2yR8iR+OmntX744YeffvrpBQsWzJs3zz651EcffbRx40Z7Bo/H0/mTHuGYdLie+RSkjWmaf/zjH6dPny685zPn0Dd8F34EemUQciR++m3YsOGNN9648847i4qKYrGYvQdBc3Pz7bffvmPHjmQy+dRTT332s5/NdJnZqcP1zKcgbd59992+ffsOHDhQeM9nzqFv+C78CPTKvUabmpruvvvuLVu2nHLKKd/73vdCoVCmK8p+TzzxxJNPPtm2ZfXq1Vrr559/fuXKlZFIZMKECd/4xjf4v+gOHa5nPgVpc/fddw8aNKi8vFwO83+R6QKzWVlZmb3X6KFv+C78CPTKIAQAoKv0yqFRAAC6CkEIAHA0ghAA4GgEIQDA0QhCAICjEYQAAEcjCAEAjtYrzzUK9GqHXuLOZh843LVPtHz58sGDB3ftYoEsQxACGfDd73430yUAaEEQAhnw+c9/PtMlAGhBEAI9yM6dO+fOnfvMM8/cd9997733Xt++fS+77LK2Q6mvvfZaRUXF7t27TzrppJkzZ9oXhrStXbv217/+9a5du4YOHTpjxgx7Ujwe/+Uvf7lu3brq6uqRI0fedNNNQ4cOFZHKyspHHnlkw4YN8Xh85MiR11133YgRI9L8YoEegp1lgB5n/vz5JSUl3/jGN0aNGrVkyZLXX3/dbn/99dcXLVp0xhln3HLLLWPGjFm0aNFf/vIXe9Kbb7555513jhs37tvf/vawYcMWLVpkb3H84Q9/qLW+6aabrr/++vr6+nvuuUdEtNa33nprOByeM2fO9ddf7/V6b7vtNi5qCMeiRwhkQIf7y7TuLHPeeed9/etfF5GLLrooGAw+88wzkydPFpGnnnrqiiuuuPLKK0Xk/PPPDwaDTz311HnnnSciTzzxxJw5c2bPni0i55577p49e1588UURmTBhwo033mgvduDAgd/+9rdFpLq6eteuXffcc09RUZG9qAcffDAajebk5HT/Swd6HIIQyIDly5cfYeoXvvCF1ttf/OIXf//73ycSCa319u3b77jjjtZJZWVlTz/9dCKREJHNmzcvXLjQbldK/fjHP06lUpdeemnbsdPCwkL7ajP5+fkFBQV33XXX9OnTP/WpT+Xm5t5yyy1d+vqA3oQgBDLgyIc0tL3cdklJiYjU1tbad+0+XNvZOpwUCATsG4WFhYcu3+v13nvvvU8++eTixYubm5vPOOOMefPmjR49+vheC9DbsY0Q6HGqq6vb3S4sLLQjraampnWSfbuwsLCgoEBE6uvrWydVVlZu2LBBRJRSHT7FkCFDFixY8Oyzz/7iF7/Iz8+/9dZb7Z4l4EAEIdDjrFq1qvX2Cy+8MGTIEJ/P5/P5hgwZ0vag+xdffHHo0KE+n8/v9w8ePPiVV15pnXTfffc9/vjjh1t+fX19eXl5JBJxu92jR4/+5je/2dTU1NjY2E0vB+jhGBoFMuCll17qsH3kyJEi8txzz0UikdGjR69fv/7//u//br31VnvqFVdccdddd0UikdNOO23Tpk0rV65s3S5YXl5+9913RyKRU0455Z133lm3bt2CBQvsTuGh8vPzXS7X7bffXlZWprV+5ZVXBg8e3HZkFXAUZW88B5A2hzvFmogsX7587ty5Dz/88AMPPLB58+a+ffvOnDlzypQprTO8+uqrFRUVe/bsGThw4KxZs+y9SW0vv/zyihUr9uzZ079//5kzZ5aVlbU7xZp9kKLdp9y+ffvSpUs//PBDERkzZsx11103cODA7nrBQM9GEAI9SNusApAebCMEADgaQQgAcDSGRoEepLm5+eOPPx4zZkymCwEchCAEADgaQ6MAAEcjCAEAjkYQAgAcjSAEADgaQQgAcLT/D9JQypcxZ62IAAAAAElFTkSuQmCC"
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Demonstrates underfitting: training loss not close to 0\n",
    "# Also slight overfitting: test loss higher than train\n",
    "trnloss,tstloss = Array{Float32}(lin[2,:]), Array{Float32}(lin[3,:]) \n",
    "plot([trnloss,tstloss],ylim=(.0,.4),labels=[:trnloss :tstloss],xlabel=\"Epochs\",ylabel=\"Loss\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dfXwU1b0/8O+Z3dmnbLJJNskmEEjCkzwEEKQWsSpaQ629GgHFIkSL8Ku19eptiag/ArZYRMCHq7faq8JtEa03/Aq2PkcQRWhpKwqhBRQQAgHyQJ73eXZnzu+PCWtIQgiY7CZ7Pu8/eO2eGWbP7k7ms3POmTOMc04AAACikmJdAQAAgFhCEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgtOgFodvtLikpmT59+pIlS9xut16oquq8efMi62zfvn3BggW33HLLz3/+8xMnTkStbgAAIKzoBWFpaanL5SotLc3IyNiwYQMRbdq06YEHHogEXlVV1erVq4uLi0tLS6+44orVq1dHrW4AACCs6AXhjh07CgsLTSZTYWHh9u3biWjIkCFz586NrFBVVXXdddeNHDnSbDZPmzatsrIyanUDAABhGaP2SvX19S6Xi4hcLldDQwMRXXrppW1XmDhx4sSJE4lIVdV169ZNnTr1XJv63e9+t2nTJsZYu/JbbrmlqKgoFArJstzj9Yeucc5VVTUao7dHgU5VVcaYJKG/P9pwqImJCz3UGI3GjmHRfp1vXKvu4pzrteGca5p2rtV27dq1Zs2aSZMmte07bGfp0qUPP/xwRkZGu/KhQ4eqqnr69OmMjIzzvnPoWcFg0OfzpaSkxLoiwmlubpZl2Wazxboiwqmrq0tLS8NPkChTFMXj8aSmpnZzfYPB0IeC0Ol01tbWZmdn63tPxxU45y+//PKBAwdKSkqys7O72JTRaLzxxhvz8vI6XWo2my0WC4IwyhhjqqpaLJZYV0Q4wWBQlmV88tGnH2oQhFEmSVIoFOrZHT56X+HkyZPLyso452VlZVOmTCGi8vLytivs3bt3586djz32mNPp9Pv9fr8/anUDAABhRS8Ii4qKjhw5Mnv27IqKijlz5hBRcXFx2xXKy8tPnDgxffr0m8+IWt0AAEBYrD/ejzAvL2/r1q3nahqtrq52uVxoGo2yYDDo9Xq733APPQV9hLFSU1OTnp6OptEoUxTF7XY7nc4e3Ca+QgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoRljXQEAAOicx+O56qqrYl2LvuLaa699+umne2PLCEIAgD5KVdXDhw9/8sknsa5I7H388cfbtm3rpY0jCAEA+i5JkiZMmBDrWsTesWPHei8I0UcIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgDABVi0aFGsq9DDEIQAAHABdu/eHesq9DAEIQAAdNfSpUuJ6J577ikoKPjwww9nzpxZUFDw0Ucf/fjHP54xY8bGjRuJqKGhYdmyZbfeemtRUdHKlSvr6+uJqO36+oMYv5M24n/S7ZBGRolYrKsBANAj/ueg9uYxHp3Xmn8Ju2nwWedLy5YtKygoePHFFwsKCvbt2/fkk0/++Mc/rq2tffHFF/fs2VNSUjJz5synn376+uuvf+ihh0Kh0KZNm5555plf//rXRBRZX38QnbfQHfEfhGM3ht+aZhjuQBQCQDy4PJ2lmqP0WqOTuzpy3nnnncnJyUR00003McYmTJigKAoRlZeX//3vf4+s5nA42q0fedBHxH8QDkuiA00cQQgA8SE/heWn9IkDWiTMbDZb23K73f7SSy9lZWURkd/vd7vd7dbvUylIIvQRjkpmB5piXQkAgDgSDoe7WPqd73zn9ddfDwQCjY2Njz766Ouvvx61il0cAYIwhR1oilJ7OgBA3Lv88svvvPPOLlaYN2+epmlFRUULFixwuVw//vGPo1a3ixP/TaOjk9mLB7RY1wIAIE4sX7687dPNmze3e2yz2YqLi9v9r8hqbdfvI+L/jHBkMvuiieOUEAAAOhX/QZhsogSZTnoRhQAA0In4D0LCeBkAADg3IYJwdDLGywAAQOeECMJRCEIAADgHMYIQV1AAAMA5iBGEyWx/I4IQAKAHdHobpn59byYhgjDTSiqnukCs6wEA0P91ehumfn1vJiGCkM5cTRjrWgAA9G+R2zC9//77s2bNmjFjxhtvvBEp7Hc3YNLF/8wyulHJbH8T/05mn5ipFgDgovn3/jV4uDw6r2Udd6V52Li2JZHbMN1yyy3PPPOMLMvPPffcqlWr9MKSkpL+dQMmnUBBiPEyABAHJHuSMW1AlF7LmnCuRePGjVu7dm1BQcGKFSsihf3uBkw6gYJw80nMOAoA/Z55SL55SH6sa0G/+tWvdu3a9cEHH7zzzjurVq3SC/vdDZh0ovQRjkomTC4DANAjwuHw3Llzs7Ky5s6de+jQoUhhv7sBky56Qeh2u0tKSqZPn75kyZLIzwRVVefNm9f1Oj0ix84agtwd6sFNAgCISL8N0+23337//fcXFxfrd1nSC/vdDZh00WsaLS0tdblcS5cuffHFFzds2DB//vxNmzZt3br1xIkTXazTU68uMRqexL5s5pPSMF4GAODiRW7DdPPNN3cs7F83YNJFLwh37NixbNkyk8lUWFi4dOnS+fPnDxkyZMCAAUuWLOlinU43parqJ598cvDgwXblgwcPHjJkiKIowWCQsfaBd0mS9M+60NhEDJnpFcEzYl0R4QSDQU3TDAZDrCsiHP1QI0m92K6GP6i2NE0LBoOKouiffDf/l8lk6hgH7UQvCOvr610uFxG5XK6GhgYiuvTSS8+7TqeCweDzzz9vNpvblU+fPv1HP/qRz+fzer0d3/kQq3lvLS/MUL75e4GOgsGgz+fr+KVAb/P5fLIsc9xzM+r0Q02vBqHP5+u9jfc7qqp6vV5FUbxeb/cPNUaj8bw/E6MXhJxzPZw455rW+QDO7qxDRDabrbS0NC8vr9OliqKkpqZ2DMLLBmjrD/HUVPtFvgHoUjAYNJvNqampsa6IcAwGgyzLNpst1hURTigUSk1N7dUgxIl+W7Isp6amKoqiP+jBLUdvsIzT6aytrSWiurq6tLS0i17nounX1PfsNgEAoL+LXhBOnjy5rKyMc15WVjZlyhQiKi9vPzlCx3V60LAkVunlQbVntwoAAP1b9IKwqKjoyJEjs2fPrqiomDNnDnU2uKjjOj1IlijXzg634KQQAAC+Fr0+QrvdHhlfq9OH0rYdUNtxnZ6lt46OScEVFAAA0EqUmWV0mF8GAADaif8gDB7cw5XWK04w9TYAALQT/5Nut7y/PumGueYRE4hoVDJbvRdTbwNAv6EoylNPPRXrWsTev/71r97bePwHoXnouODhvXoQjkxmh1q4ysmAXkIA6PMcDsf69et37twZ64rEXmpq6jXXXNNLG4//IDQNG+su+4P+2GakHDvbU88vw4yjANAf3HrrrbfeemusaxHn4r+P0Jw3Rjn5FQ+1zqz2g0HsreNoHQUAgFbxH4TMZJazcpVjX+hPb8qR3jyG8TIAANAq/oOQiMzDxgUP79UfX+lilV5+zIMsBAAAIlGCcOjY4Ff/1B8bGH0/W3q3EkEIAABEogThkDHK8YORbsKbctBNCAAArYQIQma2yq5BSmXrjXy/ny39pZq7Q7GtFAAA9AlCBCG1dhO2to7aZfp2Btt8EieFAAAgThAOHRsZL0NENw2W3jqObkIAABAmCE1D85VjX3A1rD8tzGHvVmoqohAAQHiiBKFkSTCmDwxVHtKfDrazTCv7ey2SEABAdKIEIZ19NSER3YyxowAAIFYQtrmakIhuGowpZgAAQKwgzFeO7idN1Z9+K501h+hQM7IQAEBoAgWhZEs0pLqUE4f1p4zo+9nsHUwxAwAgNoGCkM6+mpCIbhqMbkIAANEJFoRnX014/UBp12l+OhDDGgEAQIwJFoTDxilH95PWehZoM1LRcGnFHjW2tQIAgBgSKwilhCRj5mD/vr9FSh6daHjlkHbEjZ5CAABBiRWERJR4zXTPRxsjT9MtdN8Y6ZefoacQAEBQwgWhddyVakuDUvFFpKR4rGHLKW13PU4KAQBEJFwQkiTZryp0b9sUKbDL9Mh4Q8ku9BQCAIhIvCAkSrjihuCh8nB9VaTkJ6Okg8209RROCgEAhCNiEDKTJeHb0zyfvBkpkSVadpn08Ke4HQUAgHBEDEIisl99i2/Xh5rfEyn54VBJ47TxKEbNAACIRdAgNDicllGTvDvfj5QwohXfMizepSmIQgAAkQgahESUeN1tnk/+FLlVLxEVDGTjUtm8baqGFlIAAGGIG4TygDxjerZ/z/a2ha9dazgd4Pf9FSNIAQBEIW4QElHitTPdW/9f2xKTRH+83viP0/xXn6OFFABACEIHoWXUJGLM+9d32xYmyfTeDcb/PaL957+QhQAA8U/oICTGnHf93+Z3XwmduUmhLt1CH3zf8J//0tYdQhYCAMQ5sYOQyJg+MGXmT+t//7gW8LYtH5TA3vme4eF/qO+fwMgZAIB4JnoQEpF1wtWWkZc1rF9N/KzMG5PC/jzNeOfH4b/UIAsBAOIWgpCIyDH9Hs3b7PnkT+3KL09nf7jWeNuH4f1NyEIAgPiEICQiYgaj865H3B9uUI7ua7fo+oHsyW8bbnhPPeZBFgIAxCEEYStDSkbK7Q/Uv7Ky7R2adHcMlR4aLxW8p9b6Y1I1AADoRcZYV6APsYyZ7PD76tc9LmflOm68U84eFln0s9HSCS+/eXP43e8ZU80xrCMAAPQwnBGexTbpusyS/7HmT657+dHTLzwSOvFVZNHj3zJMzWL5G0OvHcY1FQAA8QNB2B4zGBOm3JhZ8jvLqEl1L5Y0bXxBC/iIiBE98S3Du98zPrtPm/pO+MtmdBkCAMQDBGHnmGxKvHama/EaYqxmxf/xfbpFL7/Uyf56k/HmwdJ33go/UY5bVQAA9HsIwq5IloTkGfc6f7TYvfWPdS8/qjbUEpFRol+MlXbdYvxrDR/zx/CmCoQhAEA/hiA8P1Pe6Izi35hyR9Y8/e/Nb61Vm+uJKMfO3pxm+O/vGJZ9rl3xZnhnLVpKAQD6JQRhtzCDMalgtmvhf/FwuGbVvY1/eCpUfYyIvjuAfTbdePcI6dYt6l3bcH0FAED/gyC8AIaUjOTp92Qu/h9jRnbdC4/UvbTEv/evUlj5PyOlL28zuqw08U/hD0/h1BAAoD+J0nWEbrd75cqV+/bty8/PX7RoUWJi4rnKy8vLX3jhhVOnTg0YMOBnP/vZuHHjolPD7pNs9sTrb7dPneH7/GPPjjcb//dpy5hv2yZMXXnZhO8PMtz5sTojlz35bYOM3xgAAP1BlI7WpaWlLpertLQ0IyNjw4YNXZQ/8cQTd9xxxxtvvDF79uwnnngiOtW7CMwoJ1xekP7TJ1yPvGwafIl7y/9WLb1j0hebdheyr1r4lW+Fj7hxaggA0A9EKQh37NhRWFhoMpkKCwu3b9/eRbnNZvN6vX6/3+/3W63W6FTvmzAkptivujn9/qcyfv6fwS8/V1+4f+OoI7cPka54M7z0M/UAZusGAOjbotQ0Wl9f73K5iMjlcjU0NHRRvmjRovvuu++ZZ54hot/85jedbs3n8xUUFJhMpnbls2fPvvfeexsaGgwGA2Osl97LuZno1p+zQ5/XrfnVHcPGX/HtOa9WOb77jinNpM0cFJo+UBlgjedQVBTF6/VqGi4mibaWlhZZln0+X6wrIpyGhgbGmCShFySqFEXxeDycd/dwmpycbDSeJ+miFISccz2ZOOdtj5Udy9esWTNr1qwZM2Zs3Lhx7dq1q1at6rg1i8Xy7LPPDhgwoF15Wlqaw+Hw+/0OhyMWQUhERJOu1UZP8rzzu9w/PPSrzMFLAz6fxxv43Ccpgc9SLpEmfW/yVZdbTHE4xWswGDQYDA6HI9YVEZEsyzabLda1EE4gEHA4HAjCKFMURZKk7h9qDAbDedeJ0hHZ6XTW1tZmZ2fX1dWlpaV1Uf7FF1888sgjqampP/zhD+fMmdPp1iRJGj16dF5eXqdLZVmWZTlmQUhEjlTzHQuVyoOap0WyJjgtNsliCxnMVX/ZRZ++deTD/z4wtGBMwQ1jh7YP8n5N0zT9k491RYQjnxHrighH/9gRhFHGOe/xHT5KQTh58uSysrK77767rKxsypQpRFReXj5+/PiO5Xl5ee+9996MGTM2b948dOjQ6FSvN5gGjWj71EB07Q3X0Q3XVRw5nrT5Pfrvn/9Dthsy83KG5SYMzJEzc43pA0g6/y8XAADoWaz7La3fhMfjWbFixVdffTV8+PCHH344ISGhoKBg8+bNHcuPHz/+1FNPHTlyZMiQIQsXLhw8eHDHreXl5W3duvVcZ4TV1dUulyuWZ4TdoIbDf9l/Ykd5hf/k0Sns+CX+Y2ZvnTFjkJw5WM7KlbNyjJk5xtRM6tvvoq1gMOj1elNTU2NdEeE0NzejaTQmampq0tPTcUYYZYqiuN1up9PZg9uMUhD2rDgIwohqP607qK09qCVw5SfOEzeajiU1HQ+dqgjVHNO8btk1SM7KM2blGJPTmcUmWRKYxSZZbFJiMjP0rY5GBGGsIAhjBUEYE70RhH3rYCqgTCs9NF56aLy0s9bw6uGhJUfyxqSwoqnStGw20OAPVx8LnToaqj6mVBzgfq8W8GkBHw94NZ/bmD5QzsyRB+QZM3MMjq72CR70awEfD/i0gJcHfMxiM+eNlrPyqM0fsOZtCXzxWeDAp+Gayrb/l1kT5KxcOTNHzso1ZuZIFhxtASDeIAj7iisy2BUZhmcmG96r1F49zBfvUlUuX+ocPsE5YsJ4dqWLDbZ/fY7LQ0q4pjJUVRGqPub96zuau6mLLTOzhZkTJItNstqYJUGrO+Xd8ZbaVCcPvsScN5oYCxzYFa49YR4+zjJykv2aW4i+fiHN2xKqqlAqvvDufC9UfVyyJhhS0g2OdENymjE1w5DqkrNyjc6sftSECwDQDoKwbzFJVJgjFeYQkaHKR7vr+Z56vqmC/8ff1FQz+142mzZQmprFbLJJzh4qZ1/8YCLN51YqDihHD3At7Pi3eaYhY87V1moZeVnrI87Vlga1oUZtqgs314UbagIH94Srj6nuRtk1WMoYpCZn+AYMNqS4jMnpkiP1QhtvtYBXbTytNp5Wm05rfq8hKcWQkmFITjMkpzMjhkQCQG9BEPZdWTbKsrEbBzEi4mTYU8/LTvAn96o/3MonpbHrB0rXD2SXpTHDRZ2MSbZEy+jLLaMvv4D/w5jB4ezYDMuD/lD1MX/lYfVURWDfP8KNp9XGWs3dKNkS9R5NZrVLFhtJElcCvLV116cFvGdtRFWJyJiSoYefZE0IVVeoDbVqU53adFpKSDJmDtZbaNFICwA9C0HYPzCiCU42wckeHi95w/RxFd9yUpv/iXbSx6/Nkq7KZJelsUudLDEWJ07MbDXljOSZeWrbwTKapnqaedCnBbx67yZpGjOZmcUmmW3MYpOs9rM2YjAw8zlm1ONcba4LVR//upG2ptKY6jLljTbnjTbljTamxdUVmQAQZQjC/ifBSD8YxH4wyEBE1X7aclLbWcv/94j2rwY+yM4uS2NXZLDrBrBRyTHtt5MkQ1IKUUoPbIoxQ3K6ITn960ZaTVVOHlGO7g/s/7T5nd/zcEgekHdmUE+eMSNbstm73CIAwNcQhP1bppXmDpPmDiMiCmu0v4l/Vsd3VPOn/qkFVH5tlnTdAPadTDY8iUnxNJxFMpgGDTcNGk5XFxKR2lwfqqoIVVUoFQe8O98L157Qgn7JmnDmapMEQ0q6Ibl1gI+UlCrZEqXW1lrMYAAACMI4YpRoXCobl8rmjSAiOurmH1Xxraf48j1aXYBPTGOXpbHL0lh+CstNZElxNPpE77n8+nyRiDjX/F7N79E7I/UBOOGa44EvPleb67jfq/k9WsDHjMZ2w3CYUTY40gzJ6a3ZmZJuTE4zpLgMSanUm5eLcTXMg4GzizQt4NV8+lvwMYNRH/SrNyyftaLfS3Jy79UNIOa0gJe01kveNUUhTe3Z7SMI41ZeIstLZHePICJqCNJndfyzOv5GBX98j3bMw2WJcu0sx86GO2h0MstPZaOTmS1udgfGJJv9vA2kXAnycOisknBIbTqtNtWpjbVqY61SeVBtqlMbajRPs5SYYkjJYHL7e56c9bJGo2SxMUuCZLVL1oSvO0Qt7dOLiNSm06HqY+FTFaHqY+G6U+23zCTJksCs+nUvCTwcbr2ENODjwbNuNMFVzWuU/UNGm/LGmPNGyYNGYJBtJ1p/G7l5wMe7vEcKDwW/HtLl93SxpubxeBISupi7g5lbv3rJYuukCzwcirxK5Ms988DfflMGo2SxMWuCvl99/X8DPs3vbb+yJDGLTbIltu6NZ+qgj1zjwYDaWBtuqlWb6tTG0+3+BM5LstmlM9s8z5+Dwai/umSzM4uNmKQPF2jdjUPBtitzTeUBX5vfr76234I+2i6ysjzrF5SecUHV7hpmlhFUXYCOefgxDz/YTP9q5Psb+RfNfICNXepk30pjl2ewy9Iu7KwxvmeW4WpYa2kIN56msNLVavrhye/VAl7N7239g9f/noPtD21SUqo+nZ6cmWN0Db7o9Gpubjb4WqSqr5SKA8rR/aGaSkPiWV2zTDZFLkQxpqQzk+XiXqgjHlL03w3hptNqQw0RGV2DWjtrB+QZnVldnEbr8zzokzyc/aH52hziPfpnSKrKzBY9VySLTUpIMiSnGRxphtQMQ3KaZLaGT58MVR3Tr6xVG2rPeqGwogV8XAlI1gTJamcWG+uySZzJpjO/XRLaDelqx+v12my2Lg41re8i4ONBb/szfiLSsy2SWHpq6j99Ovxs4mq49cNpbcyQz8RbgmRN6LCy2rqm/l+CkU/YxwNeJpsNqRl6v7sxJb3rMGu/Zc4jDSo84Os6RLkabk13n4cHfMQ1/d21ZrNsPmttSZIsNslqj3wCkfaPdr9oMbMM9Jg0C6VZ2GVpX/8Nq5wOt/DddfzTOv7oZ9qeep6dwC51spHJ7BIHjXCwSxxxdMp4gZjBaEjJMKT05I/QHiQ5nLasQbaJU4mIKwHV3dh2KVeCamPrhSiBQ+Xtfol/E8woG5LTjVm5ltHfMiSnE1Go5nioqsK368NQVUW4vpqZLK3Ha7ONGWUt6IscoM9aZLFJNrt0ZhJBY0p62zMJyZJABgMPBiIX3mjeFrW5PlT9mdpQqzad1oJ+Y9oA/VeFdewVhlRX2xkemNHU+QnZNxaoqUnCFGtxQdQDG3RgYHSJg13iYD8cSnRm6M2eev5lM//jUfqyWTvUzDNtbEwy5aey/BQ2JoWNSmYWDDfpY5jJYnRmtSuUs3Kj8+rywCFtn7Y97ePhUOvwJatdsiZgNiLoOxCE0LnI0JtIicapwsP/2cD3N9E7lXz1Xu1gM0+3sGFJNNzBcm0s3WDMcGs2I0uQySRRuoXaTgsHAmJmq8Fs7XouXICYQxBCd0mMhiSyIYmsMKe1ROVU6eGHWuhwC/+ykf7RIqs13K9qnhCFNDrl475wa5qOS2XjU1l+KkvAHgcAfQwOS3DxDIxyE1luIhUMZMEg93r9qaln9cTUB6m8nu9t4H+v5S99oR1oah2PMy6VjUmhXDvLTWSp5nNtHgAgGhCE0IucZrpuALtuQGsDaVijQy18bwMvr+frD1GFR6twc5VTjp3lJtKQRDY0iQ1JZEOSaEgieh8BIEoQhBA9RolGJbNRyez2NiMqmhU65uFH3fyImw4187IT2hE3Vbhbex+HOdiwJDY0kbJsLMtGmVZmxT4LAD0KBxWIMYep/agcItI4VXr54RY61MwPt/C/11K1Xzvlo2ofNxtooI2NTGYjk2lMChvpYCPjaSoAAIg6HD+gL5IY5dhZjp2+O6D9uNNmhSq9/IsmfqCJ3jrOVzdpXzbzTCsblUyjk/WAZEkyJciUbGI2I6GJFQC6hiCEfsZhIoeJ5aecNRVAhZvvb+L7G2l7NV/7peYOkS9MjQr3hUnjlGFlA2yUaWUuK6VZKNnU+n/1yzysRrIZyGFidpkSjJRhZa6ev/YaAPouBCH0ewZGQ5PY0CR20+BOlioa1fr5KR/V+Hm1j+qC1Ki0Tit4wkuKRr4w+cLUEtLcIfKGqDbAG4OUYWVZVsq0kdXAiEiWyC4TEVmNZDFQosxkiZJN5DBRppWlWynNzNKthKsmAfojBCHEOZNE2QksO4Go2zkVOpOd1X4eUFtLPCEiIl+Ygiq1hHhYo69aqCFItX7tdIBOB3hjkNItlG5lGRZyWVm6lbKsLNNGLisbYKMMK0s1kwmzcQH0PQhCgPZkiQYmsIEXkp1EFNZIT8RqP9X6eV2Aqv18fxPV+rWTPqr184YgGSVKkinJxJJkSjFTksySTJQkk8NE6RY2wEYuK8u00QAbhv8ARA/+2gB6hlGiLBtl2dg4onMlqC9MLSFqUXhLiJqC1KzwlhC1KNQSokMtfFs1Vfu0aj9V+bgvTClmMjJKlJnFQHaZkkyUYmoNzmQzSzFRiplSzExWpLQElsG5nq8AcKEQhADRYzOSzUiZ1khMdnXG2RikkEaeMPeHyROilhA1KbxZoRaFmhR+sJkaFWoManU+U3OIeVS1ReGeECWbv+7C1Ls2GX09PshiIP1CzBQzEVGCkZkkSjG3Vkw/Q9XvTIJMBXEgCAH6KD2rMs4Ky06Cs7nZK8uyzWYjIo1Ts9LahdmkkKKRN0ScqOnM+KCASv4wEVFjkIjIE+beMB1xky9M3jC5Q1qzQqcDVBfgQZWcZkoysUS5NT7tRiZLJDFynLmBXaSZVz9PTTGzNAs5zeS0YGIg6E8QhADxQ2Kt7aUdllzwgNagSvVB7g6RJ0T+MAVUcod4mLdmrU5v5j3i1h9QQ1CrD1J9gOqD3Mgo1cxkiejMmFs9QZPPBGeCsbXV12qkRJlsRnKamdPS+q8BAzN7+KAAABfGSURBVHAhihCEANAJs4EG2NrF0QWkkydEjQoPaURnxtyqnFoUalS43ifqDVGjws9ELHnDVB/Qc5TXBchhIgMjh4kRUZLp61xkRMlmIiKrgdmM5DCRfvVnZPyRw8SSZEowYkYFuAAIQgDoeXaZ7HKnwXn+NOVETUFSObWEOBG1KKTysxYRkV/l3hC1hFpDtNrHD4aoKUhNitYSIl+YvCFqUrgvTPoFMO2YDRQZl2tglCQzIkqUW4f1JshkM7JkE9llGmBjmVYaZGdZVspOwFS38QnfKgD0LexM/2iapYvU/EaNp0GVfOHWx2FO7jaJ26yQN8x9YWpWyB2iUz7+eR1VerUqH530cb2HVW/m1bTkFIuWYtL05ugUE0mM7DLpDcKRfCX6utBhIomR1dDahxpQyR8JeSIikqj1PJiIbEYyG85syqT/dyadmdUBehCCEACEYza0ZowuvX3inidl9Y7S06dPGxJTm0NSo0KNQd4YJE7kDlFYIyJS+ddzGB33thY2KcQ5+VVNP0+1GFqnLvp6y0TNiqY/1mdv0DfVohARNStca1MekWAkk6G13pERwl2LpKl+chwJYH3eQWrTCp1iYq2Dik1n5bS+sj59hL2fjzFGEAIAXBh9UJIia+mJTJL0YIjl8B5vmBSViM4aIdw1/5lGY30gsUbUrHAi8oZID2K9FZoTNSr8hJd8YXKHzsppIvKEqNpPp/1co9ZZBhOMrcOg2vbsElFYI06tp8VtBx5H5mxqRx9g5TAxm4FsxtZI1mM4HGbDTMx5oZ9RlxCEAAD9mx4/utROxgx308VnuS9MtX5eHyRvWI9M3rZnl4iMEjEiffBU24HHkVl829Gv/GlSeHWY9GZqOnO6rGl86Shp2EXXtTMIQgAA+EZsRspNZLmJkYJePD9WFMXt7mwE1DeAOYABAEBoCEIAABAaghAAAISGIAQAAKF1Kwhvu+220tLS3q4KAABA9HUrCCdOnLhnzx7Ou3V5CgAAQD/SrSCcNWtWKBRatmzZ3/72t6NHjx5vo7frBwAA0Ku6dR3hT37yE/3Bjh072i3avHlzD9cIAAAgiroVhJG0UxSloaEhJSXFbDb3Zq0AAACipLszy2zbtu211147evSo/jQ3N7eoqOjqq6/utYoBAABEQ7f6CD/66KPly5dfeeWVv//9799+++1169ZdeeWVjz322LZt23q7fgAAAL2qW2eEpaWl06dPv+uuu/SnAwYM+NGPfuT1ektLS6+55prerB4AAEDv6tYZ4cmTJ0ePHt2ucMyYMZWVlb1QJQAAgOjpVhAOHDhw//797QoPHDiQnZ3dC1UCAACInm4F4e23375p06ZXXnmlqqpKUZSqqqpXXnll06ZNt99+e2/XDwAAoFd1q4/w2muvlSTp1VdfXb9+vV6Sk5OzePHiqVOndvNl3G73ypUr9+3bl5+fv2jRosTExHOVq6r6/PPPb9u2LTs7e8mSJWlpaRf+pgAAALqru5NuX3PNNS+//PLbb7+9fv36t99+e82aNd1PQSIqLS11uVylpaUZGRkbNmzoonzjxo1er/e1114bM2bMunXrLuS9AAAAXLBunRHedtttt9566+233242mzMzMy/iZXbs2LFs2TKTyVRYWLh06dL58+efq3zr1q0PPvigxWKZO3fuiRMnOt2aoijr16/veLKYn58/adIkv9/v8/kY68VbJENHwWBQ/+RjXRHh+P3+UCgU61qISN/hJQn38IkqRVEu6FBjsVjO+x11Kwj1SbdnzZp10elSX1/vcrmIyOVyNTQ0dFFeW1v70UcfFRcXZ2VlPfjgg51uTdO0PXv22O32duV2u338+PGhUCgUCiEIoyx0RqwrIhz9M8cnH336Do8gjLILPdR0Zx60bgXhrFmzfvvb3y5btux73/uey+UyGAyRRYMHD+7OFjjnejJxzjVN66Lc6/VyztesWfPmm28+/fTT//Vf/9VxaxaL5amnnsrLy+v0tfx+v8PhQBBGWTAYNBqNDocj1hURkSzLNpst1rUQTiAQcDgcCMIoUxRFkqSePdREadJtp9NZW1ubnZ1dV1fXtkmzY3lycvKMGTOcTmdhYeGmTZu69SYAAAAu1oVNun3RJk+eXFZWdvfdd5eVlU2ZMoWIysvLx48f37F80qRJH3zwwcyZM995550RI0Z8w9cFAADoWpTuUF9UVHTkyJHZs2dXVFTMmTOHiIqLizstnz9//u7du2fNmrV79+5f/OIX3+RFAQAAzot1577zK1asaGlpefzxx/tIx1teXt7WrVvP1UdYXV3tcrn6SFXFEQwGvV5vampqrCsinObmZvQRxkRNTU16ejr6CKNMURS32+10Ontwm7hDPQAACA13qAcAAKFd/GCZLVu2rFy5sqfrAwAAEFVdNY0WFBRs2bKl7VO0hQIAQJxBNy8AAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQjvPdYQrV65se7Fg5Ia6AAAA8aGrIMSsMQAAEPfQNAoAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgtGkHodrtLSkqmT5++ZMkSt9t93vKjR4/+27/9WxQqBgAAEI0gLC0tdblcpaWlGRkZGzZs6Lrc4/GsWrUqGAxGoWIAAADRCMIdO3YUFhaaTKbCwsLt27d3Ua5p2qpVq+64444o1AoAAICIjFF4jfr6epfLRUQul6uhoaGL8tdffz07O/uqq67qeoMej2fUqFGMsXbl99xzzyOPPHL69Gki6rgUepWiKD6fLxQKxboiwmlpaZFl2Wq1xroiwjl9+rSmaZKEkRZRFQqFPB5POBzu5vpOp9NoPE/SRSMIOed6LHHONU07V/nnn3++e/fulStXnneDdrv9o48+ys3NbVcuy7Isy6qqpqenIwijLBgMer3e1NTUWFdEOCaTSZZlm80W64oIR9O09PR0BGGUKYpiNpudTmc31+/OFxSNIHQ6nbW1tdnZ2XV1dWlpaecq//zzz8vLy2+44QZ9aUFBwTPPPJOfn9/pNhMSEux2e6eLJEmSJAlBGGXSGbGuiHDwyccKPvmY6I2PPRpf4eTJk8vKyjjnZWVlU6ZMIaLy8vKO5QsWLNh8BhFt3rz5XCkIAADQU6IRhEVFRUeOHJk9e3ZFRcWcOXOIqLi4uNNyAACAKGOc81jX4YLl5eVt3bo1Ly+v06XV1dUulwtNo1GGPsJYaW5uRh9hTNTU1KCPMPoURXG73d3vI+wOfIUAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQjFF4DbfbvXLlyn379uXn5y9atCgxMbGL8u3bt69bt66uri4vL2/hwoXZ2dlRqCEAAAgrGmeEpaWlLpertLQ0IyNjw4YNXZRXVVWtXr26uLi4tLT0iiuuWL16dRSqBwAAIotGEO7YsaOwsNBkMhUWFm7fvr2L8qqqquuuu27kyJFms3natGmVlZVRqB4AAIgsGk2j9fX1LpeLiFwuV0NDQxflEydOnDhxIhGpqrpu3bqpU6d2usFgMLh8+XKHw9Gu/MorrywoKPB4PDabjTHWO+8GOhcMBn0+nyzLsa6IcDwejyzLqqrGuiLC8Xg8FotFkjDSIqoURfF4PCaTqZvr22w2g8HQ9TrRCELOuR5LnHNN085bvmvXrjVr1kyaNGnevHmdbpAxZrfbk5KS2pVbLBbWRs+/Ezg3fOyxon/m+OSjD/t8TFzox96dNaMRhE6ns7a2Njs7u66uLi0trYtyzvnLL7984MCBkpKSLobJmEymBx54IC8vr9OlHo/Hbrdj74wyWZb1HyixrohwVFWVZdlms8W6IsLxer12ux1nhFGmKArnvGcPNdH4CidPnlxWVsY5LysrmzJlChGVl5d3Wr53796dO3c+9thjTqfT7/f7/f4oVA8AAEQWjSAsKio6cuTI7NmzKyoq5syZQ0TFxcWdlpeXl584cWL69Ok3nxGF6gEAgMgY5zzWdbhgeXl5W7duPVfTaHV1tcvlQtNolAWDQa/Xm5qaGuuKCKe5uRlNozFRU1OTnp6OptEoUxTF7XY7nc4e3Ca+QgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKHFYRDedtttiqLEuhbC+fTTT5cuXRrrWojo2Weffffdd2NdCxHNnTu3ubk51rUQzt69ex966KGe3aaxZzfXF/zjH//QNC3WtRBOQ0PDF198EetaiOirr75yOp2xroWIPvvss1AoFOtaCKe5uXnfvn09u80oBaHb7V65cuW+ffvy8/MXLVqUmJh4rvJzrQkAANAbotQ0Wlpa6nK5SktLMzIyNmzY0EX5udYEAADoDVE6I9yxY8eyZctMJlNhYeHSpUvnz59/rvJzrdmWpmn79+/v2Dpvs9nsdjsRVVVVWSyW3n5T0FZDQ0MwGDx16lSsKyIcv9/f3NyMTz76NE2rrq4Oh8OxrohYGhoaQqFQ93f4jIwMo/E8SRelIKyvr3e5XETkcrkaGhq6KD/Xmm2NGjXq5ptv7lhutVoTEhLMZvMVV1zRG+8CuqCqajgcnjBhQqwrIpxQKPThhx8+++yzsa6IcDRNu/766xljsa6IWDRNC4VC3T/UbNmyZezYsV2vE6Ug5JzruwvnvO1Ilo7l51qzrffffz8alQYAAAFEqY/Q6XTW1tYSUV1dXVpaWhfl51oTAACgN0QpCCdPnlxWVsY5LysrmzJlChGVl5d3Wt6xBAAAoPdEKQiLioqOHDkye/bsioqKOXPmEFFxcXGn5R1LAAAAeg/jnMe6DgAAADETPzPL4Er8KNu+ffu6devq6ury8vIWLlyYnZ1NRPfff/+BAwf0FX7wgx/8x3/8R0zrGIc6fsLY86OgoKCgXcmoUaOwq/c2VVUXLFjwu9/9jnp59pX4OSNcs2aN3++/5557XnzxRZvN1ukFiNBTqqqq7rnnnlWrVuXl5f35z3/+y1/+8uyzz3LOZ86cuWbNGqvVSkQGg8FkMsW6pnGl008Ye34U+P3+yOPS0tJQKPTee+9hV+9VmzZt2rp165dffrl582bq7Ajfg3t+/Ey6vWPHjsLCQv1K/O3bt8e6OnGuqqrquuuuGzlypNlsnjZtWmVlJRHV19erqrp48eJZs2atWLHC6/XGuprxptNPGHt+FFjPqK6u3r9/f2FhIXb13jZkyJC5c+dGnnbcz3twz4+fIOzOlfjQUyZOnKi3Bamqum7duqlTpxJRQ0PD8OHDFy5c+NprryUkJLzwwgsxrmXc6fQTxp4fNaFQ6KmnnvrZz37W1NSEXb23XXrppZMnT448vbjZV7opfvoIu3MlPvSsXbt2rVmzZtKkSfPmzSOiESNGPPnkk/qiBQsWLFiwIKa1i0OdfsLY86Pmj3/848iRI3NycogIu3qUXdzsK90UP2eEuBI/mjjnL7300muvvVZSUrJgwQKDwUBEBw8ejNweRZZlWZZjWsc41OknjD0/OlRVffvtt6dPn07Y1WOhV2dfiZ8gxJX40bR3796dO3c+9thjTqfT7/frQwkCgcAvf/nLY8eOhUKhV1999corr4x1NeNNp58w9vzo2LNnT3p6+sCBAwm7eiz06uwr8TNq1OPxrFix4quvvho+fPjDDz+ckJAQ6xrFs1deeWX9+vVtSzZv3sw5//Of/7xp0yav13v55Zffd999+BZ6VqefMPb86FixYkV2dnZRURGd44uIdQXjU0FBgT5qtON+3oN7fvwEIQAAwEWIn6ZRAACAi4AgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBo8TPXKEC/0/Eudzr9CuKefaG1a9cOHjy4ZzcLEB8QhACx9NBDD8W6CgCiQxACxNL1118f6yoAiA5BCNAXHT9+fP78+a+//vpzzz33z3/+Mz09/bbbbmvblPrxxx+XlpaeOHFi0KBBs2bN0m8JqduxY8cf/vCHysrK3NzcmTNn6ouCweBvfvObXbt21dXVjRgx4v7778/NzSWiqqqql19+ee/evcFgcMSIEffee++wYcOi/GYBYguDZQD6ruLi4oyMjPvuu2/kyJGrVq3atm2bXr5t27bly5ePHTt24cKF+fn5y5cv/+STT/RFf/vb3x577LEJEyY8+OCDeXl5y5cv13scH330Uc75/fff/9Of/rSpqWn16tVExDlfvHix2+2+8847f/rTn5pMpqVLl+KmhiAanBECxFKn42Uig2Wuvvrqu+++m4i++93v2my2119//ZprriGiV199de7cuXfddRcRTZ061Wazvfrqq1dffTURvfLKK3feeeecOXOI6Kqrrjp58uQHH3xARJdffvm///u/65sdOHDggw8+SER1dXWVlZWrV692Op36pl544QWfz2e323v/rQP0FQhCgFhau3ZtF0tvuOGGyOMbb7xx48aNiqJwzisqKpYtWxZZVFBQ8NprrymKQkSHDh1asmSJXs4Y+/Wvfx0Oh2fMmNG27TQlJUW/7YzD4UhOTn788cenT58+fvz4xMTEhQsX9uj7A+gHEIQAsdT1JQ1t77udkZFBRA0NDfpT/Ryu7WqdLrJarfqDlJSUjts3mUxPPvnk+vXrV65cGQgExo4du2DBgtGjR1/cewHop9BHCNB31dXVtXuckpKiR1p9fX1kkf44JSUlOTmZiJqamiKLqqqq9u7dS0SMsU5fIicnp6Sk5I033nj22WcdDsfixYv1M0sAcSAIAfqusrKyyON33nknJyfHbDabzeacnJy2F91/8MEHubm5ZrPZYrEMHjx469atkUXPPffcunXrzrX9pqamoqIir9drNBpHjx79wAMPeDyelpaWXno7AH0TmkYBYmnLli2dlo8YMYKI/vSnP3m93tGjR5eXl7/77ruLFy/Wl86dO/fxxx/3er2jRo06cODApk2bIv2CRUVFK1as8Hq9w4cP3717965du0pKSvSTwo4cDofBYPjlL39ZUFDAOd+6devgwYPbtqwCiIDpfeYAEH3nmmKNiNauXTt//vyXXnrp+eefP3ToUHp6+qxZs6ZNmxZZ4aOPPiotLT158uTAgQNnz56tjybVffjhhxs2bDh58mRWVtasWbMKCgraTbGmX6Son1NWVFT89re//fLLL4koPz//3nvvHThwYG+9YYA+CUEI0Be1zSoA6FXoIwQAAKEhCAEAQGhoGgXoiwKBwOHDh/Pz82NdEYD4hyAEAAChoWkUAACEhiAEAAChIQgBAEBoCEIAABAaghAAAIT2/wHaor3owMAfMQAAAABJRU5ErkJggg=="
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is the error plot, we get to about 7.5% test error, i.e. 92.5% accuracy\n",
    "trnerr,tsterr = Array{Float32}(lin[4,:]), Array{Float32}(lin[5,:]) \n",
    "plot([trnerr,tsterr],ylim=(.0,.12),labels=[:trnerr :tsterr],xlabel=\"Epochs\",ylabel=\"Error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualizing the learned weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAA4CAAAAADGVp33AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AACLqSURBVHja7d1pmFxneSb+X+3VXb0v6lZr3yWrtVteZMs2lmUZL9jGSwzGEMgyJAEzMA4M80+CIWEmJObPJIGAE4gBYww2tjHYYLxIXrXYlqx9l1pSq6VWd6v3pfaaD33wNXNd82H0ve9PVdfpOuetU/3e53mf93nuO/Q3SiIiCkqKwkoiSkJKSgqiQsgKScqIygqLyopISIuKyCAiJi8kL64gJGIMcSUhBSEhIXkhIRBVCK6SFxFSUBJWVJSQExKVExZCSVZYRFRaBHFpUWFFMRlhRVE5xOUVxBUVCc4VEhJWEFJCSElIQkFBTkxeUZkSIgpyokKKIkLSYqIyIjKSSkoIC8uIoiCGfPAdosF484oS0uLCiigpCokpCsuLBGPMihmTkBNRDO45IXkxBRF5RJB7/7O/v3opuINhBWF5YSVxaTEh5EVAMbir42eNBMeiwigqCCsKC8sGxyKyYtJiChLywZ0KB39fQin4NaLIB/c1bAITuABEickpioy/kVUK+KYkIm5QVEJBVlQpmFkhMTkFieAdGRQUZOWEJTRpsceYqJCIEgrKZEUDpolIB//LJSHhgAUywauSUDALwzKKCqKIyQTc12tA3iQhA85oNtVBRTPEFRREFMTF5ZXLKQXsEZYVUxAXkZGQlxdSVApGURIxFsy2cQaMoqioqFGNlH45cbOcc0q5fhWS8kpGlRmTFwmYKKIkJq8gKiMkq9yIIUkCJqaghKKQgmgwk8fnb0FRPGDdkrwyo6IBLxBVFBURklMmryQsYpwtQkJK4jJiRuUkZcUNiUkG/BFWkhNVVBAxpkKa4KohWTFxFXJGJdXqNSIsI25MXEFRXsUEw0zgwhANK0rIBrMqLGdIQl6DlcYsd9IeA2r0C8uLBhFGUURCXlROSVFJSrNRNSIGlEwSFjeiQ4uQEeWajCnKihoVlwsijnzwukrMeQXnxUUMqFEuKyQhKY+iiIxYMAtLuvUZ0KbkmLl6dBuVlZIQERVDXFJWKWCYkKKkgpBRISNOIW/YJNM0y+rWI61cTFnANiXkxWQl1ZmnTrfLRHFWVIcx5SJqdCkZcdxhBeGAuxJBHDEqJq8kYwxFoyJBVBEXlxEN7n1USFZBSEFcLojrxpTkjMoGUVdMrVEJIWE5IWOiYrJCQXQRE1aUEZVWEjIsbiyIjoqSshLywsH4CmJywsJiRpUCth02aNCAMXEpcQkxWSlFIVEJ6QmGmcCFIUpEXklSFoP6DDmh09XSzjhjip3azFGQFlEjoTqIp4sK0uKIqlPlPmlJZ0120AFTsVHBbg2yckgJiQTrjIiChcakpHSb6jq8Y7V+bVLC+jV5yFnlWiQVxJVEZBUkpA0JOWWLMmPYZx8Yk3KZGikhJVk5peBpXxIXcwYpWVlh/ZYalXaRDiWLtJosar9tKlQZMCoqGsQEUTXyhjTqEHdCjR5hLxn2ASHPW26myxS9Y5J+eSUFxSAmyBmSN+ScKmFFcVXiyoQNSAhJKEoGsVNRQiaI4cbHXJQ3rNkpIXmTTPeKSgtUijmrICQjIa0oJBFETXlFY8YM6pWWldRgpl5hFbLKlMTfj0eLygxql5Y1T1ZcvX7rVTiuWtaLOjU4brJa1cGadYJhJnBBiIaD3AXVCnIWOWipcjEbHDSsT62smx0z4rA+e11kgQJBRqTHNJcYEVdwTEaVJ11rh30K5nlXrdedVS9vSbACCMlLqrVMQr8lTptl1JjV3pE0269c7YCXDKjwtvXiICcmIq5cXIcpujVoU63apfboskqzXrdYI2rMOR3eklYmLCKvoM8OTLVZi6t8wUnfd6Nb/UajlV719x7U6R6Lve5tJTkhRTFREYcU1TiixX5nTRFx3A1mecYUk53TpM/PNOhTKa0kLyqjYEi/Pm1OymkS1ScrplKLaeJmGhRTkBUJuKEkoSDvvApzLZaRtcIvfU6vlK2irrHEHuXaLdbrnJxMsA4tCGHIoG5j+gwIG5RzzmTv6DdDyjy1Uu/HhAVhg4pysqarNs1pa/xWxMtavelGrbrkTNIs45AmDcGacgIT+H9GNCsaZABnyOj2mBE1jjunRtqtfmqvD9ihyc26HdPiqFGp93O3LSocUTTd11VbY6+0Z2VdZa+cKeZZ5ahdegwoF1cSFjXdYotkpD2uzLtCRlU4ZK4GG52zwF4RYbUGlYnKoyCiKKzGVL1WmuGwcnN8x58Zk5K2SsGPFa12he3OqwrWOiUjtjpv0AwbLLDcDsfc4Jj7fEi1V2z0b35jmTQW22NYSFjEgEneMU2H61Sq9mNrbXSf3Z6wwXotdqv0hD9ymz0agxgkKqxksaKQudIe16lVxiFDYrYZtgXzzdKkQZmQnEiw+hlzVLkWJ6Q86a+dlLXdYY1u8HG3ySnY7aha851BXE5JSEbagAEjzprue45q8KCP+byFSpYZNuBlM8w1WUwoWNvG9TtnqTKTtLvMER1W2OC0xcrcZqs3tSqz0aiwSvEJhpnAhSEaV5JWgR1G7DAobZE13vU2XlenQ9xmN3tOUYOIcolgz4SYxbqUYaelJnnYVBEFc+y2VUqnQ8r0esNsy8QVg1xqzDm/Ns0mzear9YYjPusYjmhyl0+73kKHLBA3pM54jjNvyDTT5ZVLqzfmoMMy/kGXeitt91Hn5UxFRplKxSDXe9ioy7BA0XEnlJnloPu84KgeLf6zXfaaDerNMGI0yAT3C9tssT26fMtqVYZttMobdvmheeo0itmr3jT9UiBhrh7TtRpQ7x33W+CQH7pEszF/rk230y5x1NviwkEmJSFnWJ9K/YaUO2aGn/iCCm/7A0/aY8StHnfKkC87pU7YqKSYvFHdtoiLm+c2p/TJm+8xexwzKuJZDbZ5zYBTUupFFCSCTE+rGrUO+qUB3TpNlrfDxdrFDGt1uwrnPKJc0dwJhpnAhSFaUFQyIm6H4yY75nIJb7vJRss8ap7FXnG9Jvtl7LbJAstkhFEjYpUdZnnMhzyl3wav+Lxfec4aT/m4EWdscLXnHZcxplxeUr/z1qj0mgZRB+014npf9UWH/M5Kf+UTznnQWmeVLBCXV1QI9sxnOSfmjD4dlvq+hXqt0GGJmIQWlUZ0miEk4pSwkqyMXardpeAmz1jroILjhr3qfl3yfqLeVaKmgWEl43vgaRWave1ap4Rc7FInlSw0pNskSYfFLXdcjw85aJq4kqhqSxx3QJmZvup6Rb/Q6Fb/bpWUSVZ7w6ccFZPUJyspEqyrwiqFZYV8wBNWedxnrfKwjyr4O5/0Te0+YbsZptgc7PCMSui300FX6zZDO3a5VYteN+BVx5SLajJXyah+ERUSckoKMurV6bDZF+S02qfTMTHv+XNdKp3xz6Ju1+sFuYkYZgIXimhJUUxZkHmsNMMWNXr1WafoTz3ha57Q5TVpFTZinqGgTmZYVIOMX2qxWckfOmbIt8221E5f8AM/sc02If/FN3VpVBQyJiGuyiv2q/K2Lvc5rM9Sb0pbpN1/87oyn/JHpmGBjJiCgmqjepyRVuaQm1Wb41qDzqm1zxGr7NSk2qPKFCzyK3ERgiqOIW12qzbNXiHHLfWkWY4bNWKFZfZ61wfAZAeFZBXFpfSYbotKTbod1e1mv9SlQYdDqHXGUTu8rFneHGH1yjUpqJTR6C8d8aLbNfkbf+oiW2x3kzFdfuGTqvzOoApjyoVQgyZ/YNAb1htzpatM8q6H3OWYlMul9KvU5hIrTLbDGVlj9hhyiVs97MN+IGTIDR6y3Kvi3tVqn7d9xYAuSUUhYYRQJqPXqCHrLfQbh6z3stslXWKbUw5bY1TGxbr8XGwiDzOBC0U0JCKnoN9OSZWWqNZksUGN6v2p+bZJ2OKceY5jngZExZV0muN1V/mpWiVXy3rJYiHVVphnh//iSSGd5hrw++xETFaVWapVWmiKmQbs8bhlyn3JPB22+LXL1HlPrau8rqggoSgmb7+0kLPKTbfdgEFlltmm3zdsNsWoWYZd4qRnTFOjUZigmi2qxnK8apX1dnnZOVs86y7zveaYzzosK45qCdngU0lHhFxhv73aXa3MU5YY1aMHtJhlt0ZZh00OYp9yOVXOqPKUnFqdtrvbClc6bbKbHLDSj31c0Um1IlJS+kXljCJr1CErhB22yI+skPQdBR/wE9eCvFl6VepySC2GhZR8wDolr5nuWgcs1GObdR7SIC3qTp9zidPiIuqCar6kuEYx7cb0iLnGMzZ7wKN+5EENplulzQ+UfMwGtYoTMcwELhTRqIKwPoPijgmbI2m2KY76oCVYLeY7pjmuTEGtiHpFJXkhcVG/dYmnfF3CSxa5WsRT1nvYbLd6yTHTzHFIVoV6cZSE5c213e/MsMsuv1Ym7Hnf945XLVepUY+8A3a7QYdq5GSF9RgUs8mYS3CpsLjPux8f8SO3O2eNpy0X8UWzpZS9X6eacKsKx3xCixudscMyJQstscVjylSbhwFXgCWeDnZ5Mk4KqVJrtU1uc1CfW+wxrDyoda20w7A7nfEiiEibrVxWndM6vOcOJY/4un+1xWpT/VKT+b7lpG4HjZiraEhJER0+bod205022x2OeNC3Xea33nTc5yy0W9SlTntCpys0CRsyKmqSBQZ0a9VjF1IedLdNGu2zSrMul0uZ4XVXCQU1d5XS4uYbFFJtqn9zr722muESgz6s3T9bbYOLjFcHUphgmAlcGKJpMQlp9RYakHZSzgGn9KsG57zoFptcY8xMZRLiBFWyKYd0e1rJW8H8GscLytziJn/oQX/nEkd0utla5wwElbJRRbu9ZauYhIRbTFGuXsljvuZxtRY7a7f3VEkiLCrjlEbHfNAsXWa7wT3+xTX24F4fc8wL9jquXT2OmxrsrhblDZiv0Vpt8l6SM8k2H/FN72lxk2letMkm3ONytEnIoUK/nFV6TLXXV7xq2H3qDdrlbs9qtkanGxzyRb/woqNWKKi0RFTEr7WbLyTkM9q9rNMdRp01xxK/sViHcv161QkLB5V2U5230im3aDfimDMKPmyNAcec908WyWlwjcv9V981zxl5RacMuVyDLhe510ZNypz2d065Q9gix73pS47K+YkKA8pFhMT0qrbPEVeJGvaIRU552F1+pM+P9fiVazS41QFvmWOW6omugQlcKKLjvUVVqoVMl3dWo406fdkR0+WdtNqQ8+rNFBcLOn1iigo4psvPglOVGwXT5Zx1zjQvu16Tfr3e9Ijr7fWyMSVpv3FKWpvV2q3zET8y1QFf833X+/9dbI9uP/SMKVqD3fSIsLwO+92AT8r7tbv9Kb7tM7Za5EUDVvmUq2xy3lwhBYyKYK6p4pZ62e+kfcZmNXY67CYP67LQiFZ7sQbfNk9eTEZUWEaN68yXNKDor70t77xmte503M/daJL71fpjP5QRQfr9tVLGZe7zgEdt8Lf4iv9mmmp9NtllsiWyQuaIyxgVU9Kix6Xe9IRn3OVJ0wz7uN9pkHazt31Cl9WGJCU126qgJKZK3F+YpNGvPW+xk6L+k0+6yBRHdKh2h7dc4bwHPG006C0b7xxo1OQqT+kxT4VuX5fVB3pdqdV5P7HcInWuNBRkcCYwgf9nRONB3XlcSkSHNY47jpIoepS51Q3a1BsxqCSmXlEmqMePezc40VJhi33QW2500A/VWOqofzfog+Zb5AYsdcg+YSEDtglpVLJGxI9khHzSdk+7U71JxvwUA6apVhIO+p5qjfqIVZZpU+tqJJT8Tw9Y7wVrVdrnRjGr7bbdtfKyKhQVTTPVLF/2SUfN8rJLrfaaop85ap057lTwnAE78ZhrFZQrVxQxbJVbfdNZU13vIauVpHxRt00W+2cvW+WsWo0KYoZVydptlQ4Jd3vFPzjhKs+BkO9YYrd2O/2NgufVukzCgGJQ6zPsmJiI46YY1uNG89R42jf9wr/5gDdd6163WWdYzkCQn0qpt9Swb9vgjEdM1S6sRbVuyx1zynU6HHXIQw6qMQNElGSs0qnWOifltHpDjweCX/S0xXb6lUvNFMNCh5lgmAlcGKIlJCWElPTolXAM1Bs/EvZV2zAoqlFYtViQ3ygoqlJtwBW+4DlpcxyXVpKwWpU33G+36d4wyXW6NeKogqiIqBa/EZK0wAw9jmr2uCbXOaTeQbtc5A89bZlJRpRUCCmpMck/YJspdovJWiClxjS/tcYUP/UpP9NntjH0yJobdNxUKHlC2EvK1LrcD7ypV40F6FT0nHtU+4xD2mw0JimvKCJlk+u9aNR609T4rh97ywxJl/ipuCOqdJily3kf95pqBaPaZJxT5gfyau130LPgkLB6rT4rpsPDLvMHHtFpskhQ79Nrv4QK/91Cm9zjMnt1+aifG3GzA8LudqnPqrbC3UEPd0mdqbr9yHqb7dMga8wyIe9YKazMOQ9b4oyjXsUsVUHFclSbnI/a4TbPavZbK033Z74LInqc9XmzHHYp8hO91RO4cETHFRDyirqcVWfYefyJaS7Fv/q8u8ERywwqV0ROWExWzmVut8aY/zBDg26VgbLARQY1eNaVfmi1F5xxB6gwZnw9NcvtIt7zplNSFjmsqEG1Ye32u9OHHHCjJu2eUBfkGSdLOOCgOkOaNOjS4CU5EZOCZ/5W3zfNFaK2iVsnHXQu9xm0x0pbrZO0WVKtU94Us9hh3/O0ky7xL/7CFXpklFQhp8dBr/uI6/GSKRZptM5Cv7DPR+SwRsR3XSruWm8YUybtmJydzmi202/ltSpgrVd82wNOYpNlPiVht+POS6mUD75jwtOmuFHOOvv9o/WGXWWbgwZdrltEm5scdlhH0B0WVWbYS+q8KavVFheb6ZitDlkqo06byQ7arAwUpcUI+iRXmu7v3KTBmA971JyAX9hvgzJrPafFsHLtMhO91RO4UERzQhKy8rLaDRpwo0XGfM9is70U7MDSoFxaWosRKSlpYSH3SDioXpXnfEm1pMvN86ZdrvS3/l7cVP/TViPB5WoMiKoXtd7rfqnkpHudVWXE5/yHd2x2ix8Z9bb9rjLTTN9RLSIsZsBB33Kxi90n6k27ZLW63uVWOqPTdMf9Dyf8m0ELJOyzREHUsHIHTXPea+IedL9zciplXelJ1/mYj1rnHd/UaZuYHs1CCMuostgMowbdbkirH4ojrt5r7pf0ukka7LdCp3ig4TJioyV2muGz3nbIj/FhM7yhzCed8rJ5dhiwUYchyUABpiAqKorJlgtrd6s2N3rBdlE3OeotmyR1GPNVL9tvsoREoNYwLO81k9zp+2ZqVmmfJV5xs33qfUeLLQHLF4NqpqLxmsmou20zVb/jlvk+WCPsBmVqdGsRsk23/ZplJxhmAheGqCDLwWEHNbrGIZd4yUc04kp/i8kuklBrsiEJZzUICYuqMyQvZ9AkH5S21oC8095zp7esc6tBK/xYylZXqdHtrLQKeVdpNGiKRWr8dyRdZLtNGn3WBv/ib33LSiWn/aOT5gQxfcxBM+UdsV15oHCwz5VW2qtoTIuvuNocHaqEdJlmRFxIxhSrLHKjdkPWeF3K4yImSfiyFSZ5UZuUJ12q3oh5qoO+nTKD+myStFpMK+aqU6Eb9+oy2Sx5rXKgTtK42lVahfW2ek2tUWlrXS7q28ac9ZBVvueD3jVFxGpdisaCPvBxnYuUZXpc7iUbtIk460WfdtQWfyAsrlOL2/2rqSiK69fjP1TKWeuMFZI2udKgw/7E01JGVEpq9RKoElMyHomWG9Ntuk6v+bAPecIar6HXehd7znKj9lvltCEtyqUmGGYCF4bouEZbwTlt6DdLwUtCjvu5uZ7R5nbXO22uE3pFVLtEf6BC1+UN92r0hvs85F3TbPFnXrXA467y97aab6OppvhHKTyr3Iicgpn2ucQ0j1hsv5IGOywx0z9Z5XnzPW+RG5R7xmsWC4sE6latmkVU2qlZmadscIf5tjuo1QMuskiVs37jMilhk5QryiuT8ceOCPmSNx0wyQKNlqpQ1KHZt2zQLmaOWvXqpAJNlJyUKV52oyvttMB2O/21bt/WYjlqPCkq7pQVRsXMDvRuqtQ7acSv1KuWN9tUT/uQA0o6THdYs3eEvWOjdq0qlALFreMO2WCOLaZqcoeIM4bs8gE/1eZjMhZ6zC1qbVUurCQtKanPKVkPOO0ZU62zyr2afcN23TioyQLdSpgsqVxYWlLegJNBxdHNyp035gXUu0aVl1Xa7Ky5JhtwTlrSyATDTODCEBUoTvbpM9mfOO1qUUdsttw/+YwrfEvKzY4bkzRNi2FDskLi8l53kRPeM90Cc8TU+Zl7/cwn5KVUeN41pllns2qLzbUZBfsRM9+rwkJuE/Oee82Q9I6TPuCQvNNO2qfWxeJBBWpOxDXaLDCq3cMOiSi31l3+q1MOqJGz3RZXytppvpnKZZUJmy2h1gm1DvkzjypXJmqqN62xU6e1TrtDh0GN8hoNaJIJ9GFWqlOv3U6XWuYWvO7L3tMjabPFTptriuMO6ZOUUlIjbNCYYybpsFjYS+ots9U6v3NEuz0+pd3bBtBiVExEXlhOv5g77HVAXJu0l8yzXUGHi91kUN5OM9wu7LyonDJxWSXlqkUctdBCL/kru9ys0zekLfBLc82Td0jeMsvEpcWFFBE3YJHpKvT7nVVW+pHPG3HEHnfIOuhaTWq1SRgJNMomMIELQDSiKK+gHneJudNp691kyG8ttst1vuAinY6Yo1ePciPCEooyGkw2SZUlZugwatAdNvme3dY57C03uR9Jmw1b5ZA9ishLYb6D/sI0BRf5nmqPud9C76q001xbfcWvDGsmUMYd70Hul/V9Ce8YM027U67xqimaLDfXuzIqscyYJjECtblejZ6y1rv2usZ6L2rWpkZEuztNsc8sT0r5G684ZKETQYxWEpdz1gs+7YwOM7zrYhlJc3Ta5UY/cZ0Oj5ntjGZlosKGlYzars95sM90o2IWSxtwXAE/DHrTp4soyAtJyMnIKLfFmO94xiM+7bS51vmuOz1khYRJPmSWbkVpTaJKcorCoua51D4PqXHUUTzsZiMyIi5yRLN3dVuq1WrdzgsrSRiSsMAKhx3Waqq5Ylr8f36p0kk5B1U7oNJhh9UoF5WfYJgJXBiiIXlZ0632sCEHnHLGtb4pJ+8Ofb6nTpN3bHDaN5RZJq8vyBpUS3lHWMxGz/uiw/ps9CHlnnGnuw1a4X9otcx0J+03qCAiokWbtO+6R7kxt1mrzueFvaVVs61qrPOX+l3sSV1WCikGuklnjZjrMfyjdqe1CFnklBaPuNZpN4qKuUnOWQginx4lN3tXzP1OOy9uugHdbtHrdaucdMLtpihImK5bmRyyIqJKzjpjoc/pt1EZPooTmmwwYrZzXrfQYdH3lXUzxgwbDjgkI2aVcxLecNoc93tad1CbyIiSpIhxPe8yU9TrdJshhz3gb1zuWf/Z9bZrtc4WnWb6hYvt12WRpEhwlZSEXlMsN2qVblu0eMeIaQa8oMIOlTY4LaXSKXlpcTlFZXarkVPnUZPUmW7Qu4oSFnlcuRELDOpRZ0ydCmMTDDOBC0O0iJDz5qDSQf1qfd5m55ULmyHieimzRb1hRKOjQYftuIZjpck6LbfYDUoOyllsu/NK9nvVejtcLi1th7RuA6oUFNCnS4UO5xxzn7Pu8qi0mGYdPuhV8633pma18kH/YdZ4tJXX5WJpz2k3S9xJlzrsFSmjprnWL33c606oMa6zGxOWUZDzqAHrnDbTPlM0SIrocbkdbnPae96x3FQDRiUDTaiomBmK9toirtUiz2hU8gNf9rJXpFWJavWCPnMkFUSElOs2bLqlzlnriKgmhyzyc3cbUKtCSrONICcZ6P+POzNMlZLWrlOH/3Cly83Q5QptBr2nxWE/cY92ZRZKBeqEcXn1+jxvprQbvOce60U97ybfMGqlc95yj4wbrNArJKkUcP2I36ky1y5rJY3YIeov/VjSKX/kddOccMRSo84iZ6LibgIXiGhJRJnTeiw3pFOvBSa706c9aaP1cqIeNNspeyT0SqmQMCgs4pwjpjlln+X6pOz1QeU2WSmtIG+26w2qFdbrhGIwJ0IKJlmpUd5inZZYiqO+4t/t8KKP+GNP6zfmSTUmib6vGhDTokvIMjEVYnrEhXS71Ck3O+kaJ6zV71VFFWYHvXr5QIflfpv91pcc9U3P4Ndu0Gmpgo+72AcdccyYAYMicsZ7zqMaRVU54CmvOKjRmLg6/+yEuDXCBpw100wpUWFhIVlJVS4zU6uLzLJMu3P+SpWj5tpqpraAX0JCzpss/77KbkSXzX5mvq95S6cDWmzwc9Mt84Y5Wo0I69EUdHBE5JRQZq8G51Sp9Zq4XvWmS7lEuSpNrtVomXUO65QOItC8sJAy24xpNWypGfJe95om3UZMdchWaZc7IR3Ec7kJhpnAhSH0VQVJvYaViznisBXOOC2pT4V6b/iopUJOCCuoFhZXlFAQktdu0D45dRaY6ZgaV2rzMU/ot06L14VcYqsql9pnj3yQLbzaFfa7CL82bI2od7WKeNVKZzwibKoWYwblxSWMO/+UhAyJqpLRZ4FTbvcvxsx22I2eNd8CIVmjhoxoVFSUE1cwyXkVRsS1uNWruiyySKcRO6wVslnYeaud161NQlRYTjxQhcrq0a1Nt3OS0mi1StKgkmZZcZUgYUw8UMye7aPyDjrmDZOMOuNGr0k6YY7HzDSiSreiK9Sr9HsNjfP2OKDgMg846by75O1TbrJWt/i6pTZ60pg6oypUKgbeJyPiphs13Yif+qRXnHLWbl9WNMmICjcb0aTdj5GSNO7gkjBqzEkjZmuSct4fG9ThhMNidjokZLrZJisncFyawAQuAKGvCMtj0Dlj+iT02o4ZhvSC+eYoN1uKwHVoXEc/Ii8fPPPbbDeoRtFCMy02Radue/WrUCOiRYMyZGRVyOpRbrU1mv+PwezSYZ6Doub6neOqgt7o31eJ5STl9ehw0FFzXeOgTa5Trs6vxDTb4xphi5QrSQRObVEldZK6RKRxmyEzpVV4yn/SKaTNqDPma9Cj3V5R0UDhNyKmKKtkmCCO6pYWFlMZ9ERHxGUk3vd1oiRhnuMGLHaZFm+7Srd3NDgkLqTGKdtViztnCoESxnjup81pzVKOq/PnOu2Rc6taOXW2CTujXV5UKlASLgR+V+U+omDQLFtcZJ9pzljjCbNlTdagBd8xpktcUTjIn0eEjDqjX5c23f/bbzFbr0oXyarSolZcUWSi4m4CF4rQ1xQD/8KwXOCk2KEgJy2hRkpETiJQoI2JBJ5jhBUwrhkyJKog7rwq/ZLyhuXUSaGkTFJWKtDJFvQKpZSEjShqFtOvSVHWJCW90uiVC1w38sHqKhbMiR7HnLE9UAj/3xGxVo1mdeKBL0gkUOEPYVC7Po2muFjGOb2WO+WME2aZKmZQVkhOubHAfbEs6JKIyQoH7gO/zwCHgtGEgshMkHfNI6ZDh4JrzTIf3fb4HOhQbdioR5WrdlKZjLhxN8mSghEFY8oldDunVtyYToPqTBcxrFKZqApZ0WDGj+v/TbXAxYrK/q8csE+lk96TNRR4s3nfkyYa9FIPCIkr02tIlbgxKXWGZaWkAj+D5ATDTODCEHpQOPCTHfc3JaMYdOIVhIXEgvqJ8WxkLtiF+P2MGp+Jv/eiReBfGlUInF7HOSksEnidZQLt60zgSZQN9GLG/V4TwVwOKUnIBj6vwWAVhIPcS0mZjHH9rDbNkgETRo2KqBCWFwucRcZ9JbNC8gqSwUqGAsJy8hJKMkHlTUKeIE9BXFFOVFhEVklYSE5ISUQs8LxMi4pJSxqTwHiN0bh/QASNhoWs1mNASKOsMWdUK3M6iIfyAVdGld53nRM40v7eazMW+ByFA/W937tpkhOXR1iFkrScgnI7LZM1FPjRNAb+TVGCPExYVkksqEceH3U4yLaM981nAhfLrHLF9x3cJjCBC8D/AhuebrR2GjKIAAAAAElFTkSuQmCC",
      "text/plain": [
       "28×280 Array{Gray{Float64},2} with eltype Gray{Float64}:\n",
       " Gray{Float64}(0.493193)  …  Gray{Float64}(0.49208) \n",
       " Gray{Float64}(0.494923)     Gray{Float64}(0.521179)\n",
       " Gray{Float64}(0.501938)     Gray{Float64}(0.490341)\n",
       " Gray{Float64}(0.5027)       Gray{Float64}(0.499529)\n",
       " Gray{Float64}(0.511651)     Gray{Float64}(0.492984)\n",
       " Gray{Float64}(0.487019)  …  Gray{Float64}(0.494325)\n",
       " Gray{Float64}(0.50801)      Gray{Float64}(0.469668)\n",
       " Gray{Float64}(0.497876)     Gray{Float64}(0.496014)\n",
       " Gray{Float64}(0.492468)     Gray{Float64}(0.489192)\n",
       " Gray{Float64}(0.498149)     Gray{Float64}(0.512669)\n",
       " Gray{Float64}(0.513642)  …  Gray{Float64}(0.503171)\n",
       " Gray{Float64}(0.481137)     Gray{Float64}(0.494214)\n",
       " Gray{Float64}(0.496695)     Gray{Float64}(0.499167)\n",
       " ⋮                        ⋱                         \n",
       " Gray{Float64}(0.490783)     Gray{Float64}(0.48122) \n",
       " Gray{Float64}(0.488553)     Gray{Float64}(0.534483)\n",
       " Gray{Float64}(0.495936)     Gray{Float64}(0.488851)\n",
       " Gray{Float64}(0.495618)     Gray{Float64}(0.501894)\n",
       " Gray{Float64}(0.506605)  …  Gray{Float64}(0.485813)\n",
       " Gray{Float64}(0.511504)     Gray{Float64}(0.504919)\n",
       " Gray{Float64}(0.500037)     Gray{Float64}(0.505348)\n",
       " Gray{Float64}(0.499642)     Gray{Float64}(0.487168)\n",
       " Gray{Float64}(0.507527)     Gray{Float64}(0.494356)\n",
       " Gray{Float64}(0.497572)  …  Gray{Float64}(0.483329)\n",
       " Gray{Float64}(0.508945)     Gray{Float64}(0.527273)\n",
       " Gray{Float64}(0.500492)     Gray{Float64}(0.494183)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Epoch 99\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let us visualize the evolution of the weight matrix as images below\n",
    "# Each row is turned into a 28x28 image with positive weights light and negative weights dark gray\n",
    "using Images, ImageMagick\n",
    "for t in 10 .^ range(0,stop=log10(size(lin,2)),length=20) #logspace(0,2,20)\n",
    "    i = ceil(Int,t)\n",
    "    f = lin[1,i]\n",
    "    w1 = reshape(Array(value(f.w))', (28,28,1,10))\n",
    "    w2 = clamp.(w1.+0.5,0,1)\n",
    "    IJulia.clear_output(true)\n",
    "    display(hcat([mnistview(w2,i) for i=1:10]...))\n",
    "    display(\"Epoch $(i-1)\")\n",
    "    sleep(1) # (0.96^i)\n",
    "end"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "julia.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Julia 1.3.1",
   "language": "julia",
   "name": "julia-1.3"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}