15. MNIST 优化算法对比.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实验：MNIST 优化算法对比"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验概要\n",
    "\n",
    "在之前的实验中，我们主要对各种梯度下降优化算法的数学逻辑进行讲解，并绘制其梯度下降轨迹图；在本实验中，我们将进一步把它们放到同一个数据集与神经网络中，在完全相同的『赛道』上，测试各种梯度下降优化算法的性能。\n",
    "\n",
    "<img src=\"./img/compare_15.png\" width=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验目标\n",
    "\n",
    "在本实验中，我们通过 TensorFlow 快速的建立一个卷积神经网络，分别使用：SGD，AdaGrad，RMSProp，AdaDelta，Adam 优化器对 MNIST 数据集进行模型训练，测试不同的优化算法在相同的数据集上的准确率对比。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 导入库\n",
    "\n",
    "由于通过 Python 实现完整的卷积神经网络需要大量的代码，这里为了简便实现，使用工业标准深度学习框架 TensorFlow 2.x 版本实现，前端 API 为 Keras。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 加载数据\n",
    "\n",
    "MNIST 是 TensorFlow 的内置数据集，直接从框架中导入即可。\n",
    "\n",
    "```keras.datasets.mnist.load_data()```\n",
    "\n",
    "但由于需要从互联网上下载，连接可能不稳定，我们直接下载到本地，使用下面的代码进行加载。\n",
    "\n",
    "与之前实验一样，训练集与测试据依然是 60000 张 / 10000 张 `28*28` 像素图片。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)   (60000,)\n",
      "(10000, 28, 28)   (10000,)\n"
     ]
    }
   ],
   "source": [
    "base_path = os.environ.get(\"BATH_PATH\",'./data/')\n",
    "data_path = os.path.join(base_path + \"lab15/\")\n",
    "result_path = \"result/\"\n",
    "os.makedirs(result_path, exist_ok=True)\n",
    "\n",
    "path = data_path+'mnist.npz'\n",
    "\n",
    "f = np.load(path)\n",
    "x_train, y_train = f['x_train'], f['y_train']\n",
    "x_test, y_test = f['x_test'], f['y_test']\n",
    "f.close()\n",
    "\n",
    "print(x_train.shape, ' ', y_train.shape)\n",
    "print(x_test.shape, ' ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 查看数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAADoBJREFUeJzt3X2MXOV1x/HfyXq9jo1JvHHYboiLHeMEiGlMOjIgLKCiuA5CMiiKiRVFDiFxmuCktK4EdavGrWjlVgmRQynS0ri2I95CAsJ/0CR0FUGiwpbFMeYtvJlNY7PsYjZgQ4i9Xp/+sdfRBnaeWc/cmTu75/uRVjtzz71zj6792zszz8x9zN0FIJ53Fd0AgGIQfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQU1r5M6mW5vP0KxG7hII5bd6U4f9kE1k3ZrCb2YrJG2W1CLpP9x9U2r9GZqls+2iWnYJIKHHuye8btVP+82sRdJNkj4h6QxJq83sjGofD0Bj1fKaf6mk5919j7sflnSHpJX5tAWg3moJ/8mSfjXm/t5s2e8xs7Vm1mtmvcM6VMPuAOSp7u/2u3uXu5fcvdSqtnrvDsAE1RL+fZLmjbn/wWwZgEmglvA/ImmRmS0ws+mSPi1pRz5tAai3qof63P2Ima2T9CONDvVtcfcnc+sMQF3VNM7v7vdJui+nXgA0EB/vBYIi/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+IKiaZuk1sz5JByWNSDri7qU8mkJ+bFr6n7jl/XPruv9n/np+2drIzKPJbU9ZOJisz/yKJesv3zC9bG1n6c7ktvtH3kzWz75rfbJ+6l89nKw3g5rCn/kTd9+fw+MAaCCe9gNB1Rp+l/RjM3vUzNbm0RCAxqj1af8yd99nZidJut/MfuHuD45dIfujsFaSZmhmjbsDkJeazvzuvi/7PSjpHklLx1mny91L7l5qVVstuwOQo6rDb2azzGz2sduSlkt6Iq/GANRXLU/7OyTdY2bHHuc2d/9hLl0BqLuqw+/ueyR9LMdepqyW0xcl697Wmqy/dMF7k/W3zik/Jt3+nvR49U8/lh7vLtJ//WZ2sv4v/7YiWe8587aytReH30puu2ng4mT9Az/1ZH0yYKgPCIrwA0ERfiAowg8ERfiBoAg/EFQe3+oLb+TCjyfrN2y9KVn/cGv5r55OZcM+kqz//Y2fS9anvZkebjv3rnVla7P3HUlu27Y/PRQ4s7cnWZ8MOPMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCM8+eg7ZmXkvVHfzsvWf9w60Ce7eRqff85yfqeN9KX/t668Ptla68fTY/Td3z7f5L1epr8X9itjDM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRl7o0b0TzR2v1su6hh+2sWQ1eem6wfWJG+vHbL7hOS9ce+cuNx93TM9fv/KFl/5IL0OP7Ia68n635u+au7930tuakWrH4svQLeoce7dcCH0nOXZzjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQFcf5zWyLpEslDbr74mxZu6Q7Jc2X1Cdplbv/utLOoo7zV9Iy933J+sirQ8n6i7eVH6t/8vwtyW2X/vNXk/WTbiruO/U4fnmP82+V9PaJ0K+T1O3uiyR1Z/cBTCIVw+/uD0p6+6lnpaRt2e1tki7LuS8AdVbta/4Od+/Pbr8sqSOnfgA0SM1v+PnomwZl3zgws7Vm1mtmvcM6VOvuAOSk2vAPmFmnJGW/B8ut6O5d7l5y91Kr2qrcHYC8VRv+HZLWZLfXSLo3n3YANErF8JvZ7ZIekvQRM9trZldJ2iTpYjN7TtKfZvcBTCIVr9vv7qvLlBiwz8nI/ldr2n74wPSqt/3oZ55K1l+5uSX9AEdHqt43isUn/ICgCD8QFOEHgiL8QFCEHwiK8ANBMUX3FHD6tc+WrV15ZnpE9j9P6U7WL/jU1cn67DsfTtbRvDjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPNPAalpsl/98unJbf9vx1vJ+nXXb0/W/2bV5cm6//w9ZWvz/umh5LZq4PTxEXHmB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgKk7RnSem6G4+Q58/N1m/9evfSNYXTJtR9b4/un1dsr7olv5k/cievqr3PVXlPUU3gCmI8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjjOb2ZbJF0qadDdF2fLNkr6oqRXstU2uPt9lXbGOP/k4+ctSdZP3LQ3Wb/9Qz+qet+n/eQLyfpH/qH8dQwkaeS5PVXve7LKe5x/q6QV4yz/lrsvyX4qBh9Ac6kYfnd/UNJQA3oB0EC1vOZfZ2a7zWyLmc3JrSMADVFt+G+WtFDSEkn9kr5ZbkUzW2tmvWbWO6xDVe4OQN6qCr+7D7j7iLsflXSLpKWJdbvcveTupVa1VdsngJxVFX4z6xxz93JJT+TTDoBGqXjpbjO7XdKFkuaa2V5JX5d0oZktkeSS+iR9qY49AqgDvs+PmrR0nJSsv3TFqWVrPdduTm77rgpPTD/z4vJk/fVlrybrUxHf5wdQEeEHgiL8QFCEHwiK8ANBEX4gKIb6UJjv7U1P0T3Tpifrv/HDyfqlX72m/GPf05PcdrJiqA9ARYQfCIrwA0ERfiAowg8ERfiBoAg/EFTF7/MjtqPL0pfufuFT6Sm6Fy/pK1urNI5fyY1DZyXrM+/trenxpzrO/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOP8U5yVFifrz34tPdZ+y3nbkvXzZ6S/U1+LQz6crD88tCD9AEf7c+xm6uHMDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBVRznN7N5krZL6pDkkrrcfbOZtUu6U9J8SX2SVrn7r+vXalzTFpySrL9w5QfK1jZecUdy20+esL+qnvKwYaCUrD+w+Zxkfc629HX/kTaRM/8RSevd/QxJ50i62szOkHSdpG53XySpO7sPYJKoGH5373f3ndntg5KelnSypJWSjn38a5uky+rVJID8HddrfjObL+ksST2SOtz92OcnX9boywIAk8SEw29mJ0j6gaRr3P3A2JqPTvg37qR/ZrbWzHrNrHdYh2pqFkB+JhR+M2vVaPBvdfe7s8UDZtaZ1TslDY63rbt3uXvJ3UutasujZwA5qBh+MzNJ35H0tLvfMKa0Q9Ka7PYaSffm3x6AepnIV3rPk/RZSY+b2a5s2QZJmyR9z8yukvRLSavq0+LkN23+Hybrr/9xZ7J+xT/+MFn/8/fenazX0/r+9HDcQ/9efjivfev/Jredc5ShvHqqGH53/5mkcvN9X5RvOwAahU/4AUERfiAowg8ERfiBoAg/EBThB4Li0t0TNK3zD8rWhrbMSm775QUPJOurZw9U1VMe1u1blqzvvDk9Rffc7z+RrLcfZKy+WXHmB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgwozzH/6z9GWiD//lULK+4dT7ytaWv/vNqnrKy8DIW2Vr5+9Yn9z2tL/7RbLe/lp6nP5osopmxpkfCIrwA0ERfiAowg8ERfiBoAg/EBThB4IKM87fd1n679yzZ95Vt33f9NrCZH3zA8uTdRspd+X0Uadd/2LZ2qKBnuS2I8kqpjLO/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QlLl7egWzeZK2S+qQ5JK63H2zmW2U9EVJr2SrbnD38l96l3SitfvZxqzeQL30eLcO+FD6gyGZiXzI54ik9e6+08xmS3rUzO7Pat9y929U2yiA4lQMv7v3S+rPbh80s6clnVzvxgDU13G95jez+ZLOknTsM6PrzGy3mW0xszlltllrZr1m1jusQzU1CyA/Ew6/mZ0g6QeSrnH3A5JulrRQ0hKNPjP45njbuXuXu5fcvdSqthxaBpCHCYXfzFo1Gvxb3f1uSXL3AXcfcfejkm6RtLR+bQLIW8Xwm5lJ+o6kp939hjHLO8esdrmk9HStAJrKRN7tP0/SZyU9bma7smUbJK02syUaHf7rk/SlunQIoC4m8m7/zySNN26YHNMH0Nz4hB8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoipfuznVnZq9I+uWYRXMl7W9YA8enWXtr1r4keqtWnr2d4u7vn8iKDQ3/O3Zu1uvupcIaSGjW3pq1L4neqlVUbzztB4Ii/EBQRYe/q+D9pzRrb83al0Rv1Sqkt0Jf8wMoTtFnfgAFKST8ZrbCzJ4xs+fN7LoieijHzPrM7HEz22VmvQX3ssXMBs3siTHL2s3sfjN7Lvs97jRpBfW20cz2Zcdul5ldUlBv88zsJ2b2lJk9aWZ/kS0v9Ngl+irkuDX8ab+ZtUh6VtLFkvZKekTSand/qqGNlGFmfZJK7l74mLCZnS/pDUnb3X1xtuxfJQ25+6bsD+ccd7+2SXrbKOmNomduziaU6Rw7s7SkyyR9TgUeu0Rfq1TAcSvizL9U0vPuvsfdD0u6Q9LKAvpoeu7+oKShty1eKWlbdnubRv/zNFyZ3pqCu/e7+87s9kFJx2aWLvTYJfoqRBHhP1nSr8bc36vmmvLbJf3YzB41s7VFNzOOjmzadEl6WVJHkc2Mo+LMzY30tpmlm+bYVTPjdd54w++dlrn7xyV9QtLV2dPbpuSjr9maabhmQjM3N8o4M0v/TpHHrtoZr/NWRPj3SZo35v4Hs2VNwd33Zb8HJd2j5pt9eODYJKnZ78GC+/mdZpq5ebyZpdUEx66ZZrwuIvyPSFpkZgvMbLqkT0vaUUAf72Bms7I3YmRmsyQtV/PNPrxD0prs9hpJ9xbYy+9plpmby80srYKPXdPNeO3uDf+RdIlG3/F/QdLfFtFDmb4+JOmx7OfJonuTdLtGnwYOa/S9kaskvU9St6TnJP23pPYm6u27kh6XtFujQessqLdlGn1Kv1vSruznkqKPXaKvQo4bn/ADguINPyAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQf0/sEWOix6VKakAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.imshow(x_train[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 数据变换\n",
    "\n",
    "`mnist.load_data()` 为 MNIST 提供的数据结构是（nb_samples，28，28）3 个维度，即每个样本分别具有 2 个维度，表示 `28x28` 的灰度图像。但是，Keras 中的 Convolution2D 图层设计为每个数据样本设计了 3 个维度，即：它们具有 4 维输入和输出。因此，MNIST 输入的灰度图像需要通过 `reshape` 转换为匹配 Keras 图层设计的标准。当 `reshape` 使用 `-1` 时，Numpy 会根据剩下的维度，计算出新数组的 shape 属性值。详细信息请 [查阅官方文档](https://numpy.org/doc/stable/reference/generated/numpy.reshape.html)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape((-1,28,28,1))\n",
    "x_test = x_test.reshape((-1,28,28,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输出重塑之后的形状："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28, 1)\n",
      "(10000, 28, 28, 1)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 模型搭建\n",
    "\n",
    "这里使用贯序模型，一层一层叠加模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.Sequential()\n",
    "\n",
    "model.add(layers.Conv2D(input_shape=(x_train.shape[1], x_train.shape[2], x_train.shape[3]),\n",
    "                        filters=32, kernel_size=(3,3), strides=(1,1), padding='valid',\n",
    "                       activation='relu'))\n",
    "\n",
    "model.add(layers.MaxPool2D(pool_size=(2,2)))\n",
    "model.add(layers.Flatten())\n",
    "model.add(layers.Dense(32, activation='relu'))\n",
    "\n",
    "# 分类层\n",
    "model.add(layers.Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. 配置优化器\n",
    "\n",
    "我们将在相同的网络下，分别使用：**SGD**，**AdaGrad**，**RMSProp**，**AdaDelta**，**Adam** 优化器进行训练并记录精确度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1 使用 SGD 优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 32)                173088    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 173,738\n",
      "Trainable params: 173,738\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer=keras.optimizers.SGD(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "              metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2  SGD 优化器训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "844/844 [==============================] - 21s 24ms/step - loss: 0.4350 - accuracy: 0.9204 - val_loss: 0.1901 - val_accuracy: 0.9457\n",
      "Epoch 2/5\n",
      "844/844 [==============================] - 35s 41ms/step - loss: 0.1100 - accuracy: 0.9655 - val_loss: 0.0892 - val_accuracy: 0.9737\n",
      "Epoch 3/5\n",
      "844/844 [==============================] - 38s 45ms/step - loss: 0.0827 - accuracy: 0.9748 - val_loss: 0.1234 - val_accuracy: 0.9680\n",
      "Epoch 4/5\n",
      "844/844 [==============================] - 28s 33ms/step - loss: 0.0668 - accuracy: 0.9791 - val_loss: 0.0836 - val_accuracy: 0.9758\n",
      "Epoch 5/5\n",
      "844/844 [==============================] - 25s 30ms/step - loss: 0.0561 - accuracy: 0.9823 - val_loss: 0.0825 - val_accuracy: 0.9785\n"
     ]
    }
   ],
   "source": [
    "# 记录训练历史\n",
    "history_sgd = model.fit(x_train, y_train, batch_size=64, epochs=5, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.3 使用 AdaGrad 优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 32)                173088    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 173,738\n",
      "Trainable params: 173,738\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer=keras.optimizers.Adagrad(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.4 使用 AdaGrad 优化器训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "844/844 [==============================] - 30s 34ms/step - loss: 0.0292 - accuracy: 0.9912 - val_loss: 0.0740 - val_accuracy: 0.9815\n",
      "Epoch 2/5\n",
      "844/844 [==============================] - 24s 29ms/step - loss: 0.0237 - accuracy: 0.9936 - val_loss: 0.0740 - val_accuracy: 0.9817\n",
      "Epoch 3/5\n",
      "844/844 [==============================] - 25s 30ms/step - loss: 0.0218 - accuracy: 0.9946 - val_loss: 0.0733 - val_accuracy: 0.9817\n",
      "Epoch 4/5\n",
      "844/844 [==============================] - 25s 30ms/step - loss: 0.0204 - accuracy: 0.9952 - val_loss: 0.0740 - val_accuracy: 0.9820\n",
      "Epoch 5/5\n",
      "844/844 [==============================] - 26s 31ms/step - loss: 0.0194 - accuracy: 0.9956 - val_loss: 0.0743 - val_accuracy: 0.9817\n"
     ]
    }
   ],
   "source": [
    "# 记录训练历史\n",
    "history_adagrad=model.fit(x_train,y_train,batch_size=64,epochs=5,validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5 使用 RMSprop 优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 32)                173088    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 173,738\n",
      "Trainable params: 173,738\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer=keras.optimizers.RMSprop(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.6 使用 RMSprop 优化器训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "844/844 [==============================] - 23s 25ms/step - loss: 0.0708 - accuracy: 0.9787 - val_loss: 0.0837 - val_accuracy: 0.9797\n",
      "Epoch 2/5\n",
      "844/844 [==============================] - 26s 31ms/step - loss: 0.0534 - accuracy: 0.9844 - val_loss: 0.0848 - val_accuracy: 0.9790\n",
      "Epoch 3/5\n",
      "844/844 [==============================] - 26s 31ms/step - loss: 0.0435 - accuracy: 0.9874 - val_loss: 0.0825 - val_accuracy: 0.9795\n",
      "Epoch 4/5\n",
      "844/844 [==============================] - 32s 38ms/step - loss: 0.0343 - accuracy: 0.9902 - val_loss: 0.0915 - val_accuracy: 0.9812\n",
      "Epoch 5/5\n",
      "844/844 [==============================] - 25s 29ms/step - loss: 0.0267 - accuracy: 0.9928 - val_loss: 0.0948 - val_accuracy: 0.9822\n"
     ]
    }
   ],
   "source": [
    "# 记录训练历史\n",
    "history_rms=model.fit(x_train,y_train,batch_size=64,epochs=5,validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.7 使用 AdaDelta 优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 32)                173088    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 173,738\n",
      "Trainable params: 173,738\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer=keras.optimizers.Adadelta(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.8 使用 AdaDelta 优化器训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "844/844 [==============================] - 27s 30ms/step - loss: 0.0152 - accuracy: 0.9953 - val_loss: 0.0934 - val_accuracy: 0.9823\n",
      "Epoch 2/5\n",
      "844/844 [==============================] - 29s 34ms/step - loss: 0.0144 - accuracy: 0.9957 - val_loss: 0.0922 - val_accuracy: 0.9827\n",
      "Epoch 3/5\n",
      "844/844 [==============================] - 25s 30ms/step - loss: 0.0138 - accuracy: 0.9958 - val_loss: 0.0912 - val_accuracy: 0.9828\n",
      "Epoch 4/5\n",
      "844/844 [==============================] - 25s 29ms/step - loss: 0.0133 - accuracy: 0.9959 - val_loss: 0.0904 - val_accuracy: 0.9832\n",
      "Epoch 5/5\n",
      "844/844 [==============================] - 24s 29ms/step - loss: 0.0129 - accuracy: 0.9961 - val_loss: 0.0896 - val_accuracy: 0.9833\n"
     ]
    }
   ],
   "source": [
    "# 记录训练历史\n",
    "history_delta=model.fit(x_train,y_train,batch_size=64,epochs=5,validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.9 使用 Adam 优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 32)                173088    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 173,738\n",
      "Trainable params: 173,738\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.10 使用 Adam 优化器训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "844/844 [==============================] - 28s 32ms/step - loss: 0.0360 - accuracy: 0.9891 - val_loss: 0.0981 - val_accuracy: 0.9793\n",
      "Epoch 2/5\n",
      "844/844 [==============================] - 25s 30ms/step - loss: 0.0240 - accuracy: 0.9927 - val_loss: 0.0962 - val_accuracy: 0.9820\n",
      "Epoch 3/5\n",
      "844/844 [==============================] - 27s 32ms/step - loss: 0.0164 - accuracy: 0.9944 - val_loss: 0.1080 - val_accuracy: 0.9793\n",
      "Epoch 4/5\n",
      "844/844 [==============================] - 26s 31ms/step - loss: 0.0185 - accuracy: 0.9942 - val_loss: 0.1123 - val_accuracy: 0.9795\n",
      "Epoch 5/5\n",
      "844/844 [==============================] - 31s 36ms/step - loss: 0.0139 - accuracy: 0.9956 - val_loss: 0.1276 - val_accuracy: 0.9795\n"
     ]
    }
   ],
   "source": [
    "# 记录训练历史\n",
    "history_adam=model.fit(x_train, y_train, batch_size=64, epochs=5, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. 不同优化器训练准确率对比"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAABUCAYAAACC9EBIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAGwJJREFUeJztnXt4VdWZ8H/vueQkgYSEBEgIaNCqXBUsWvA26NQWUNEqBW2lYL/WZ9Qp6MxUoT5q8dGvPtZvvpnq9HOqoh1HqxQvRQe8tah1RKcgKBEQ1KJiEggRQkJu5/J+f+yd5OTkJDlJzoXL+3ue/Zy19lp77XevZL/v2muv/b6iqhiGYRiGJ9MCGIZhGIcHZhAMwzAMwAyCYRiG4WIGwTAMwwDMIBiGYRguZhAMwzAMwAyCYRiG4WIGwTC6QUTOEZG3RaRORL4Skf8WkTPcslIReUhEKkWkQUQ+FZHHRGSsW14uIuqWNYjIHhF5UUQuzOxVGUb3mEEwjDiISD7wInA/MBQoA5YDLSJSBLwN5ALnAnnA6cAbQKzCL1DVwcBpwKvAcyKyKB3XYBh9RexLZcPoiohMBV5T1YI4ZXcBlwBTVDXSzfHlwF8Bv6qGovb/E/BToLS7Yw0jU9gTgmHEZwcQFpHfisgsESmMKvsm8Fw/FfqzwHDglGQIaRjJxAyCYcRBVQ8C5wAKPATUiMhqERkBFAPVbXVFZI6IHBCRehF5pZemK93foamQ2zAGghkEw+gGVd2mqotUdRQwERgJ/AtQC5RG1VvtTi3dBGT10myZ+/tVCkQ2jAFhBsEwEkBVtwOP4RiGPwKXiUh/7p/vAHuBj5InnWEkBzMIhhEHERkrIv8oIqPc/GjgKuAd4J+BQuBxETlRHPKAyT20N0JE/h64A1hmL5SNwxEzCIYRn3rgG8C7InIIxxBUAP+oqvuAaUAz8JZbdzPO8tPrYto54B6/BZgNfFdVV6TnEgyjb9iyU8MwDAOwJwTDMAzDxQyCYRiGAZhBMAzDMFzMIBiGYRgA+DItQF8oLi7W8vLyTIthGIZxRLFx48Z9qjqst3pHlEEoLy9nw4YNmRbDMIxjjEhECUYiBMNKMBQhGI7QGnbyoah0MBwhGIrJx6RbQzH5cIRgyMmHIhFaQ9HHRWh1z/l/5p3GyIKcfskvIp8lUu+IMgiGYRwdqCrhiDqKMRJxlaxGKVpHSbZGKcZQuHM+tjxWAbeGOpe1hiPtyjwUiS7vet7YNkOR1CzPFwG/10OW14PfK/i9Hifv65z3e4VwimSIJiGDICIzgX8FvMDDqnpPTPnxwApgGI6PlqtVdbeInA/836iqY4ErVfV5EXkM+Bugzi1bpKqbB3IxhmEkhqoSiigtoQgtwbDzG4rQEgrTEnTSze37O/a1hNx9wah0r+Xx66TqEyifR9qVqKNYnc3nFVfxdijbQQFfp3xsOsvnwe9Rsj0RAp4IAYkQ8ITxS5iARMjyhMmSMH6JkCURJ00In0TwE8YnIedXw/gI4SOMlxA+dX49GoZwECIh9zcYJx9yfv134YTgSB29GgQR8QL/hhP4YzfwFxFZrapbo6rdB/yHqv5WRC4AfgEsUNV1uJ/zi8hQ4GMg2hvkT1V1VXIuxTCOLELhSCeF2RyMozi7KNqelWwiiritzkAHnH6vEPB5Cfg8zuaPSvu8DAr4GDrI014nxxchX5rJkyZytZmAJ0xAXOXqaVOqYfw4aR+OEnUUa7g979MQXlexejWENxLG46Y9kRDSrmR7UK5t+5ui94fiH5NOLyMeP3j97q8vKu+D1kMpP30iTwhnAh+r6qcAIvIUcCkQbRDGA//gptcBz8dpZy6wVlUb+y+uYaSfYDhCXVOwY2sMds43BWloDiWsiNvqDHQKwOeRuIo44HfSOX4vBTl+N9+94m6r316nU30vAR9kaxPZ4UYC4QYC4UP4Qw14W+uh5SC01EOz+9ty0Nna8o1R5aGmrn2bVcDu02+hecgJgMTvf3eLjzjzLu1potrpoUwkJh1bFntMvLb7UdbpXN2VdcPeVti7rccq2dnZjBo1Cr/f33Nb3ZCIQSgDvojK78bx8RLN+8DlONNK3wHyRKRIVWuj6lyJ4xQsmrtF5HYc75FLVbUl9uQici1wLcBxxx2XgLhGn2naD1XvQ+Wmjq1hL/hzwD/I+c3KBX/blgNZg7op7+6YXDfvlnv9vd8ASSQUq9Td7WBTkANxFHz01tga7rHtQVleBmf7yPZ7uyjWIclQyHHKs7wefN5eVo2rQqg5SmEfjK/A6w/GKPRYBV+PExaiJwQCeRDId36z8yF3KBSWd+QD+R3lWYPAm8XuUDF5RaWUFw5BxNOhqBP6Ja3/Q4c7qkptbS27d+9mzJgx/WojWS+V/wl4wI0V+ybwJdB+F4lIKTAJeDnqmGU4QUaygN8AtwB3xjasqr9xy5k6dao5XhoozQe7Kv/9f+0oLyyHkadDwWgItUBrIwSjtuY6qK+G4CEINrnlh/r+WC3eKKMSx2DEMSgRfw5NGqBRs2jQLBrCWRyM+KkL+TkQ9LM/6GVfi499LR72NXs40BzhYNsIviXUozg5fi8FuX6G5PjJz/EzemguE3OcfEGOnyFRZe373Ly/N8XcH8IhVxnvdxRyfTcj8C4KvK5zPtL92LodX46rsKMUevHwzgq8U3l+1/pZg8HT935o3raN8pJRiCn2ASMiFBUVUVNT0+82EjEIXwKjo/Kj3H3tqGolzhMCIjIYuEJVD0RVmYcTcjAYdUyVm2wRkUdxjIqRTFoaoPqDKOW/GWp3dpQPOQ5GnganL4CRU6B0sjOq6yuqEG51DEZro2Mo2g2G++salEjLIVqaGmhtaqC1+RCh5gbCLc5+DjUiwQN4QlX4wk34Is1kRZrJ0hYCBPEAg9yt1wXVQIsECHqyCQVyCOfmoD7HyEhWLp6sQXgDufhzBpOVPRhvYFDMk0+8p6Fc8HsgKwD+gPOUE0sk4lx7d1MovSnwtvJgAjOrHl+Mws6H/FFxFHgeZA/pOoJvS8e7jjRixiB5DLQvEzEIfwFOEpExOIbgSuB7MUIUA1+5Pt6X4aw4iuYqd3/0MaWqWiXOFVyG41rY6C+tjVC9xVH8VZud35qPaH/Uzy9zFP6p8x3lP3IyDCru16kiEaW+JRR3Lr2uKciBpiAHm4S6Jj91TYOoa8qirimXusYg9S2hHleXZPk8nUbg7Vu2h6KsMEWBMIX+IAW+EEN8reR5gwz2tJIrrWRFmjsZoUDwEIH2p5i2rQlaG6CxBuqin3Ia6X1aJAaPv+PJRjyOAW45mFg7gRiFnTsUCo/vRYHnQWBIR96XbVMmRlLp1SCoasgN7PEyzrLTFar6oYjcCWxQ1dXADOAXIqI4U0Y3tB0vIuU4TxhvxDT9hIgMw3mjshn4uwFfzbFCsBn2VHSM+is3Qc22jmmbwSMcpT/hOx0j/7wRCTV9qCXE6x/VUFFZ1+1L1IPNwZ6VutfjTq34GJLjZ9jgAF8bNpiC3Kz2KZe2rW2apm3L9nuT0EH9oG2+vdNTTYzBaNvan4Ki9kXCvSvwtvJ+Tq8YyefAgQM8+eSTXH/99X06bvbs2Tz55JMUFBR0W+f222/nvPPO45vf/OZAxUwbR1Q8hKlTp+ox96VyqAX2fNgx6q/c5Kw0iLhz4rnF7og/assv7bnNGOqbg/xp+17WbKni9Y9qaAlF8HmkXUnnx1HcXbao8hy/16YBjITYtm0b48aNy9j5d+3axcUXX0xFRecJilAohM93ZH63G69PRWSjqk7t7dgj84qPVsJB2Lu1Y9RfuckxBm0vBnMKHYV/9reilH9Zv6YNDjS28urWPbxUUc2fd+6jNRxheF6AK88YzaxJpZxRPhSvx5S6cXSzdOlSPvnkEyZPnozf7yc7O5vCwkK2b9/Ojh07uOyyy/jiiy9obm5myZIlXHvttUCHG52GhgZmzZrFOeecw9tvv01ZWRl/+MMfyMnJYdGiRVx88cXMnTuX8vJyFi5cyAsvvEAwGOT3v/89Y8eOpaamhu9973tUVlYyffp0Xn31VTZu3Ehxcf+mcweKGYRMEQ5BzfbOI//qCgi7K28DQ5x5/uk3dCj/guMGNGdc29DCK1v3sGZLFes/qSUUUcoKclgw/XhmTyphyuhCPGYEjAyx/IUP2Vp5MKltjh+Zzx2XTOi2/J577qGiooLNmzfz+uuvc9FFF1FRUdG+bHPFihUMHTqUpqYmzjjjDK644gqKioo6tbFz505+97vf8dBDDzFv3jyeeeYZrr766i7nKi4u5r333uPXv/419913Hw8//DDLly/nggsuYNmyZbz00ks88sgjSb3+vmIGIR1EwrBvZ+elntVbOj7UycpzlP83ru1Q/oVjkvLCcO/BZl7+sJo1W6p596+1RBSOL8rlf507htkTSzl11BCb3jEMlzPPPLPTGv5f/epXPPfccwB88cUX7Ny5s4tBGDNmDJMnTwbg61//Ort27Yrb9uWXX95e59lnnwXgrbfeam9/5syZFBYWJvV6+ooZhGQTicBXn3RW/lUfOC8owVljX3oaTP1hx2qfoScm9SVj5YEmXqqoZm1FFRs+248qnDhsEDec/zVmTixhfGm+GQHjsKOnkXy6GDRoUHv69ddf57XXXmP9+vXk5uYyY8YMmpubuxwTCATa016vl6amrl9kR9fzer2EQj1/F5MpzCAMBFX46tPO6/yr3ofWeqfclwOlp3Ze5198EniSv5Lm89pG1lZUsbaims1fOJ+AjC3J48a/PZnZk0o4aURe0s9pGEc6eXl51NfXxy2rq6ujsLCQ3Nxctm/fzjvvvJP085999tmsXLmSW265hVdeeYX9+/cn/Rx9IaXeTt2yMLDFrfq5qs5x948BngKKgI04zvBaB3xFqUIVDnwWM/J/3/lyF8AbgJJJcNqVzqh/5BQoPsVxUJUiPq1pYK37JFDxpTP3OqlsCD/99inMmljCCcMGp+zchnE0UFRUxNlnn83EiRPJyclhxIiO5dkzZ87kwQcfZNy4cZxyyilMmzYt6ee/4447uOqqq3j88ceZPn06JSUl5OVlbvDW67JT19vpDqK8nQJXRXs7FZHfAy9GeTu9RlUXuGUNqtpFM4nISuBZVX1KRB4E3lfV/9eTLGlbdqoKdbujFL/74rfJtd4eP5RM7Bj1j5wCw8el/ItPVWXn3gbWbKli7ZZqPtrjjGymHFfArIklzJpYyuihqXWPaxjJJNPLTjNNS0sLXq8Xn8/H+vXrue6669i8eWBRAFK97DRZ3k6jhRPgAjq+eP4t8HOgR4OQElShvqrztE/lJmjc55R7fDB8PIyb0zHyHz4efIGe202aeMqHlQd5qaKaNRVVfFpzCBE44/ih3HHJeL49oaTfUZQMw8gsn3/+OfPmzSMSiZCVlcVDDz2UUXnS4e00W0Q2ACHgHlV9Hmea6ICqhqLaLIt38qR7O63f03nUX7kJGva4J/M6I/1TZroj/9NhxATwZw/8vH1AVXl/dx1rtzjvBD7/qhGPwLQTirjm7DF8e8IIhuelVybDMJLPSSedxKZNmzItRjvp8HZ6vKp+KSInAH8SkS10REnrlaR4O/3LI/DxHx3lX1/p7hQYNhZOvKBjqeeIiY5fmgwQiSjvfb6fNVuqefnDar480ITPI5z1tWKun3EiF44fQdHg9DyVGIZxbJJyb6eq+qX7+6mIvA5MAZ4BCkTE5z4ldGkzqXz2NuzbAeXndCj/kkkQyOxL13BE+Z+/fsXaiipeqqhmb30LWV4P551czE0XnsyF40YwJDeznigNwzh2SKm3UxEpBBpVtcWtczZwr6qqiKzDiaL2FLAQ+EOSrqkrlz902DgTC4YjrP+klrUV1bzyYTW1h1rJ9nuYcfJwZk0q4YKxw8nLNiNgGEb6SbW303HAv4tIBPDgvENoexl9C/CUiNwFbAJS9812ho1BSyjMf3+8j7Vbqnll6x7qmoIMyvJy/tjhzJ5UyoxThpGbZZ+EGIaRWRLSQqq6BlgTs+/2qPQqYFWc497GiZQWr81PcVYwHZU0B8O8saOGtVuq+OO2vdS3hMjL9nHhuBHMnFjCeScPy5yrZ8Mw+sXgwYNpaGigsrKSxYsXs2pVF7XXK5s3b6ayspLZs2cDsHr1arZu3crSpUuTLW6fsWFpEmmLJbCmoop12/fS2BqmINfPrEnONwJnfa2IgM+MgGEc6YwcObJfxgAcg7Bhw4Z2gzBnzhzmzJmTTPH6jRmEARIvlkDx4Cwum1LG7ImlfOOEoamJuWsYxoBZunQpo0eP5oYbnFnun//85/h8PtatW8f+/fsJBoPcddddXHrppZ2Oi46jMG3aNB555BEmTHB8Mc2YMYP77ruPSCTCkiVLaG5uJicnh0cffZQxY8Zw++2309TUxFtvvcWyZctoampiw4YNPPDAA+zatYsf/vCH7Nu3j2HDhvHoo49y3HHHsWjRIvLz89mwYQPV1dXce++9zJ07N+n9YQahH8SLJTAi32IJGMaAWLvU8QKcTEomwax7ui2eP38+N954Y7tBWLlyJS+//DKLFy8mPz+fffv2MW3aNObMmdOtQ8j58+ezcuVKli9fTlVVFVVVVUydOpWDBw/y5z//GZ/Px2uvvcbPfvYznnnmGe688852AwDw2GOPtbf1k5/8hIULF7Jw4UJWrFjB4sWLef555zvfqqoq3nrrLbZv386cOXPMIGSS7mIJ/GD68cyyWAKGcUQyZcoU9u7dS2VlJTU1NRQWFlJSUsJNN93Em2++icfj4csvv2TPnj2UlJTEbWPevHl861vfYvny5axcubJdUdfV1bFw4UJ27tyJiBAMBnuVZ/369e2usRcsWMDNN9/cXnbZZZfh8XgYP348e/bsScLVdyWlzu1EZDKOO4p8nA/V7lbVp91jHgP+ho6P1Bap6sCceCSZ7mIJ/OjcE5g1scRiCRhGMulhJJ9Kvvvd77Jq1Sqqq6uZP38+TzzxBDU1NWzcuBG/3095eXlct9dtlJWVUVRUxAcffMDTTz/Ngw8+CMBtt93G+eefz3PPPceuXbuYMWPGgOSMdrOdqtDHvRoE17ndvxHl3E5EVkc7twPuA/4jyrndL4AFQCPwA1XdKSIjgY0i8nLbR2vAT90VSocNlQeaWFtRzUtxYgnMmljKuNI8MwKGcRQxf/58fvzjH7Nv3z7eeOMNVq5cyfDhw/H7/axbt47PPvssoTbuvfde6urqOPXUUwHnCaGszPHIEz0t1JPL7bPOOounnnqKBQsW8MQTT3DuuecO/AL7QEqd26nqjrYKqlopIntxniIOcBjRFktgTUU171ssAcM4ppgwYQL19fWUlZVRWlrK97//fS655BImTZrE1KlTGTt2bK9tzJ07lyVLlnDbbbe177v55ptZuHAhd911FxdddFH7/vPPP5977rmHyZMns2zZsk7t3H///VxzzTX88pe/bH+pnE4ScX89F5ipqj9y8wuAb6jq30fVeRJ4V1X/VUQux3FNUew6t2urcyaOV9MJqhpxp4ymAy3AH4GlqtrSkyzJdH/9SU2D40F0SxUfVnbEEmhbIjqmeFAvLRiGMVCOdffXqSDV7q8ToSfndohIKfA4sNB1bwGOi4tqIAvHed0twJ2xDSfL26mqsmNPgxNVLCaWwK2zxzFzYonFEjAM45gm5c7tRCQf+C/gVlV9J+qYKjfZIiKP4hiVLiTD2+m/v/EJT2/4oiOWQLkTS2DmxBJKh1gsAcMwDEi9c7ss4DmcF86rYo4pVdUqN1jOZUDFQC+mO3bVHqJ0SLbFEjCMwxBVtYUaSWKgq49S7dxuHnAeUOROJ0HH8tInRGQYIMBm4O8GdCU98L+/M8n+4QzjMCQ7O5va2lqKiorsHh0gqkptbS3Z2f0f8Pb6UvlwQkRqgN7XgMWnGNiXRHGShcnVN0yuvnFYyzVs2DDf3XffXV5eXp5zOBiESCTi8Xg8kd5rppdE5FJVdu3a1XTrrbfuqqmpCcUUH6+qw3o7zxFlEAaCiGxI5C17ujG5+obJ1TdMrr5xrMtlXtcMwzAMwAyCYRiG4XIsGYTfZFqAbjC5+obJ1TdMrr5xTMt1zLxDMAzDMHrmWHpCMAzDMHrADIJhGIYBHIUGQURmishHIvKxiHSJWi0iARF52i1/V0TKDxO5FolIjYhsdrcfpUGmFSKyV0TifiUuDr9yZf5ARE5PtUwJyjVDROqi+ur2NMk1WkTWichWEflQRJbEqZP2PktQrrT3mYhki8j/iMj7rlzL49RJ+/2YoFxpvx+jzu0VkU0i8mKcstT2l6oeNRvOl9SfACfgOM17HxgfU+d64EE3fSXw9GEi1yLggTT313nA6UBFN+WzgbU4X5NPw/FoezjINQN4MQP/X6XA6W46D9gR5++Y9j5LUK6095nbB4PdtB94F5gWUycT92MicqX9fow69z8AT8b7e6W6v462J4T22A2q2gq0xW6I5lIcN9wAq4C/ldR/IpmIXGlHVd/EiXDXHZfi+KFSdRwTFrieazMtV0ZQ1SpVfc9N1wPbgLKYamnvswTlSjtuHzS4Wb+7xa5iSfv9mKBcGUFERgEXAQ93UyWl/XW0GYQy4Iuo/G663hjtdVQ1hBPCs+gwkAvgCneaYZWIjI5Tnm4SlTsTTHcf+deKyIR0n9x9VJ+CM7qMJqN91oNckIE+c6c/NgN7gVdVtdv+SuP9mIhckJn78V+Am4Hu3FSktL+ONoNwJPMCUK6qpwKv0jEKMLryHo5vltOA+3Ej9KULcVy8PwPcqKoH03nunuhFroz0maqGVXUyjtv8M0VkYjrO2xsJyJX2+1FELgb2qurGVJ+rO442g9Br7IboOiLiA4YAtaSWRGJK1GpHxLiHga+nWKZESKQ/046qHmx75FfVNYBfHBfsKUdE/DhK9wlVfTZOlYz0WW9yZbLP3HMewAmvOzOmKBP3Y69yZeh+PBuYIyK7cKaVLxCR/4ypk9L+OtoMQnvsBnFiMVwJrI6psxpY6KbnAn9S9w1NJuWKmWeegzMPnGlWAz9wV85MA+q0I7BRxhCRkrZ5U3FCs3pIgxJxz/kIsE1V/7mbamnvs0TkykSficgwESlw0znAhcD2mGppvx8TkSsT96OqLlPVUapajqMj/qSqV8dUS2l/JSuE5mGBJha74RHgcRH5GOfF5ZWHiVyLRWQOEHLlWpRquUTkdzirT4pFZDdwB84LNlT1QWANzqqZj4FG4JpUy5SgXHOB60QkBDQBV6bBqIMzglsAbHHnnwF+BhwXJVsm+iwRuTLRZ6XAb0XEi2OAVqrqi5m+HxOUK+33Y3eks7/MdYVhGIYBHH1TRoZhGEY/MYNgGIZhAGYQDMMwDBczCIZhGAZgBsEwDMNwMYNgGIZhAGYQDMMwDJf/D5SPv11djxhWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAABUCAYAAACC9EBIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAF+hJREFUeJztnXl0VdW5wH/fHZIQiCYELEjE0NZVQUjBomUQy2BXcSj1KUOfE7yhrtfW0uFVC3YVgeqr7fK1dXy2Kg4tyOiAXQ5oC69SURsULAiK9qEgWMOQgDIk997v/XH2vTm5uTf3BpJ7Q/L91jrr7rPH7+zkfN/e++zzHVFVDMMwDCOQbwEMwzCMjoEZBMMwDAMwg2AYhmE4zCAYhmEYgBkEwzAMw2EGwTAMwwDMIBhGE0RkhoiszbccfkRkroj8Pt9yGJ0fMwhGl0FE1ojIfhEpbMM6zxCRxSJSIyIHRGSbiNwpIhVt1YZh5AozCEaXQEQqgTGAApPaqM7PAq8Au4BhqnoSMBp4FzgvTZlQW7RtGO2BGQSjq3AN8DLwEDA9Hiki5SKy0o3uXwU+4y8kIreLyA6Xvl5ExviS5wJ/UdUfqOpOAFX9SFV/raqLXfmxIrJTRH4kIh8CD4pImYj8wc0q9rtwha/NASLyvyJyUESeB3q1T5cYRlPMIBhdhWuAhe74ioh8ysXfDRwB+gL/6g4/fwWGAj2BRcAyESlyaRcAK7Jou48rfzpwLd5996A77w8cBu7y5V8ErMczBD/FZ8AMoz0R82VkdHZE5DxgNdBXVfeIyFbgN8AdeMZgiKpudXn/CzhfVdMt+ewHxqrqRhGJAJeo6rMu7TrgZiAEPKqq3xCRscAq4CRVPZKmzqHAalUtE5H+wN+Bk1X1E5e+CIip6lVt0iGGkQabIRhdgenAKlXd484XubjeeMp7hy/ve/6CIvJDEdkiInUiUgucTOMSzl68mQUAqnqXqpYCvwbCvmpq/MZARIpF5Dci8p6IHAD+DJSKSBA4FdgfNwapZDKM9sIecBmdGhHpBkwFgm4NH6AQKAU+BUSA04CtLq2/r+wY4AZgArBZVWNuhiAuyx+By/CWf1oieRr+n8DngC+q6oduhvC6q3c3UCYi3X1GoX+KOgyjzbEZgtHZuRSIAoPwngUMBQYCL+I9V3gMmOtG7YNoul5fgmcwaoCQiMwBTvKlzwXGiMgvRaQfgIj0cvW3RAnec4NaEekJ3BRPUNX3gGpgnogUuOWurx7LhRtGazGDYHR2pgMPqur7qvph/MB7iHslcB3QA/gQbweSf7T/HPAs8Dbess0RfMtLqvo28EWgAtgoIgeBv+BtQ/1JCzL9GugG7MHb+fRsUvoVrt59eMbikVZftWEcA/ZQ2TAMwwBshmAYhmE4zCAYhmEYgBkEwzAMw2EGwTAMwwCyfA9BRCYCtwNB4H5VvTUp/XRgAd6LPvuAq+K+XUTk58DFLutPVXWJi38I+BJQ59JmqOqGluTo1auXVlZWZiOyYRiG4Vi/fv0eVe2dKV9Gg+Denrwb+DKwE/iriKxU1Td92W4DHlHVh0VkPPAz4GoRuRg4G2/vdyGwRkSeUdUDrtz1qro824uqrKykuro62+yGYXRRYjElElNi6v1G3RGJxYjFaPIbjSlRVSLR5vmjTcr6zlWJxmJNysRiyW0lxamLT5SJEY3h1ePLG3Oy+MtEY8ptUz7PqaXdjqk/RCSrt92zmSGcC7yjqn93FS8Gvgb4DcIg4AcuvBp4whf/Z1WNABEReQOYCCzNRjjDMNoO1UZFFYkpkWiMhqinmCJRpSHqKaaGqHceibn0qNLg8kSiMRpc2Sbx8Thf+fRlGpVpstJLr4xjxDSFIk9VRpWOtps+IBAKBAgEvN9gQBoP8X5DwcZw/AgFhID7jeXgorIxCP1o6utlJ95LM3424r3CfzvwT0CJiJS7+JtE5L+BYmAcTQ3JLe7tzz8Cs1T16DFdhWHkiEg0xtFI/IhytMEL10diKRVgKoXrj48r3MZwo1KNxjQ7hd2s/vR5c0VAIBQMEA6I9xsUQoEAoaAQDgYSyi5Z6QUDQjgcIBgIEBQIBgKJ+HSKMiCurFOoTesLEAzQWF/Q1deC4k1W1qGgqydFmazkCggikrnTOgBt5cvoh8BdIjIDz1HXB0BUVVeJyDnAS3iv/6/DcyMAMBvv7dAC4LfAj4D5yRWLyLV4LoPp379/crLRhVD1FOTRSLRRKTc0huuTlHQiny9P6vRY0zojUVdX87RorH2UqgiE3cgxrjRDAfcblKRwo4ItCqfK69KbhL08jUq5eVrz8o0KPRQUwkll/co9nhbPFwicGArQaEo2BuEDPOdfcSpcXAJV3YU3Q0BEegCXq2qtS7sFuMWlLcJzA4Cq7nbFj4rIg3hGpRmq+ls8g8Hw4cM72ESwa6Gq1EdTKMpkhZqlAq6PZFbMyWWPd9YcCgiFoQAFoQCFoSCF4QCF8XAoQFE4wMndwi4udR7vvGk4nEFpplO48fhgF1WgDQ0N7Ny5kyNHUnoGN1pJUVERFRUVhMPhzJlTkI1B+CtwhogMwDMEX8fztZLAOfTap6oxvJH/AhcfBEpVda+IVAFVeL7hEZG+qrpbvLnUpcCmY7oCI2uONESpPdTA/kP17P+knv2HGth3qJ5aF95/qN4dDRw80tBMGddHYsctQzgojco0FKAwHGyifLsXhujZvVHhFqTJ16KSTpNeEAwQCtpO647Ezp07KSkpobKy8oRZVumoqCp79+5l586dDBgw4JjqyGgQVDXiPvzxHN620wWqullE5gPVqroSGAv8TEQUb8no2654GHjR/aEP4G1Hjbi0hSLSG8/l7wbgP47pCrogqsqh+qhT7D5F3kSxN1B7qJ59n9QnjMCh+mjaOrsXBCktLqBn9wJKi8NUlHZLORJOp8gLwwEKgy0r5oJQ1x0JG6k5cuSIGYM2QkQoLy+npqbmmOvI6hmCqj4NPJ0UN8cXXg402z7qPgoyKE2d41slaSdFVTlwJOKUuae8931S32S0nqzY93/SQH00/Wj9pKKQU+wFnFJSyOf6lFDmU/ZlxQXe0d0LlxaHKQwFc3jVhtGIGYO243j70j6Q04ZEY0ptGiXuLc24JRqXZ/8n9dQebkj7oDIgJBR2z+4FnNazmKqKkynr7hR6XLl3bwyf3C1syyKGYRwTZhDSUB+JJRS3p9h9yzHJa+7u/MCRhrQPPQuCgYRiLy0Oc8YpPZoo8viIvbS4gJ7uvKQoZLs1DKMdqa2tZdGiRXzrW99qVbmLLrqIRYsWUVpamjbPnDlzOP/887nggguOV8yc0SUMwpGGKPs+SVp2SV5/TxrVf3w0kra+buFgk+WXirJiyorjyjxMmVuu6elG92XdC+heELSpsWF0MGpra7nnnnuaGYRIJEIolF49Pv3002nT4syf32wXfYenSxiEbzxSzYvb9qRMKykKJZZfenYv4DO9e3gj+eICSt0I3lPsjWvwRWFbbzeMzsCsWbN49913GTp0KOFwmKKiIsrKyti6dStvv/02l156KTt27ODIkSN897vf5dprrwUa3eh8/PHHXHjhhZx33nm89NJL9OvXjyeffJJu3boxY8YMLrnkEiZPnkxlZSXTp0/nqaeeoqGhgWXLlnHmmWdSU1PDFVdcwa5duxg5ciTPP/8869evp1evXnnpj3w6txsALAbKgfXA1apaf9xXlIJrRlZy0ZC+TRR/qRu9h2293TA6BPOe2sybuw5kztgKBp16Ejd99ay06bfeeiubNm1iw4YNrFmzhosvvphNmzYltm0uWLCAnj17cvjwYc455xwuv/xyysvLm9Sxbds2Hn30Ue677z6mTp3KihUruOqqq5q11atXL1577TXuuecebrvtNu6//37mzZvH+PHjmT17Ns8++ywPPPBAm15/a8mnc7ufA79S1cUici/wb8D/tOXFxfnyoE+1R7WGYXQyzj333CZ7+O+44w4ef/xxAHbs2MG2bduaGYQBAwYwdOhQAL7whS+wffv2lHVfdtlliTyPPfYYAGvXrk3UP3HiRMrKytr0elpLXpzbicgyYDyNL7g9DMylnQyCYRgdn5ZG8rmie/fuifCaNWt44YUXWLduHcXFxYwdOzblG9WFhYWJcDAY5PDhwynrjucLBoNEIumfUeaTbNZLUjm365eUJ+7cDpo7t5soIsXubeZxeG4wyoFa30tqqeoEPF9GIlItItXH88KFYRhGMiUlJRw8eDBlWl1dHWVlZRQXF7N161ZefvnlNm9/9OjRLF3qOX9etWoV+/fvb/M2WkNbLaD/EPiSiLyO99GbhHM7vBfaXgIepalzu6xQ1d+q6nBVHd67d8bvOxiGYWRNeXk5o0ePZvDgwVx//fVN0iZOnEgkEmHgwIHMmjWLESNGtHn7N910E6tWrWLw4MEsW7aMPn36UFJS0ubtZItoBm9hIjISmKuqX3HnswFU9Wdp8vcAtqpqRYq0RcDvgWfwvJ/2ca4xmrSRjuHDh6t9IMcwOg9btmxh4MCB+RYjbxw9epRgMEgoFGLdunV885vfZMOGFj8cmZFUfSoi61V1eKayeXFup6oqIquByXg7jaYDT2Yhi2EYRqfh/fffZ+rUqcRiMQoKCrjvvvvyKk8+ndv9CFgsIjcDrwP53W9lGIaRY8444wxef/31fIuRIJ/O7f6Ot4PJMAzD6ADYW1mGYRgGYAbBMAzDcJhBMAzDMIAsDYKITBSRt0TkHRGZlSL9dBH5o4i8ISJrRKTCl/YLEdksIltE5A73yUxcvrdEZIM7Tmm7yzIMw2h7evToAcCuXbuYPHnyMdWxYcOGJt5SV65cya233tpCidyR0SD4fBldiPeA+J9FJPlBcdyXURUwH8+XESIyChiNt910MHAO3otrca5U1aHu+Oh4L8YwDCMXnHrqqSxf3mwfTVYkG4RJkyYxa1azcXZeyGaGkPBl5LyRxn0Z+RkE/MmFV/vSFSgCCvCc24WBfxyv0IZhGG3BrFmzuPvuuxPnc+fO5eabb2bChAmcffbZDBkyhCefbP6K1Pbt2xk8eDAAI0aMYPPmzYm0sWPHUl1dzauvvsrIkSMZNmwYo0aN4q233qK+vp45c+awZMkShg4dypIlS3jooYe47rrrEvWOHz+eqqoqJkyYwPvvvw/AjBkzmDlzJqNGjeLTn/70MRujTGSz7TSVL6MvJuWJ+zK6HZ8vI1Vd515A2w0IcJeqbvGVe1BEosAK4GZN8dq0iFwLXAvQv3//7K7KMIwTj2dmwYd/a9s6+wyBC9Mvx0ybNo3vfe97fPvb3qtTS5cu5bnnnmPmzJmcdNJJ7NmzhxEjRjBp0qS0H7iaNm0aS5cuZd68eezevZvdu3czfPhwDhw4wIsvvkgoFOKFF17gxhtvZMWKFcyfP5/q6mruuusuAB566KFEXd/5zneYPn0606dPZ8GCBcycOZMnnvB8he7evZu1a9eydetWJk2adMxLVi3Rrr6MROSzwECgAs+wjBeRMa7Mlao6BBjjjqtTVWy+jAzDaC+GDRvGRx99xK5du9i4cSNlZWX06dOHG2+8kaqqKi644AI++OAD/vGP9AsbU6dOTYzYly5dmlDUdXV1TJkyhcGDB/P973+/ySwiHevWreOKKzxHEFdffTVr165NpF166aUEAgEGDRrUojzHQzYzhA/wPJTGqXBxCVR1F87bqfNldLmq1orIN4CXVfVjl/YMMBJ4UVU/cGUPOh9H5wKPHOf1GIZxotLCSL49mTJlCsuXL+fDDz9k2rRpLFy4kJqaGtavX084HKaysjKl2+s4/fr1o7y8nDfeeIMlS5Zw7733AvCTn/yEcePG8fjjj7N9+3bGjh17XHL63Wxn8kF3rGQzQ0j4MhKRAjxfRiv9GUSkl4jE60r4MgLex5s5hEQkjDd72OLOe7myYeASYNPxX45hGEbrmDZtGosXL2b58uVMmTKFuro6TjnlFMLhMKtXr+a9997Lqo5f/OIX1NXVUVVVBXgzhH79PK/+/mWhllxujxo1isWLFwOwcOFCxowZkzJfe5HRIDjfQ3FfRluApXFfRiIyyWUbC7wlIm8DnwJucfHLgXeBv+E9Z9ioqk/hPWB+zn0wZwPejCO/Xp0Mw+iSnHXWWRw8eJB+/frRt29frrzySqqrqxkyZAiPPPIIZ555ZsY6Jk+ezOLFi5k6dWoi7oYbbmD27NkMGzasyQdxxo0bx5tvvpl4qOznzjvv5MEHH6Sqqorf/e533H777W13oVmQ0f11R8LcXxtG56Kru79uD9rb/bXRWVD1Dtyvxlw45ouPh41WobHGIxZ14agLR70+TYRjjeFYLCmvP6xJdfjr9tfhL5euvljztv31NWu7La8lVdsKEoARt0HiDSTx9iIivnPfzh6RpDRfueSykpzvWM6T20iWIenc32am84wypLhOCTbtj3agaxiEtb+C3W+kUICkUYYtKMl4OFFXS+FUyrclRUyWcvjyZyWHCxsGeIpYghAI+sLuVwIuPh5OER9wiqlZHfHzcFIdSb9xxaYxCIQh5B6WJgYi2ngv+M81ljrdXy7TeXK9JxK9B0K4qF2b6BoGYd//efubRbx/yLgFToRJE58mHAiAhFx8oHGUkDYszeMztpeNTBnayBhuoQ3/KMjITFxBtqhEA80VY7P4FpRys3J+pZysdNPUF+hg7su2bEHLBqTd49/uZDJCKc/VZ0v8RsdnYPznTWbc2jqD5T8PZFbXx/sIoGsYhEl35FsCwzBSUFRUxN69eykvL8+PUUi1NHOCoqrs3buXoqJjn0WcUA+VRaQGyLwHLDW9gD1tKE5bYXK1DpOrdXRouXr37h265ZZbKisrK7vlbZbgIxaLBQKBQCxzztySjVyqyvbt2w//+Mc/3l5TUxNJSj5dVTO+2XtCGYTjQUSqs3nKnmtMrtZhcrUOk6t1dHW5OtiComEYhpEvzCAYhmEYQNcyCL/NtwBpMLlah8nVOkyu1tGl5eoyzxAMwzCMlulKMwTDMAyjBcwgGIZhGEAnNAgiMlFE3hKRd0Sk2YdKRaRQRJa49FdEpLKDyDVDRGpEZIM7/j0HMi0QkY9EJKXrcfG4w8n8hoic3d4yZSnXWBGp8/XVnBzJdZqIrBaRN0Vks4h8N0WenPdZlnLlvM9EpEhEXhWRjU6ueSny5Px+zFKunN+PvraDIvK6iPwhRVr79peqdpoDCOK52/403necNwKDkvJ8C7jXhb8OLOkgcs3A+8RoLvvrfOBsYFOa9IuAZ/De4RwBvNJB5BoL/CEP/199gbNduAR4O8XfMed9lqVcOe8z1wc9XDgMvAKMSMqTj/sxG7lyfj/62v4BsCjV36u9+6uzzRDOBd5R1b+raj2wGPhaUp6vAQ+78HJggrT/K5LZyJVzVPXPwL4WsnwNeEQ9XgZKRaRvB5ArL6jqblV9zYUP4n0fpF9Stpz3WZZy5RzXBx+707A7knex5Px+zFKuvCAiFcDFwP1psrRrf3U2g9AP2OE730nzGyORR72P/9QB5R1ALoDL3TLDchE5LUV6rslW7nww0k35nxGRs3LduJuqD8MbXfrJa5+1IBfkoc/c8scGPCfXz6tq2v7K4f2YjVyQn/vx18ANQDo3Fe3aX53NIJzIPAVUqmoV8DyNowCjOa/h+Wb5PHAn8EQuGxfvu+ErgO+p6oFctt0SGeTKS5+palRVh+J9i/1cERmci3YzkYVcOb8fReQS4CNVXd/ebaWjsxmEDwC/Ja9wcSnziEgIOBnYm2+5VHWvqh51p/cDX2hnmbIhm/7MOap6ID7lV9WngbC4b3S3N+J9A3wFsFBVH0uRJS99lkmufPaZa7MWWA1MTErKx/2YUa483Y+jgUkish1vWXm8iPw+KU+79ldnMwh/Bc4QkQEiUoD30GVlUp6VwHQXngz8Sd0TmnzKlbTOPAlvHTjfrASucTtnRgB1qro730KJSJ/4uqmInIv3f9zuSsS1+QCwRVV/mSZbzvssG7ny0Wci0ltESl24G/BlYGtStpzfj9nIlY/7UVVnq2qFqlbi6Yg/qepVSdnatb861fcQVDUiItcBz+Ht7FmgqptFZD5Qraor8W6c34nIO3gPLr/eQeSaKSKTgIiTa0Z7yyUij+LtPuklIjuBm/AesKGq9wJP4+2aeQc4BPxLe8uUpVyTgW+KSAQ4DHw9B0YdvBHc1cDf3PozwI1Af59s+eizbOTKR5/1BR4WkSCeAVqqqn/I9/2YpVw5vx/Tkcv+MtcVhmEYBtD5lowMwzCMY8QMgmEYhgGYQTAMwzAcZhAMwzAMwAyCYRiG4TCDYBiGYQBmEAzDMAzH/wOZL5Gm1RvsvQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAABUCAYAAABurecbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAGRRJREFUeJztnXt4VdWZ/z9vzjWXExOScA0YnFrBW70gg8WhlKkdRAWrCGq10P6mPj91FOt0WnB+RWFsdfrQ+fXnqA+jgmhHqhRrxT5SLzNQy4gdg1JLlRGtKFcJAUICOclJzvv7Y+9zcnJyTnICyTm5vJ/n2c9ee621136zcvZ3v+fd66wlqophGIYxOMjLtQGGYRhG9jDRNwzDGESY6BuGYQwiTPQNwzAGESb6hmEYgwgTfcMwjEGEib5hGMYgwkTfGBCIyE4RaRSRBhHZLyKrRKTILVslIiois5LO+b9u/nz32C8iPxGR3W47O0Xkp2mu8VniNQyjv2CibwwkrlTVIuA84HxgUULZB8A3Ygci4gXmAB8l1FkETAAmAiFgKvB2mmtc4Nb9P8lGiIPdW0afxD6YxoBDVfcDL+OIf4wXgUtEpNQ9ng68C+xPqHMR8Lyq7lWHnar6VJpr7AHWA2cDiMhGEfmhiPwXcBw4TURGisg6ETkkIh+KyLdj54vIvSKyVkSeFZF6EXlbRL7QQ11gGGkx0TcGHCJSCVwGfJiQHQZeAK5zj78BJAv6m8BdInKriJwjItLJNUYDM4B3ErJvAm7G+ZbwCfAMsBsYCcwGfiQi0xLqzwJ+AQwBVgO/EhFfN/5Uw+g2JvrGQOJXIlIP7AIOAPcklT8FfENESoAvAb9KKr8f+Gfg60A1sEdE5qW4xhFgE/Bb4EcJZatU9U+q2gIMByYD31fVsKpuBR4nIcQEbFHVtaoaAf4FCAKTTuQPN4xMMdE3BhJXqWosFj8OKE8sVNVNQAXwj8CvVbUxqbxVVR9W1clACfBDYKWIjE+6Romqnqqqtya1sSshPRI4pKr1CXmfAKNS1VfVKG3fCgyj1zDRNwYcqvpbYBWwLEXxvwN/T8fQTnIbjar6MHAYODPTSyek9wJDRCSUkDcG2JNwPDqWcF/8VrrnGUavYaJvDFR+Clya4uXog8ClwOvJJ4jInSIyVUTyRcTrhnZCtI/bZ4Sq7gLeAO4XkaCInAv8L5yHTowLReRqdyTRnUATznsFw+g1vLk2wDB6A1WtEZGngMVAfUL+IeA/0px2HPgJ8Dkcr/0D4BpV/fMJmnE9sBzHez8M3KOqryWUvwDMBZ7Eeel8tRvfN4xeQ2wRFcPIPiJyL/A5Vb0x17YYgwsL7xiGYQwiTPQNwzAGERbeMQzDGESYp28YhjGI6HOjd8rLy7WqqqpdXigUYv78+VRWVtLJL+ONDFBVdu/ezapVq6ivr+/6BMMw+gVbtmw5qKoVXdXrc+GdCRMmaHV1dbu8jz/+mFAoRFlZmYn+SaKq1NbWUl9fz9ixY3NtjmEMClSV+qYWDh9rpvZYc7v9oYRtVGk+S2edfULXEJEtqjqhq3p9ztNPRTgcpqqqygS/BxARysrKqKmpybUphtFvibRGHcE+3syhBnd/LP12+HgzkdbUDrbfk8eQQj9DCv0MLQ70uu39QvQBE/wexPrSMNpQVRqaWlILtivqh4+399Drwy1p2zsl30dZoZ/SQj+VpQV8obKE0kJ/PC95X+j3ZPWe7DeibxiGkQmR1iiHkzzvDmGV483UumJ++FiE5tZoyrYSvfAhhX5GlxbE03HhLvBTVuTsSwt8eD19e3yMiX6GHDlyhNWrV3Prrbd267wZM2awevVqSkpK0tZZvHgxU6ZM4Stf+crJmmkYA4p0Xnii551cdrQLLzwm2n3RC88GGYm+iEwH/h/gAR5X1QeSyk8FVuJMW3sIuFFVd7tl/wxc7lb9J1V9todszypHjhzhkUce6SD6LS0teL3pu/Gll17qsu2lS5eetH2G0R/oTS+8Mo0XHssrKfDh6+NeeDboUvRFxAM8jDMz4W7gLRFZp6rvJVRbBjylqk+6KwPdD9wkIpfjrCV6HhAANorIelU9eqIGL3nxT7y394RPT8mZI4u558qzOq2zcOFCPvroI8477zx8Ph/BYJDS0lK2b9/OBx98wFVXXcWuXbsIh8MsWLCAm2++GYCqqiqqq6tpaGjgsssu45JLLuGNN95g1KhRvPDCC+Tn5zN//nyuuOIKZs+eTVVVFfPmzePFF18kEonwi1/8gnHjxlFTU8MNN9zA3r17ufjii3n11VfZsmUL5eXlndptGL1JbFTKwfomauqbONjQTE192N03UXusybzwPkYmnv5E4MPYTIMi8gzOMm+Jon8mcJeb3kDbikRnAq+7Kwm1iMi7OGuTrukB27PKAw88wLZt29i6dSsbN27k8ssvZ9u2bfFhjytXrmTIkCE0NjZy0UUXcc0111BWVtaujR07dvDzn/+cxx57jDlz5vDcc89x440d59sqLy/n7bff5pFHHmHZsmU8/vjjLFmyhGnTprFo0SJ+85vfsGLFiqz83cbg5FhTiyviTe32Na6YJ+Y3tXT0xD15Qlmhn7KiAGWFfkaVFrR53kV+hiR44OaFZ5dMRH8U7VcE2g38ZVKdPwBX44SAvgaERKTMzb9HRH4CFABfpv3DAgARuRlnbVHGjBnTqTFdeeTZYuLEie3GuT/44IM8//zzAOzatYsdO3Z0EP2xY8dy3nnOWt0XXnghO3fuTNn21VdfHa/zy1/+EoBNmzbF258+fTqlpaUpzzWMdDQ2t7rC3ZRS0A82OGUH65tpjLR2OD9PYEhhgPIiPxWhAKeVF1IRClBeFEjaO+Kel2deeF+kp17kfhd4SETm4yxOsQdoVdVXROQinMUkaoDNQIdPk6o+CjwKzo+zesimXqWwsDCe3rhxI6+99hqbN2+moKCAqVOnEg6HO5wTCLSNwfV4PDQ2Nnaok1jP4/HQ0pL+67BhhCOtCaLd0QtP3B9r7ijkAEMK/VQUBSgP+blwTGkKEXf2Qwr9eEzI+z2ZiP4eEpZ1w1nSLXHJN1R1L46nj4gU4Sw8ccQt+yHOWqOIyGqchSn6HaFQKO20BXV1dZSWllJQUMD27dt5882eX/xo8uTJrFmzhu9///u88sorHD58uMevYfQNmluiHGxIJd7NcU/9oLtPN168pMDniHVRgHMqS+Ki7uyd/JiQW1hlcJGJ6L8FnC4iY3HE/jrghsQKIlKOswh0FFiEM5In9hK4RFVr3eXizgVe6UH7s0ZZWRmTJ0/m7LPPJj8/n2HDhsXLpk+fzvLlyxk/fjxnnHEGkyZN6vHr33PPPVx//fX87Gc/4+KLL2b48OGEQqGuTzT6BJHWKIeONacJr7R/+VnXmHrxrFDQGxft8SOKmRJqC7XEvPGKUICywgB+rwm5kZqM5t4RkRk4a456gJWq+kMRWQpUq+o6EZmNM2JHccI7t6lqk4gEgbfdZo4C/1tVt3Z2rVRz77z//vuMHz++m3/awKKpqQmPx4PX62Xz5s3ccsstbN3aaVd2ivXpydMaVWqPOTHwRO87vk8Q9UPHmlO2Uej3pImLt8XHY8dBnyfLf6HRn+jRuXdU9SXgpaS8xQnptcDaFOeFcUbwGCfJp59+ypw5c4hGo/j9fh577LFcmzQgUVWONrZQ0xDmgDsMsd2W4KUfOtZMNIXPFPTlOV53UYCqskIuqhrSQcxj4ZYCv/0+0sgu9onrJ5x++um88847uTaj3xKOtB+5UlPf1F7UYx56fVPKHwP5PXmO110cpLI0n/PHlLSLjyfubfy40Zcx0Tf6La1R7RAnbxP0cLv8VC88RWBIgT8eC/+LisK4Fx7LGxoKUFEUpDjfa0JuDAhM9I0+RWyuleRwSgfvvKGJ2oamlOGVWJy8IhRg3PAQU06v6CDmNnLFGKyY6BtZITYMMb2Yh+N54UjH8Io3T+Lx8OGnBDln1CkMLQ50EPPyogCFAftYG0Y67O4wTphoVDnSGEkQ8nDKl54H6ps4cjz1MMSSAl9ctC8YU+qEU2JbUTCeLsn32S88DaMHMNHvJYqKimhoaGDv3r3ccccdrF3bYXBTl2zdupW9e/cyY8YMANatW8d7773HwoULe9rcdhxvbkkp3qlGsKRaDSjgzXO88KIAY8sLmTh2CBVFwXhefDx5kZ+A14YhGkY2MdHvZUaOHHlCgg+O6FdXV8dFf+bMmcycOfOk7FFVWqJKONLKv/32I/bVhTuIeUNTx5eeeQJlRW2i/flhoXhoJVnMiwL20tMw+ir9T/TXL4T9f+zZNoefA5c90GmVhQsXMnr0aG677TYA7r33XrxeLxs2bODw4cNEIhHuu+8+Zs2a1e68nTt3csUVV7Bt2zYmTZrEihUrOOssZ9K4qVOnsmzZMqLRKAsWLCAcDpOfn88TTzzB2LFjWbx4MY2NjWzatIlFixbR2NhIdXU1Dz30EDt37uRb3/oWBw8epKKigieeeIIxY8Ywf/58iouLqa6uZv/+/dz3o/uZfuXXCEda3S1KSzTKwYZm7l//ifMrT1e8zxpZnBBaCTC0ONju5/o274ph9H/6n+jniLlz53LnnXfGRX/NmjW8/PLL3HHHHRQXF3Pw4EEmTZrEzJkz03q5c+fOZc2aNSxZsoR9+/axb98+JkyYwNGjR/nd736H1+vltdde4+677+a5555j6dKlcZEHWLVqVbyt22+/nXnz5jFv3jxWrFjB391+O08/u9YZj/7JblY+t57t72/njm9dz9mX/A15IgR8eRQHvQT9HqKH/WxdfCklBf5e7zvDMPoO/U/0u/DIe4vzzz+fAwcOsHfvXmpqaigtLWX48OF85zvf4fXXXycvL489e/bw2WefMXz48JRtzJkzh69+9assWbKENWvWMHv2bMCZsG3evHns2LEDESESSf3SE5zwTGNzC2+8sZmHVj7Nn2samPCVq/juP3yPjw8eo7G5lS99dQZBn5dJF57L4doaPj8sRMCb1+5hVOP1mOAbxiCk/4l+Drn22mtZu3Yt+/fvZ+7cuTz99NPU1NSwZcsWfD4fVVVVKadUjjFq1CjKysp49913efbZZ1m+fDkAP/jBD/jyl7/M888/z86dO5k6dSqqSms0SnNLlAP1YcKRKPvrwhw6FmHHgQai6vwwqaggQHHQS54Ip1UUcUqBj9OGlVBV7kz9rKo2Z4thGHHslyndYO7cuTzzzDOsXbuWa6+9lrq6OoYOHYrP52PDhg188sknGbXx4x//mLq6Os4991wAjtTVUTFsOIeONfOvyx+jJaq8v6+euhYv+2sPs78uzLGmFrx5Qr4/jzFDCpj8xS/yx9df4vShIX67/nmmTPkrigKO+BuG0U+ItkLzcTh+CI7uc7Zexjz9bnDWWWdRX1/PqFGjGDFiBF//+te58sorOeecc5gwYQLjxo3r9HxVZdbXrmbBggXc9b1FfHroOOFIK7Pn38rC79zCkn+6jyl//TcAFAe9XDn9Up7+twe56Yqp3L1oEeWhAAV+LyUFfh5++CG++c1vsmzZsviLXMMwuokqtEagJQwtTQn7xoTj5LIwRNLkdzg3xT6SUB5NCuVWXgR/+1qv/skZTa2cTQbK1MpRVZrc0TKNSSNnYvg8eeT7PAR9eQR9HoI+T4fYe2/RH/vUGKBEo+nFNW1+sviexLmcpAZ6AuANgjcAvmBb2psqnXjs7hPPKRoOZ0w/ITN6dGplIz2J494bXWEPR1ppikRR98OUJ0LA2zZyJuh1hN5r874Y/QVViByHpvr2W3ND58eRxq7FtzX1WgMZI3ngze8opnFRzYf80jTlqc4LphDvNOd4/JDXv+7jjERfRKbjLHruAR5X1QeSyk/FWS2rAjgE3Kiqu92yHwOX47w/eBVYoH3t60WGdMd7Lw56s+69G0Y7VB3RTRbiuDgfhaaG1GLdLq8BmutBO86J1AHxQCDkbP4i8Bc64hgsSe/dnqwQ53mdKVONjOhS9N0lDx8GLgV2A2+JyDpVfS+h2jLgKVV9UkSm4ayidZOIfBGYjLNMIsAm4EvAxu4aqqpZE85E793x4Dt67yJCMOa9u+LeX7z3fvrMHRyoOh5wU70jtDHRbSfUiXmp6tS3CbqmXgy9HZIHfleoA0Vtol08si0dE/HOjgMhR4RNgPs0mXj6E4EPVfXPACLyDDALSBT9M4G73PQG4FduWoEg4AcE8AGfddfIYDBIbW0tZWVl6YVfFaIt7gdOnA+yY3CnbWfqvQcHiPeuqtTW1hIMBnNtysCipTlJhGNCfLS9x5yyTpJ4J7/cS4kkia6bLhoKgeK243id5LyEY1+BCfUgIhPRHwXsSjjeDfxlUp0/AFfjhIC+BoREpExVN4vIBmAfjug/pKrvJ19ARG4GbgYYM2ZMBwMqKyvZvXs3NTU16a2MtsLRPSkKJP6BVtr2ivOc0IRjEERiG4jkOUMgRQgD4dgDxTG6Q/tO2t1nlKbtIZW2zZ4nGAxSWVnZa+13G1Xn/xdtSdiSj1PltToC2Wl5F220ZnKNWDrihEuaErzpmKBnGpf2FyUIsbsvHNsxzx/qKOiJ3rivsN/Fko2+QU+9yP0u8JCIzMdZGH0P0CoinwPGAzGFeVVE/kpVf5d4sqo+CjwKzuid5MZ9Ph9jx47t3ILm4/DOGxCN0BJp4tDRY9TWNXCo/hh1Dcc5euw4LZFmfLTilRaKfUppUCjxQ7EfinxKfl4rEm1xbuDYvjXSPt0acW7+k335lAl5PvD42vaexGM/eLwJ6RR1E+t5/O3LPvE5+RkJbmfiGemGQKc4bo1kFoLo9b72upsP8jwJx96EY4/jFQdCUDI6haddnCTexe29bX+h04Zh5JBMRH8PMDrhuNLNi6Oqe3E8fUSkCLhGVY+IyLeBN1W1wS1bD1wMtBP9nuBgs4f7Pr6I7fvr+fBAAy3ukkp+bx5nDAsx7tQQ40cU8xcjihk3PERp4UlOQRAXS/cB0Bp7WEQ6PhzalaWoF3/IRDIoS/UAirSNNc70eqmGqeX5UghdBsexB4kvvxvnJ+d1JrYZtuHJpI0UeZJn4Q1j0JCJ6L8FnC4iY3HE/jrghsQKIlIOHFLVKLAIZyQPwKfAt0XkfpwYxpeAn/aQ7e0oCnh5a+dhPj+siGnjhjJuRDFnjghRVVbYOy9XRVwP2uuIXX8j2uo8AERcwbVQgWEMBroUfVVtEZG/A17GGbK5UlX/JCJLgWpVXQdMBe4XEcUJ79zmnr4WmAb8Ece1/I2qvtjzfwYEfR7+a+G03mh6YJLnsVCDYQxC+twvckWkBuh6Epv0lAMHe8icnsTs6h5mV/cwu7rHQLTrVFWt6KpSnxP9k0VEqjP5KXK2Mbu6h9nVPcyu7jGY7bJArmEYxiDCRN8wDGMQMRBF/9FcG5AGs6t7mF3dw+zqHoPWrgEX0zcMwzDSMxA9fcMwDCMNJvqGYRiDiH4p+iIyXUT+R0Q+FJGFKcoDIvKsW/57EanqI3bNF5EaEdnqbn+bJbtWisgBEdmWplxE5EHX7ndF5II+YtdUEalL6K/FWbJrtIhsEJH3RORPIrIgRZ2s91mGdmW9z0QkKCL/LSJ/cO1akqJO1u/JDO3KyT3pXtsjIu+IyK9TlPVef6lqv9pwfhX8EXAazpTNfwDOTKpzK7DcTV8HPNtH7JqPM9NotvtsCnABsC1N+QxgPc5UGZOA3/cRu6YCv85Bf40ALnDTIeCDFP/LrPdZhnZlvc/cPihy0z7g98CkpDq5uCczsSsn96R77buA1an+X73ZX/3R04/P76+qzUBsfv9EZgFPuum1wF9L709+n4ldOUFVX8dZ0Swds3AWwVFVfRMoEZERfcCunKCq+1T1bTddD7yPM8V4IlnvswztyjpuHzS4hz53Sx4hkvV7MkO7coKIVOKsKPh4miq91l/9UfRTze+f/MGP11HVFqAOKOsDdgFc44YD1orI6BTluSBT23PBxe7X8/Uicla2L+5+rT4fx0tMJKd91oldkIM+c0MVW4EDwKuqmra/snhPZmIX5Oae/CnwPSDdGpS91l/9UfT7My8CVap6Ls56wU92UX+w8zbOfCJfAP6VthXZsoI404Q/B9ypqkezee3O6MKunPSZqraq6nk4U69PFJGzs3HdrsjArqzfkyJyBXBAVbf09rVS0R9Fv8v5/RPriIgXOAWozbVdqlqrqk3u4ePAhb1sU6Zk0qdZR1WPxr6eq+pLgE+cabx7HRHx4Qjr06r6yxRVctJnXdmVyz5zr3kEZ8nU6UlFubgnu7QrR/fkZGCmiOzECQNPE5F/T6rTa/3VH0U/Pr+/iPhxXnKsS6qzDpjnpmcD/6nuG5Fc2pUU852JE5PtC6wDvuGOSJkE1KnqvlwbJSLDY3FMEZmI83ntdaFwr7kCeF9V/yVNtaz3WSZ25aLPRKRCRErcdD5wKbA9qVrW78lM7MrFPamqi1S1UlWrcHTiP1X1xqRqvdZfPbVcYtbQzOb3XwH8TEQ+xHlReF0fsesOEZkJtLh2ze9tuwBE5Oc4ozrKRWQ3cA/OSy1UdTnwEs5olA+B48A3+4hds4FbRKQFaASuy8LDGxxP7Cbgj248GOBuYEyCbbnos0zsykWfjQCeFBEPzkNmjar+Otf3ZIZ25eSeTEW2+sumYTAMwxhE9MfwjmEYhnGCmOgbhmEMIkz0DcMwBhEm+oZhGIMIE33DMIxBhIm+YRjGIMJE3zAMYxDx/wFcXQTX7+b9vwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAABUCAYAAACC9EBIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAFxpJREFUeJztnXuYVMWVwH+nZwYQGQMOGB4qg4lZBwYcXmaEZUVQFx+ZEHmuj0B2NYmrYdWoAfwEYUNi/DQPg67xgWhEmRESJVlUQgKrfAHNoGBA8JUdkZcMowyYBWSmz/5xq3tu3+me7nl1D8P5fV9/XbfqVNWp6q4699a995SoKoZhGIYRyrQChmEYRtvADIJhGIYBmEEwDMMwHGYQDMMwDMAMgmEYhuEwg2AYhmEAZhAMI4qITBeRdWmsr0JELkpXfYaRDDMIxgmBiKwVkU9FpGMLlXe3iBwTkUPu866ILBSRXs0o7+mW0M0wmooZBKPdIyL5wChAgZIWLLpUVXOBU4FvAD2BjU01CoaRacwgGCcC3wQ2AIuBaZFIEckTkRUiclBEXge+5M8kIr8QkY9c+kYRGRWvcFU9pqpbgSlAJfB9XxlXiMgmETkgIn8WkUHB/CIyDpgNTBGRz0Rks4v/lohsc1cgfxOR7zS3IwyjIcwgGCcC3wSWuM8/i8gXXfyDwBGgF/Cv7uPnL0AR3hXAM8BzItIpUSWqWgu8gHc1gogMBhYB3wHygF8BK4LLVqr6EvAjvCuOLqp6rkvaB1wBnAJ8C/iZiAxpdOsNI0XMIBjtGhH5R6AvUKaqG4EPgKtEJAuYAMxR1b+r6hbgSX9eVX1aVatUtUZV7wc6Av+QpMrdeAYE4NvAr1T1NVWtVdUngaNAcSq6q+p/q+oH6vE/wCqcsTGM1sAMgtHemQasUtX97vgZF9cDyAY+8sl+6M8oIre5JZtqETkAfAHonqS+PsAnLtwX+L5bLjrgyjgD6J2K4iJyqYhsEJFPXN7LUqjfMJpMdqYVMIzWQkROAiYDWSKy10V3BLoCXwRq8Cbo7S7tTF/eUcAdwFhgq6qGReRTQBqoLwR8DVjtoj4CFqjqghTUjXE77JaVluMtd72gqsdE5PmG6jeM5mJXCEZ7ZjxQC/THuxdQBBQAr+JNtL8B7haRziLSH98NZyAXz2BUAtkiMgdvLb8eIpItIgXAs3hPGv3UJT0KfFdEvioeJ4vI5SKSG6eYj4F8Z1QAOuAZr0qgRkQuBS5pUi8YRoqYQTDaM9OAJ1R1h6rujXyAhcDVwE1AF2Av3hNIT/jyvgy8BLyLt5R0hNjlJXBPBQHVwAqgChiqqrsBVLUcuN7V9ynwPjA9ga7Pue8qEXlDVQ8BM4Ayl/cqV4dhtBpiG+QYhmEYYFcIhmEYhsMMgmEYhgGYQTAMwzAcZhAMwzAM4Dh7D6F79+6an5+faTUMwzCOKzZu3LhfVXskkzuuDEJ+fj7l5eWZVsMwWhRVJay+bxRVUIWwKmFVFNCwl9aQrALhsIsLyBI9drLuGyL1NCyrkfJ9eTWif5iYtHiyxBw7GX8b/Pm9jqlrj3cYrc+fz98P1JOP1Se2X7wwGluvX99gOyLpEGlzrO74dAz79fXrHdCdBO0PtvO+SefSu+tJTfqPiciHyaVSNAjOG+MvgCzgMVW9J5DeF8+JVw+81/avUdWdLu0nwOVO9D9VtdTFLwYuwHuGG2C6qm5KRZ9ME/mxa8PqfdT7DrtwOCaOurAqNbXed61fNhqmXlykjnC9cKxsWJWacCI9iJNfY/T3ZIkpz//nh9gBGYmNDl5/fGSgRcLBY5c5GBe33HpxwXLr8sWtO1AuCeuKLZdkMnH6JmZCjpZVN3H7J8HY/jXSgYj3mndIxIW9iJAL+9NdEiLipYu4Y4BIXF2+kJfgxbnjSH6JxvvCkfp8cXV6eREhnw5CnZFqTZIaBOcE7EHgYmAn8BcRWaGqb/vE7gOeUtUnRWQM8GPgWhG5HBiC94ZoR2CtiLyoqgddvttVdVkLticuP1q5jfKKT+pNeLWBiTMchppwOGYCjZnco9+trXHLkRUSskQIhXDfEo3LcuFQTJhoHNT9Gb1w7J8Z6v68MTLx8kVGmMsjIRBCdeUEyvXX7a8rWC6+gRyvnLh1JymXwASRqNxg3SH/5OHkIxOHN9gjk0ddWt3A900yUTmJnax8k0Qo5JtwfBNQdFKR2LrxpcVMVv68obp+aFDWtSEyMfpl69UTkJV6bapfp/h+g+BkLD6ZYFkEygjmM5KTyhXCecD7qvo3ABFZCnwd8BuE/sCtLrwGeN4X/4qq1uC9fv8WMA7v7cu0kR0SOnfI9iZD34SXFXITpNRNjNmROP8EGpCNfGdnRcohMKmKbyKuS88OhWJkg3VnheqXlRWoM1YPYuKy49RtGIaRKqkYhD7EvrK/E/hqQGYzcCXestI3gFwRyXPxc0XkfqAzcCGxhmSB8xHzR2Cmqh4NVi4i38ZzI8yZZ54ZTE6JO8ad06R8hmG0LseOHWPnzp0cOXIk06q0Czp16sTpp59OTk5Ok/K31E3l24CFIjIdeAXYBdSq6ioRGQ78Gc9J13o8Z2MAs/B8yHQAHgF+AMwPFqyqj7h0hg0bdhwt1hiGkYydO3eSm5tLfn6+Les0E1WlqqqKnTt30q9fvyaVkcp7CLvwXARHON3F+RXZrapXqupg4E4Xd8B9L1DVIlW9GG9p710Xv8dt/HEUz6nYeU1qgWEYxy1HjhwhLy/PjEELICLk5eU162orFYPwF+BsEeknIh2AqQS8LopId5/b3ll4TxwhIllu6Qi3l+wgvF2fiGxELt4/YTywpcmtMAzjuMWMQcvR3L5MumSkqjUichOeO+AsYJGqbhWR+UC5qq4ARgM/FhHFWzK60WXPAV51Sh7Eexy1xqUtEZEeeFcNm4DvNqslhmEYRrNIyXWFqq5U1a+o6pciuz+p6hxnDFDVZap6tpO5LnJzWFWPqGp/9yn2v2egqmNUdaCqFqrqNar6WWs00DAMIxEHDhzgoYceanS+yy67jAMHDjQoM2fOHFavXt2gTFvDfBkZhnHCksgg1NTUxJGuY+XKlXTt2rVBmfnz53PRRRc1S790YwbBMIwTlpkzZ/LBBx9QVFTE8OHDGTVqFCUlJfTv3x+A8ePHM3ToUAYMGMAjjzwSzZefn8/+/fupqKigoKCA66+/ngEDBnDJJZdw+PBhAKZPn86yZcui8nPnzmXIkCEMHDiQ7du9bbwrKyu5+OKLGTBgANdddx19+/Zl//79ae6FOo4rX0aGYbRf5v1uK2/vPphcsBH0730Kc782IGH6Pffcw5YtW9i0aRNr167l8ssvZ8uWLdHHNhctWsSpp57K4cOHGT58OBMmTCAvLy+mjPfee49nn32WRx99lMmTJ7N8+XKuueaaenV1796dN954g4ceeoj77ruPxx57jHnz5jFmzBhmzZrFSy+9xOOPP96i7W8sKV0hiMg4EXlHRN4XkZlx0vuKyB9F5C0RWSsip/vSfiIiW9xnii++n4i85sosdU8wGYZhZIzzzjsv5hn+Bx54gHPPPZfi4mI++ugj3nvvvXp5+vXrR1FREQBDhw6loqIibtlXXnllPZl169YxdepUAMaNG0e3bt1asDWNJ5O+jH4C/ExVl4rIw8C/Af/Vko0zDOP4oaEz+XRx8sknR8Nr165l9erVrF+/ns6dOzN69Oi4z/h37NgxGs7KyoouGSWSy8rKSnqPIlOkcoUQ9WWkqp8DEV9GfvoDf3LhNb70qC8jVf078BYwzr17MAaIOLZ7Eu9dBMMwjLSRm5vLoUOH4qZVV1fTrVs3OnfuzPbt29mwYUOL1z9y5EjKyjzXbqtWreLTTz9t8ToaQyoGIZ4voz4BmYgvI6jvy2iciHQWke54vozOAPKAA753EuKVCXi+jESkXETKKysrU2mTYRhGSuTl5TFy5EgKCwu5/fbbY9LGjRtHTU0NBQUFzJw5k+Li4havf+7cuaxatYrCwkKee+45evbsSW5ubovXkyqiSXxsi8hEYJyqXueOrwW+qqo3+WR6AwuBfngvpk0AClX1gIjcCUzC82W0D+/N56eBDar6ZZf/DOBFVS1sSJdhw4apbZBjGO2Hbdu2UVBQkGk1MsbRo0fJysoiOzub9evXc8MNN7BpU/O2hYnXpyKyUVWHJcubylNGKfkywl0hiEgXYILflxGwwKU9g+fLqAroKiLZ7iqhXpmGYRjtnR07djB58mTC4TAdOnTg0Ucfzag+qRiEqC8jvEl7KnCVX8AtB32iqmECvoyArqpa5fdlpKoqImuAiXj3JKYBL7RQmwzDMI4Lzj77bN58881MqxEl6T0EdwYf8WW0DSiL+DISkRInNhp4R0TeBb6IuyKgzpfR23gurP2+jH4A3Coi7+PdU8jsA7iGYRgnOCm9mKaqK4GVgbg5vvAy6p4Y8sscwXvSKF6Zf8NcXhuGYbQZzHWFYRiGAZhBMAzDMBxmEAzDMFKkS5cuAOzevZuJEyc2qYxNmzaxcmXdCvyKFSu45557WkS/5pIOX0b3ishWEdkmIg+4t5Rxcu+IyCb3Oa3lmmUYhtF69O7dO+rJtLEEDUJJSQkzZ9abVjNCUoPg82V0Kd4N4n8RkeCN4ogvo0HAfDxfRojICGAk3uOmhcBw4AJfvqvdfstFqrqvuY0xDMNoDDNnzuTBBx+MHt9999388Ic/ZOzYsVFX1S+8UP+J+IqKCgoLvfdoi4uL2bp1azRt9OjRlJeX8/rrr3P++eczePBgRowYwTvvvMPnn3/OnDlzKC0tpaioiNLSUhYvXsxNN90ULXfMmDEMGjSIsWPHsmPHDsBzpT1jxgxGjBjBWWed1WRjlIxUnjKK+jICEJGILyO/c7v+wK0uvAZ43oUV6AR0wNsqMwf4uPlqG4bR7nhxJuz9a8uW2XMgXJp4OWbKlCncfPPN3Hijt+tvWVkZL7/8MjNmzOCUU05h//79FBcXU1JSknC/4ilTplBWVsa8efPYs2cPe/bsYdiwYRw8eJBXX32V7OxsVq9ezezZs1m+fDnz58+nvLychQsXArB48eJoWd/73veYNm0a06ZNY9GiRcyYMYPnn/em0z179rBu3Tq2b99OSUlJk5esGqJVfRmp6no8A7HHfV5W1W2+fE+45aK7JEFvmy8jwzBai8GDB7Nv3z52797N5s2b6datGz179mT27NkMGjSIiy66iF27dvHxx4nPYydPnhw9Yy8rK4tO1NXV1UyaNInCwkJuueWWmKuIRKxfv56rrvLe+7322mtZt25dNG38+PGEQiH69+/foD7NoaU2yLkNWCgi0/F8Ge0CakXky0ABnmsKgD+IyChVfRVvuWiXiOQCy4FrgaeCBavqI3gvtTFs2LCGHS8ZhnH80sCZfGsyadIkli1bxt69e5kyZQpLliyhsrKSjRs3kpOTQ35+fly31xH69OlDXl4eb731FqWlpTz88MMA3HXXXVx44YX89re/paKigtGjRzdLT7+b7WQ+6JpKKlcIKfkyUtUrVXUwcKeLO4B3tbBBVT9T1c+AF4HzXfou930IeAZ7Sc0wjAwwZcoUli5dyrJly5g0aRLV1dWcdtpp5OTksGbNGj788MOUyrj33nuprq5m0KBBgHeF0KePt5jiXxZqyOX2iBEjWLp0KQBLlixh1KhRzWxd40jFIER9GbldzaYCK/wCItJdRCJlRX0ZATuAC0QkW0Ry8G4ob3PH3V3eHOAKYEvzm2MYhtE4BgwYwKFDh+jTpw+9evXi6quvpry8nIEDB/LUU09xzjnnJC1j4oQJLF26lMmTJoGGQcPccdttzJo1i8GDB1Nz7HNPMFzDhReM4u23t1JUdC6lzy6BcK2Xp/YYv/z5/TyxaBGDBg7k1089yS/uvxdqjtbJtDJJ3V8DiMhlwM+BLGCRqi4QkflAuaqucC6yf4x3E/kV4EZVPeqeUHoI+CeX9pKq3ioiJzu5HFfmauBWVa1tSA9zf20c94TDbsKodYPcDfSw/zsQDofjxNWCamw5kTL8eWLiaqOTVWweV1a9uHDD5dSLq21c+1TZVnALBf16BzpJvdkiEk6YFic9Zj6LM7c1lF5vLtRAMEnZrU2PAsjplFSstd1fN8eXUS3wnTjxfweGplJ3u0C1gcGbaBJozOBNYRKIN3jrDXi/rLozEg2Ew27g+MKRM5fG5InKJMoTLz+BOpPlj1dnQ/lJoZ3BsiPpwYkwQZ8fz0gWSAhCWV445I7rxWWBSBy5LAiF6sqREK7TIxXUfYvvOPq4if+5k2BanPS4+fB0i40IBOPUEzctmY5BPRKUnWrerJa65ZuY1q+hLbDuZ7BncxPPbJpyNhcnT3tAQnh/UkkQDrnBFolPJBfMQ5z8oST5JfU89eLj5InJL0naFgxTN+G11IQZE+cvNxQoJxTIE5CtFxepI5Anoa4+vVqDbdug+1dap2yj0ZwYBuGT/4W9WwIDIs7gDWWB5DRukCQavHEngeDgTWHAxxu8DU4CDbRPpBGTQOQMzj8BG0bLo6oJn/E3Gkdznz46MQxCyQOZ1sAwjDh06tSJqqoq8vLyzCg0E1WlqqqKTp2S32dIREo3ldsKIlIJJH8GLD7dgf0tqE5LYXo1DtOrcbRpvXr06JG9YMGC/Pz8/JPagkEIh8OhUCjU5tZ4U9FLVamoqDh85513VlRWVtYEkvuqao9k9RxXBqE5iEh5KnfZ043p1ThMr8ZhejWOE10vc39tGIZhAGYQDMMwDMeJZBAeybQCCTC9Gofp1ThMr8ZxQut1wtxDMAzDMBrmRLpCMAzDMBrADIJhGIYBtEODkML+zx1FpNSlvyYi+W1Er+kiUunbY/q6NOi0SET2iUhcT7Pi8YDT+S0RGdLaOqWo12gRqfb11Zx4cq2g1xkiskZE3nb7hP9HHJm091mKeqW9z0Skk4i8LiKbnV7z4sikfTymqFfax6Ov7iwReVNEfh8nrXX7S1XbzQfPc+oHwFl423ZuBvoHZP4deNiFpwKlbUSv6cDCNPfXPwFDgC0J0i/D28NCgGLgtTai12jg9xn4f/UChrhwLvBunN8x7X2Wol5p7zPXB11cOAd4DSgOyGRiPKaiV9rHo6/uW/H2iKn3e7V2f7W3K4To/s+q+jkQ2f/Zz9eBJ114GTA20fadadYr7ajqK8AnDYh8HXhKPTYAXUWkVxvQKyOo6h5VfcOFDwHbqL+dbNr7LEW90o7rg8/cYY77BJ9iSft4TFGvjCAipwOXA48lEGnV/mpvBiGV/Z+jMqpaA1QDeW1AL4AJbplhmYicESc93aSqdyY4313yvygiA9JdubtUH4x3dukno33WgF6QgT5zyx+bgH3AH1Q1YX+lcTymohdkZjz+HLgDSOSmolX7q70ZhOOZ3wH5qjoI+AN1ZwFGfd7A881yLvBL4Pl0Vi4iXfD2Ab9ZVQ+ms+6GSKJXRvpMVWtVtQhv693zRKQwHfUmIwW90j4eReQKYJ+qbmztuhLR3gxC0v2f/TIikg18AajKtF6qWqWqR93hY7SNDYRS6c+0o6oHI5f86m3elCNuS9bWRrwtX5cDS1T1N3FEMtJnyfTKZJ+5Og8Aa4BxgaRMjMekemVoPI4ESkSkAm9ZeYyIPB2QadX+am8GIen+z+54mgtPBP6k7g5NJvUKrDOX4K0DZ5oVwDfdkzPFQLWq7sm0UiLSM7JuKiLn4f2PW30ScXU+DmxT1Z8mEEt7n6WiVyb6TER6iEhXFz4JuBjYHhBL+3hMRa9MjEdVnaWqp6tqPt4c8SdVvSYg1qr91a72Q1DVGhG5CXiZuv2ft4pv/2e8gfNrEXkf78bl1Dai1wwRKQFqnF7TW1svEXkW7+mT7iKyE5iLd4MNVX0Yb9vUy4D3gf8DvtXaOqWo10TgBhGpAQ4DU9Ng1ME7g7sW+KtbfwaYDZzp0y0TfZaKXpnos17Ak+LtrR4CylT195kejynqlfbxmIh09pe5rjAMwzCA9rdkZBiGYTQRMwiGYRgGYAbBMAzDcJhBMAzDMAAzCIZhGIbDDIJhGIYBmEEwDMMwHP8P/P8YPJDgM7AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAABUCAYAAABurecbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAFpxJREFUeJztnXuQVNWdxz+/7umZniczzMCAgzqYNYoCC0oMDzUIiYuPoKsIxkcgqcQtjUGT2iWQraCwyWosdys+i42KDwICokZM+a5AKRWMgqKCoGgKAUEYEGZ4zLt/+8e93XO7p3umZ+jpHujfp+rWvfc8f/d03+89/bunzxFVxTAMw8gOfJk2wDAMw0gfJvqGYRhZhIm+YRhGFmGibxiGkUWY6BuGYWQRJvqGYRhZhIm+YcRBRGaIyJpM22EYqcZE38g6RGS1iBwQkbxM22IY6cZE38gqRKQauABQYHJGjTGMDGCib2QbPwTeBp4ApocDRaRcRFaKSJ2IvAN8w5tJRO4TkR1u/HoRucATd6eIPCMifxKRQyLykYh8U0TmiMheN9/Fabo+w+gQE30j2/ghsNjd/kVEKt3wh4AGYCDwY3fz8i4wAugLLAGeEZGgJ/77wCKgDHgfeBXn/qoC5gP/1xMXYxhdRWzuHSNbEJHzgVXAQFXdJyJbcMT4fhzBH6aqW9y0/w1cqKrnJyjrADBeVT8QkTuBcar6PTfu+8DTQB9VbRWRYqAOKFPVgz17lYbRMdbTN7KJ6cBrqrrPPV/ihvUDcoAdnrRfeDOKyL+LyGYRqRWRg0AfoMKTZI/nuB7Yp6qtnnOAotRchmF0n5xMG2AY6UBE8oGpgF9EvnKD84BSoBJoAU4Gtrhxp3jyXgDMAiYCm1Q15Pb0JU3mG0bKMNE3soUrgVZgGNDkCV+O4+d/DrhTRH4MVOP8AtjmpinGeSjUADkiMhsoSYvVhpFizL1jZAvTgcdVdbuqfhXegAeB64FbcdwvX+GM7Hnck/dV4BXgUxy3TwPRriDDOG6wF7mGYRhZhPX0DcMwsggTfcMwjCzCRN8wDCOLMNE3DMPIInrdkM2Kigqtrq7OtBmGYRjHFevXr9+nqv06S9frRL+6upp169Zl2gzDMIxOUVWaW5X65lYamlupb2qlocXZt4WFqG92z93wejdtoydtfXOI6vIC5l8xtFu2iMgXnafqhaJvGIaRClpavWIbihLbhmaPKMeEhUU6LOLhfI3NXsEORdK3hro+7D3gF4IBP/kBP/m5zj4v4Ke5teeH0JvoG0YCVJWm1hBHG1s50tTC0SbnBhcBnwgCiIBEjp29TwRxJ2jw+drShfMgIAg+ic6D0L5sxK2jfZ5w3b2d5uZmdu7cSUNDAwCqoGj0PpljnM8k5DnWRMduPR0RcLdiAL+zSTD2M/U5G7lxPw/n8wIin038NJHvh+c4EZs3b+7Q7mAwyKBBgwgEAp03fhxM9I0TgtaQctQV5iONzv5okyvWYdFubOFIU2sknVfMw3mONLVQ7zlv6UYvLt2EhcQXEZXEDwrEm877gGovWj5XmCROHjzi5YuqM/oh19wa4sah+Zx9aiU5JQNRhHh/CBU6nsiorS6nPp8I4sM9b7vGcFzbuSe9J87JHz99b0ZV2b9/Pzt37mTw4MHdKsNE30grqkpjSyhKZKMFuIUjja2RfX2zR5C9eRo9At3UQkNzKGkb/D6hINdPYW4OBXl+CnL9FOTmUFGUyyl5BRS654V5zj6cNj/XT45PIr3IUKRH6YhYSNt6p+E4otI5vdaQe6JAKKRR5dGubG+vV924mLCYsjvLE2WrJw8ooVB0zzrkHmhUHm+4ttkb0+sOqRLw+zi1NEB5eTl+n6+98CYj1JFfS71bkNOBiFBeXk5NTU23yzDRNxLS0hqK9IzDQnw05jyyb2qN6iFH9bDDvWv3vCud5/yAP0p8C3L9FOXl0L84LyLahbk5bfGRcz+FeW17pxznPC/HZwKSRjZv3szJfQszbcYJw7F+d030s4CG5lZqDjWyp66BPXXu/lADNXWN1DW0RET7aKNH1JtaaWpJvvec4xMK83KcXrK7z8/1M6AkGDmP7T3HCnOk9+2WkR/w4/eZOBvHzsGDB1myZAm33HJLl/JdeumlLFmyhNLS0oRp5s6dy4UXXsh3v/vdYzUzLZjoH8c0toTFvJG9dQ2umDeyt66RvYcaIiJfW9/cLm+u30e/4jz65AcoyPXTJz/ASX2CUcIcFu4oQXZ70oV5fvJz28Q8N8f+52f0Xg4ePMjDDz/cTvRbWlrIyUksgy+99FKnZc+fP/+Y7UsnJvq9kObWUFTPPCzge+saXVF3zg8cbS/mOT6hf3Ee/UuCDK4oZPRp5VSWBOlXnEdlSZDKkjwqi4OUFgTMxWFkDbNnz+bzzz9nxIgRBAIBgsEgZWVlbNmyhU8//ZQrr7ySHTt20NDQwG233cZNN90EtP1v6PDhw1xyySWcf/75/O1vf6OqqooXXniB/Px8ZsyYweWXX86UKVOorq5m+vTpvPjiizQ3N/PMM89w5plnUlNTw3XXXceuXbsYM2YMr7/+OuvXr6eioqITy1OPiX4aaWkNse9wkyvmDREBd8S8IdJj33+kqV1ev0/oV5RHZUkeJ/ctYFR1GZXFQfqXOAJfWewIellBLj5ziRi9lHkvbuLjXXUpLfOsk0q44/tnd5jm7rvvZuPGjWzYsIHVq1dz2WWXsXHjxsgImIULF9K3b1/q6+v51re+xdVXX015eXlUGVu3buXpp5/mkUceYerUqTz77LPccMMN7eqqqKjgvffe4+GHH+bee+/l0UcfZd68eUyYMIE5c+bwyiuv8Nhjj6WuAbpIUqIvIpOA+3BGsz6qqnfHxJ8KLMRZa/Rr4AZV3enG/R64zE36X6q6LEW29xpaQ8r+w41R/vI2F0tbj33/kcZ2Y4d9AhVFTi+8qjTIyFNKI2JeWZJH/+IglSVB+hbmmn/bMFLEeeedFzXk8f777+f5558HYMeOHWzdurWd6A8ePJgRI0YAcO6557Jt27a4ZV911VWRNM899xwAa9asiZQ/adIkysrKUno9XaFT0RcRP/AQ8D1gJ/CuiKxU1Y89ye4FnlLVJ0VkAnAXcKOIXAacA4zAWY90tYi8rKqpfdT3EKGQsv+I0zPfG+mJN7qi3ibo+w43thuRIgLlhY5wV5YEGT6oD/2K29wrYVdLeVGeibmRNXTWI08XhYVto4lWr17NG2+8wdq1aykoKGD8+PGRP5J5ycvLixz7/X7q6+vbpfGm8/v9tLS0pNjyYyeZnv55wGeq+g8AEVkKXAF4Rf8s4Jfu8Srgz57wN1W1BWgRkQ+BSTjrkmaMUEg5cLTJEW1XwGNdLHvqGqk53Bj3L9blhbmOS6Ukj7MGlnhcLGG/eZDyolwCfnu5aRi9geLiYg4dOhQ3rra2lrKyMgoKCtiyZQtvv/12yusfN24cy5cv51e/+hWvvfYaBw4cSHkdyZKM6FcRvR7oTuDbMWk+AK7CcQH9K1AsIuVu+B0i8j9AAXAR0Q8LAETkJuAmgFNOOaWLl9CGqnLwaHNEvPfUNXheiLYJes3hxrhzXJQVBKgsCdK/JMg3K4tdF0vQdbE4xxVFeTZSxTCOM8rLyxk3bhxDhw4lPz+fysrKSNykSZNYsGABQ4YM4YwzzmD06NEpr/+OO+7gBz/4AYsWLWLMmDEMGDCA4uLilNeTDJ2ukSsiU4BJqvoT9/xG4NuqeqsnzUk4C0wPBt4ErgaGqupBEflP4BqgBtgLvKuqf0hU36hRo7Q7s2zuqWvggt+voqm1/djyPvmBiGj3D/vL3V55uMferziPvBx/l+s1DKNjNm/ezJAhQzJtRkZpbGzE7/eTk5PD2rVrufnmm9mwYUO3y4vXpiKyXlVHdZY3mZ7+l8DJnvNBblgEVd2F09NHRIqAq1X1oBv3O+B3btwS4NMk6uwyfQtz+dH51VH+8vBQxWDAxNwwjMyxfft2pk6dSigUIjc3l0ceeSRjtiQj+u8Cp4vIYByxvxa4zptARCqAr1U1BMzBGckTfglcqqr7RWQ4MBx4LYX2Rwj4fcy5JLt7E4Zh9E5OP/103n///UybASQh+qraIiK3Aq/iDNlcqKqbRGQ+sE5VVwLjgbtERHHcOz9zsweAt9w/AdXhDOXsfa+zDcMwsoSkxumr6kvASzFhcz3HK4AVcfI14IzgMQzDMHoBNgzFMAwjizDRNwzDyCJM9A3DMGIoKioCYNeuXUyZMqVbZWzYsCFqls6VK1dy9913d5AjPZjoG4ZhJOCkk05ixYp2ryuTIlb0J0+ezOzZs1NlWrcx0TcM44Rn9uzZPPTQQ5HzO++8k9/+9rdMnDiRc845h2HDhvHCCy+0y7dt2zaGDh0KwOjRo9m0aVMkbvz48axbt4533nmHMWPGMHLkSMaOHcsnn3xCU1MTc+fOZdmyZYwYMYJly5bxxBNPcOutt0bKnTBhAsOHD2fixIls374dgBkzZjBz5kzGjh3Laaed1u0HTkfY1MqGYaSPl2fDVx+ltswBw+CSjt0m06ZN4/bbb+dnP3NGky9fvpxXX32VmTNnUlJSwr59+xg9ejSTJ09OuM7EtGnTWL58OfPmzWP37t3s3r2bUaNGUVdXx1tvvUVOTg5vvPEGv/71r3n22WeZP38+69at48EHHwTgiSeeiJT185//nOnTpzN9+nQWLlzIzJkz+fOfnSnLdu/ezZo1a9iyZQuTJ0/utnspESb6hmGc8IwcOZK9e/eya9cuampqKCsrY8CAAfziF7/gzTffxOfz8eWXX7Jnzx4GDBgQt4ypU6dy8cUXM2/ePJYvXx4R49raWqZPn87WrVsREZqb2y9uFMvatWsj0y7feOONzJo1KxJ35ZVX4vP5OOuss9izZ08Krj4aE33DMNJHJz3ynuSaa65hxYoVfPXVV0ybNo3FixdTU1PD+vXrCQQCVFdXx51SOUxVVRXl5eV8+OGHLFu2jAULFgDwm9/8hosuuojnn3+ebdu2MX78+GOy0zuFc2dzo3UH8+kbhpEVTJs2jaVLl7JixQquueYaamtr6d+/P4FAgFWrVvHFF18kVcY999xDbW0tw4cPB5yeflVVFRDtwuloOuexY8eydOlSABYvXswFF1xwjFeXPCb62UYoBEe/hv2fw851jn+19ktojr8ghGGcKJx99tkcOnSIqqoqBg4cyPXXX8+6desYNmwYTz31FGeeeWanZUyZMoWlS5cyderUSNisWbOYM2cOI0eOjFo05aKLLuLjjz+OvMj18sADD/D4448zfPhwFi1axH333Ze6C+2ETqdWTjfdnVo561CFpiNQ/zXUH3CEvP5rd3/QcxwTX38QSPCZ5wQhvy8U9IX8Mmcr6OsJ69s+LFgKfvMSGomxqZVTT09PrWz0NC1NjjDHFfAD0cfesNb2C6hHyC1yhbnM2fc5OY6Al0JLQ0w9HjtqPmk7DnUwT15en7Z6Ig+M2AdFTHxeibOmpGEYaSUdC6Pfg7Mwug94HbhNe9vPi1QRCkFjrUdEOxLwsMAegKb4fj8AfIFosS7/Ruc98PwyyMlN3XWpQmNdzPUciHM97rb/Mye+sbaD68qJeTiURT+kEl1jIJi66zKMLKSnF0YfC4zDmUcfYA3wHWB16i6hB1B1fNyJXCSJBK/hIGj7lbscxOlZh8WtqBL6DYnvSvEe5xZmvkcsAsE+zlZWnXy+1hanTaIefAl+zRzcDrs2OGEtiUdQECjwPBxiflEk+pWRXwo+W0jHMKDnF0ZXIAjkAoIzv37qB552RGtzxz5ur6/bG9bamLjMQKErKqWu62RYJ66Nvo5gZpvw+HOgsMLZukJzfQcPihgX155NbfEdPXCDfRK/l0j00M0tyvwD93jG84NeQ6GEf3oyYuiknY7VUdKjC6Or6loRWQXsxhH9B1V1c2wFKVkY/ejX8NJ/tBeIxrrEeXw50Td+WTVUjexcwHPyEpdpHDuBfOhT5WzJEgq5LqiY9xLx3GtHamDfJ066ZF1r4c/eH3DFTKP3ccNCnYThiQvFKSNRGN2oSz35kq2rC9cVW4aH4Oi72O8/RHlhjgl/ZwQKoN8ZCaNVlf379xMMdt/NmaoXuf8OPCgiM3BWzvoSaBWRfwKG4KyrC/C6iFygqm95M6vqH4E/gjN6p1sW+HLgy/XOjVlQDuWnd96byyu2ntyJgs/n/vIqhb5dyNfaHDO6KdF7mAPw9T+c9CKAOHvxtR0jTtemXVh470sQ5msfF6+MuOXSQbmxYd4y6KTcjsroqNz2cYOknp11tdQcKe7eZ5tN+FpgX7t+cRTBYJBBgwZ1mKYjenRhdBH5KfC2qh52414GxgBRop8SgiVwW/dXlzeyFH8Aivo7m9EjBIDBmTbCiNCjC6MD24GfishdOH2L7wB/6Kiy9evX7xORzv8al5gKYN8x5O8pzK6uYXZ1DbOra5yIdp2aTKKeXhh9BTAB+AjH0feKqr7YSX39kjE8ESKyLpk/KKQbs6trmF1dw+zqGtlsV08vjN4K/Nsx2mgYhmGkCJt7xzAMI4s4EUX/j5k2IAFmV9cwu7qG2dU1stauXjfhmmEYhtFznIg9fcMwDCMBJvqGYRhZxHEp+iIySUQ+EZHPRGR2nPg8EVnmxv9dRKp7iV0zRKRGRDa420/SZNdCEdkrIhsTxIuI3O/a/aGInNNL7BovIrWe9pobL10P2HWyiKwSkY9FZJOI3BYnTdrbLEm70t5mIhIUkXdE5APXrnlx0qT9nkzSrozck27dfhF5X0T+Eieu59pLVY+rDee/Ap8Dp+FM5PYBcFZMmluABe7xtcCyXmLXDJz5h9LdZhcC5wAbE8RfCryM8we60cDfe4ld44G/ZKC9BgLnuMfFwKdxPsu0t1mSdqW9zdw2KHKPA8DfgdExaTJxTyZjV0buSbfuXwJL4n1ePdlex2NPPzLrp6o2AeFZP71cATzpHq8AJkrPz/SUjF0ZQVXfxFnnIBFX4EyNrar6NlAqIgN7gV0ZQVV3q+p77vEhYDPOxINe0t5mSdqVdtw2OOyeBtwtdoRI2u/JJO3KCCIyCGedkUcTJOmx9joeRT/erJ+xX/xIGlVtAWqB8l5gF8DVrjtghYicHCc+EyRreyYY4/48f1lEzk535e7P6pE4vUQvGW2zDuyCDLSZ66rYAOwFXlfVhO2VxnsyGbsgM/fkH4BZQKL5wHusvY5H0T+eeRGoVtXhOKuIPdlJ+mznPeBUVf1n4AHa1mlIC+JMHvgscLuqdjBHd3rpxK6MtJmqtqrqCJwJGc8TkaHpqLczkrAr7fekiFwO7FXV9T1dVzyOR9HvdNZPbxoRyQH6APszbZeq7lfV8OosjwLn9rBNyZJMm6YdVa0L/zxXZyqQgDiT+/U4IhLAEdbFqvpcnCQZabPO7Mpkm7l1HsRZSGlSTFQm7slO7crQPTkOmCwi23DcwBNE5E8xaXqsvY5H0Y/M+ikiuTgvOVbGpFkJTHePpwB/VfeNSCbtivH5TsbxyfYGVgI/dEekjAZqVXV3po0SkQFhP6aInIfzfe1xoXDrfAzYrKr/myBZ2tssGbsy0WYi0k9ESt3jfJylVbfEJEv7PZmMXZm4J1V1jqoOUtVqHJ34q6reEJOsx9orVYuopA1NbtbPx4BFIvIZzovCa3uJXTNFZDLQ4to1o6ftAhCRp3FGdVSIyE7gDpyXWqjqApzJ9C4FPgOOAj/qJXZNAW4WkRagHrg2DQ9vcHpiNwIfuf5ggF8Dp3hsy0SbJWNXJtpsIPCkOOtp+4DlqvqXTN+TSdqVkXsyHulqL5uGwTAMI4s4Ht07hmEYRjcx0TcMw8giTPQNwzCyCBN9wzCMLMJE3zAMI4sw0TcMw8giTPQNwzCyiP8HRH8/dv2uBMYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(5,1,1)\n",
    "plt.plot(history_sgd.history['accuracy'])\n",
    "plt.plot(history_sgd.history['val_accuracy'])\n",
    "plt.legend(['training', 'valivation'])\n",
    "plt.title('SGD')\n",
    "plt.show()\n",
    "\n",
    "plt.subplot(5,1,2)\n",
    "plt.plot(history_adagrad.history['accuracy'])\n",
    "plt.plot(history_adagrad.history['val_accuracy'])\n",
    "plt.legend(['training', 'valivation'])\n",
    "plt.title('AdaGrad')\n",
    "plt.show()\n",
    "\n",
    "plt.subplot(5,1,5)\n",
    "plt.plot(history_rms.history['accuracy'])\n",
    "plt.plot(history_rms.history['val_accuracy'])\n",
    "plt.legend(['training', 'valivation'])\n",
    "plt.title('RMSProp')\n",
    "plt.show()\n",
    "\n",
    "plt.subplot(5,1,4)\n",
    "plt.plot(history_delta.history['accuracy'])\n",
    "plt.plot(history_delta.history['val_accuracy'])\n",
    "plt.legend(['training', 'valivation'])\n",
    "plt.title('AdaDelta')\n",
    "plt.show()\n",
    "\n",
    "plt.subplot(5,1,3)\n",
    "plt.plot(history_adam.history['accuracy'])\n",
    "plt.plot(history_adam.history['val_accuracy'])\n",
    "plt.legend(['training', 'valivation'])\n",
    "plt.title('Adam')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验小结\n",
    "\n",
    "在本实验中，你实现了通过 TensorFlow 快速的建立一个卷积神经网络，分别使用：SGD，AdaGrad，RMSProp，AdaDelta，Adam 优化器对 MNIST 数据集进行模型训练，并且测试不同的优化算法在相同的数据集上的准确率。你可以看到使用不同的优化器对我们的分类准确率有相当大的区别。相对而言，SGD 与其余几个优化器相比精确率表现较差。另外，虽然在之前的实验中讲解优化器背后的数学逻辑很复杂，但在应用实践中是很简单方便的，TensorFlow 还支持很多其他的深度学习优化器，详细信息可 [查阅官方文档](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}