17. MNIST 权重初始值对比.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实验：MNIST 权重初始值对比"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验概要\n",
    "\n",
    "深度学习框架中往往会包含大量可以被调用的初始化工具，用于定义设置神经网络各层权重随机初始值的方法。\n",
    "\n",
    "```python\n",
    "import tensorflow.keras as keras\n",
    "\n",
    "#  ######################可用初始化器汇总########################\n",
    "keras.initializers.Initializer()  # 初始化器基类：所有初始化器继承这个类。\n",
    "keras.initializers.Zeros()  # 将张量初始值设为 0 的初始化器。\n",
    "keras.initializers.Ones()  # 将张量初始值设为 1 的初始化器。将张量初始值设为 1 的初始化器。将张量初始值设为 1 的初始化器。\n",
    "keras.initializers.Constant(value=0)  # 将张量初始值设为一个常数的初始化器。value: 浮点数，生成的张量的值。\n",
    "keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None)  # 按照正态分布生成随机张量的初始化器。\n",
    "keras.initializers.RandomUniform(minval=-0.05, maxval=0.05, seed=None)  # 按照均匀分布生成随机张量的初始化器。\n",
    "keras.initializers.TruncatedNormal(mean=0.0, stddev=0.05, seed=None)  # 按照截尾正态分布生成随机张量的初始化器。\n",
    "keras.initializers.VarianceScaling(scale=1.0, mode='fan_in', distribution='normal', seed=None)  # 初始化器能够根据权值的尺寸调整其规模。\n",
    "keras.initializers.Orthogonal(gain=1.0, seed=None)  # 生成一个随机正交矩阵的初始化器。\n",
    "keras.initializers.Identity(gain=1.0)  # 生成单位矩阵的初始化器。\n",
    "keras.initializers.lecun_uniform(seed=None)  # LeCun 均匀初始化器。\n",
    "keras.initializers.glorot_normal(seed=None)  # keras.initializers.glorot_normal(seed=None)\n",
    "keras.initializers.glorot_uniform(seed=None)  # Glorot 均匀分布初始化器，也称为 Xavier 均匀分布初始化器。\n",
    "keras.initializers.he_normal(seed=None)  # He 正态分布初始化器。\n",
    "keras.initializers.lecun_normal(seed=None)  # LeCun 正态分布初始化器。\n",
    "keras.initializers.he_uniform(seed=None)  # He 均匀方差缩放初始化器。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验目标\n",
    "\n",
    "在本实验中，我们将通过 TensorFlow （以 Keras 作为前端），实现了分别通过：`正态分布随机值`，`Xavier/Glorot 权重初始值`，以及 `HE 权重初始值` 对 MNIST 数据集，在相同的神经网络架构中进行训练。从而，对比各个权重初始化策略对模型训练正确率以及学习效率的影响。注意：`正态分布随机值`，`Xavier/Glorot 权重初始值`，以及 `HE 权重初始值` 仅仅是 TensorFlow 所支持的多种初始化策略中的一部分，最新完整的初始化列表可以 [查阅官方文档](https://www.tensorflow.org/api_docs/python/tf/keras/initializers)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 导入库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 数据加载\n",
    "\n",
    "MNIST 是 TensorFlow 的内置数据集，直接从框架中导入即可。\n",
    "\n",
    "```keras.datasets.mnist.load_data()```\n",
    "\n",
    "但由于需要从互联网上下载，连接可能不稳定，我们直接下载到本地，使用下面的代码进行加载。\n",
    "\n",
    "与之前实验一样，训练集与测试据依然是 60000 张 / 10000 张 `28*28` 像素图片。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)   (60000,)\n",
      "(10000, 784)   (10000,)\n"
     ]
    }
   ],
   "source": [
    "base_path = os.environ.get(\"BATH_PATH\",'./data/')\n",
    "data_path = os.path.join(base_path + \"lab17/\")\n",
    "result_path = \"result/\"\n",
    "os.makedirs(result_path, exist_ok=True)\n",
    "\n",
    "path = data_path+'mnist.npz'\n",
    "\n",
    "f = np.load(path)\n",
    "x_train, y_train = f['x_train'], f['y_train']\n",
    "x_test, y_test = f['x_test'], f['y_test']\n",
    "f.close()\n",
    "\n",
    "x_train = x_train.reshape([x_train.shape[0], -1])\n",
    "x_test = x_test.reshape([x_test.shape[0], -1])\n",
    "print(x_train.shape, ' ', y_train.shape)\n",
    "print(x_test.shape, ' ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个实验中，由于我们采用全连接（Dense / 密集）层架构作为输入层，而 [当前全连接层仅应用在输入数组的最后一个轴上](https://github.com/keras-team/keras/issues/10736) 。譬如：假如你输入了一个图像的多维数组 (height, width) 或者 (height, width, channels)；那么全连接层只会采用最后一个轴，即：`width` 或者 `channels`；所以在输入神经网络前，我们需要使用 `reshape()` 函数，将图片『拉直』，就是把 28x28 的数组，变为 784 的一维数组。\n",
    "\n",
    "<img src=\"./img/1_17.png\" width=\"80%\">\n",
    "\n",
    "目前这个 issue 还在打开的状态，也看不出开发者有将它关闭的意思。大部分情况下我们都不会将全连接层作为输入层设置，这里只是为了简化实验结构。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 模型搭建\n",
    "\n",
    "我们将构建一个以 `Relu` 作为主要激活函数的神经网络，仅在最后的输出层采用 `Softmax`，同时为了简化架构，全部采用全连接（Dense / 密集）层模型。\n",
    "\n",
    "我们采用全连接层架构作为输入层，输入的一维数组尺寸是 `784` ，输出的一维数组尺寸是 `64`。因此，我们在\n",
    "\n",
    "- 输入层：**784x64（权重）+ 64（偏置）= 50240** 个参数\n",
    "- 两层隐藏层：**64x64（权重）+ 64（偏置）= 4160** 个参数\n",
    "- 输出层：**64x10（权重）+ 10（偏置）= 650** 个参数\n",
    "\n",
    "整个神经网络架构的参数总和即：**50240 + 4160 + 4160 + 650 = 59210** 个参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 64)                50240     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 59,210\n",
      "Trainable params: 59,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(64, activation='relu', input_shape=(784,)),\n",
    "    layers.Dense(64, activation='relu'),\n",
    "    layers.Dense(64, activation='relu'),\n",
    "    layers.Dense(10, activation='softmax')\n",
    "])\n",
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 模型训练\n",
    "\n",
    "在训练过程中，可以通过设置 `model.fit()` 函数的 `verbose` 参数控制是否显示训练进度，以及训练进度的显示方式：\n",
    "\n",
    "- **`verbose=0`：** 不显示进度\n",
    "- **`verbose=1`：** 显示详细进度条\n",
    "- **`verbose=2`：** 显示进度概要\n",
    "\n",
    "详细信息可 [查阅官方文档](https://keras.io/api/models/model_training_apis/#fit-method)\n",
    "\n",
    "> **预计训练时间 5 分钟，请耐心等待**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "165/165 [==============================] - 4s 15ms/step - loss: 3.1775 - accuracy: 0.7066 - val_loss: 0.7677 - val_accuracy: 0.8412\n",
      "Epoch 2/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.5592 - accuracy: 0.8692 - val_loss: 0.5137 - val_accuracy: 0.8793\n",
      "Epoch 3/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.3601 - accuracy: 0.9054 - val_loss: 0.4136 - val_accuracy: 0.9028\n",
      "Epoch 4/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.2686 - accuracy: 0.9238 - val_loss: 0.3658 - val_accuracy: 0.9125\n",
      "Epoch 5/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.2174 - accuracy: 0.9376 - val_loss: 0.3102 - val_accuracy: 0.9234\n",
      "Epoch 6/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.1808 - accuracy: 0.9468 - val_loss: 0.2842 - val_accuracy: 0.9307\n",
      "Epoch 7/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.1503 - accuracy: 0.9551 - val_loss: 0.2834 - val_accuracy: 0.9315\n",
      "Epoch 8/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.1316 - accuracy: 0.9595 - val_loss: 0.2554 - val_accuracy: 0.9376\n",
      "Epoch 9/100\n",
      "165/165 [==============================] - 3s 21ms/step - loss: 0.1134 - accuracy: 0.9647 - val_loss: 0.2544 - val_accuracy: 0.9392\n",
      "Epoch 10/100\n",
      "165/165 [==============================] - 6s 34ms/step - loss: 0.1001 - accuracy: 0.9690 - val_loss: 0.2533 - val_accuracy: 0.9419\n",
      "Epoch 11/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0927 - accuracy: 0.9705 - val_loss: 0.2506 - val_accuracy: 0.9436\n",
      "Epoch 12/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0804 - accuracy: 0.9744 - val_loss: 0.2530 - val_accuracy: 0.9399\n",
      "Epoch 13/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0757 - accuracy: 0.9759 - val_loss: 0.2429 - val_accuracy: 0.9452\n",
      "Epoch 14/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0823 - accuracy: 0.9746 - val_loss: 0.2523 - val_accuracy: 0.9443\n",
      "Epoch 15/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0666 - accuracy: 0.9791 - val_loss: 0.2429 - val_accuracy: 0.9477\n",
      "Epoch 16/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0580 - accuracy: 0.9813 - val_loss: 0.2528 - val_accuracy: 0.9464\n",
      "Epoch 17/100\n",
      "165/165 [==============================] - 3s 20ms/step - loss: 0.0578 - accuracy: 0.9814 - val_loss: 0.2729 - val_accuracy: 0.9473\n",
      "Epoch 18/100\n",
      "165/165 [==============================] - 3s 21ms/step - loss: 0.0627 - accuracy: 0.9796 - val_loss: 0.2645 - val_accuracy: 0.9467\n",
      "Epoch 19/100\n",
      "165/165 [==============================] - 3s 19ms/step - loss: 0.0507 - accuracy: 0.9837 - val_loss: 0.2655 - val_accuracy: 0.9465\n",
      "Epoch 20/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0580 - accuracy: 0.9810 - val_loss: 0.2883 - val_accuracy: 0.9462\n",
      "Epoch 21/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0523 - accuracy: 0.9828 - val_loss: 0.2832 - val_accuracy: 0.9479\n",
      "Epoch 22/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0791 - accuracy: 0.9755 - val_loss: 0.2960 - val_accuracy: 0.9464\n",
      "Epoch 23/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0521 - accuracy: 0.9831 - val_loss: 0.2598 - val_accuracy: 0.9524\n",
      "Epoch 24/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0468 - accuracy: 0.9843 - val_loss: 0.2741 - val_accuracy: 0.9504\n",
      "Epoch 25/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0561 - accuracy: 0.9830 - val_loss: 0.2718 - val_accuracy: 0.9538\n",
      "Epoch 26/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0416 - accuracy: 0.9867 - val_loss: 0.2740 - val_accuracy: 0.9537\n",
      "Epoch 27/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0454 - accuracy: 0.9860 - val_loss: 0.2942 - val_accuracy: 0.9543\n",
      "Epoch 28/100\n",
      "165/165 [==============================] - 3s 18ms/step - loss: 0.0431 - accuracy: 0.9863 - val_loss: 0.2636 - val_accuracy: 0.9552\n",
      "Epoch 29/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0405 - accuracy: 0.9872 - val_loss: 0.2657 - val_accuracy: 0.9567\n",
      "Epoch 30/100\n",
      "165/165 [==============================] - 4s 21ms/step - loss: 0.0303 - accuracy: 0.9904 - val_loss: 0.2562 - val_accuracy: 0.9546\n",
      "Epoch 31/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0256 - accuracy: 0.9915 - val_loss: 0.2953 - val_accuracy: 0.9543\n",
      "Epoch 32/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0317 - accuracy: 0.9898 - val_loss: 0.2814 - val_accuracy: 0.9557\n",
      "Epoch 33/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0374 - accuracy: 0.9880 - val_loss: 0.3149 - val_accuracy: 0.9527\n",
      "Epoch 34/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0322 - accuracy: 0.9900 - val_loss: 0.2967 - val_accuracy: 0.9530\n",
      "Epoch 35/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0343 - accuracy: 0.9887 - val_loss: 0.3043 - val_accuracy: 0.9544\n",
      "Epoch 36/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0416 - accuracy: 0.9873 - val_loss: 0.2889 - val_accuracy: 0.9561\n",
      "Epoch 37/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0314 - accuracy: 0.9898 - val_loss: 0.3134 - val_accuracy: 0.9527\n",
      "Epoch 38/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0407 - accuracy: 0.9883 - val_loss: 0.3229 - val_accuracy: 0.9531\n",
      "Epoch 39/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0350 - accuracy: 0.9890 - val_loss: 0.3268 - val_accuracy: 0.9536\n",
      "Epoch 40/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0264 - accuracy: 0.9913 - val_loss: 0.3066 - val_accuracy: 0.9569\n",
      "Epoch 41/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0211 - accuracy: 0.9929 - val_loss: 0.3072 - val_accuracy: 0.9601\n",
      "Epoch 42/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0348 - accuracy: 0.9901 - val_loss: 0.3547 - val_accuracy: 0.9552\n",
      "Epoch 43/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0327 - accuracy: 0.9901 - val_loss: 0.3162 - val_accuracy: 0.9577\n",
      "Epoch 44/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0245 - accuracy: 0.9924 - val_loss: 0.3164 - val_accuracy: 0.9569\n",
      "Epoch 45/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0504 - accuracy: 0.9867 - val_loss: 0.3235 - val_accuracy: 0.9577\n",
      "Epoch 46/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0274 - accuracy: 0.9923 - val_loss: 0.3230 - val_accuracy: 0.9573\n",
      "Epoch 47/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0332 - accuracy: 0.9900 - val_loss: 0.3014 - val_accuracy: 0.9586\n",
      "Epoch 48/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0304 - accuracy: 0.9912 - val_loss: 0.3438 - val_accuracy: 0.9594\n",
      "Epoch 49/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0137 - accuracy: 0.9961 - val_loss: 0.3144 - val_accuracy: 0.9603\n",
      "Epoch 50/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0136 - accuracy: 0.9954 - val_loss: 0.3482 - val_accuracy: 0.9603\n",
      "Epoch 51/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0144 - accuracy: 0.9955 - val_loss: 0.3044 - val_accuracy: 0.9623\n",
      "Epoch 52/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0152 - accuracy: 0.9952 - val_loss: 0.3293 - val_accuracy: 0.9633\n",
      "Epoch 53/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0290 - accuracy: 0.9919 - val_loss: 0.3471 - val_accuracy: 0.9590\n",
      "Epoch 54/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0307 - accuracy: 0.9914 - val_loss: 0.3604 - val_accuracy: 0.9557\n",
      "Epoch 55/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0305 - accuracy: 0.9917 - val_loss: 0.3482 - val_accuracy: 0.9578\n",
      "Epoch 56/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0236 - accuracy: 0.9926 - val_loss: 0.3432 - val_accuracy: 0.9582\n",
      "Epoch 57/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0207 - accuracy: 0.9938 - val_loss: 0.3492 - val_accuracy: 0.9590\n",
      "Epoch 58/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0164 - accuracy: 0.9950 - val_loss: 0.3503 - val_accuracy: 0.9601\n",
      "Epoch 59/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0156 - accuracy: 0.9955 - val_loss: 0.3655 - val_accuracy: 0.9579\n",
      "Epoch 60/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0297 - accuracy: 0.9918 - val_loss: 0.3486 - val_accuracy: 0.9572\n",
      "Epoch 61/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0283 - accuracy: 0.9920 - val_loss: 0.3360 - val_accuracy: 0.9598\n",
      "Epoch 62/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0168 - accuracy: 0.9947 - val_loss: 0.3372 - val_accuracy: 0.9626\n",
      "Epoch 63/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0376 - accuracy: 0.9902 - val_loss: 0.3578 - val_accuracy: 0.9593\n",
      "Epoch 64/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0171 - accuracy: 0.9946 - val_loss: 0.3410 - val_accuracy: 0.9623\n",
      "Epoch 65/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0173 - accuracy: 0.9948 - val_loss: 0.3504 - val_accuracy: 0.9584\n",
      "Epoch 66/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0148 - accuracy: 0.9958 - val_loss: 0.3556 - val_accuracy: 0.9603\n",
      "Epoch 67/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0141 - accuracy: 0.9957 - val_loss: 0.3574 - val_accuracy: 0.9618\n",
      "Epoch 68/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0093 - accuracy: 0.9972 - val_loss: 0.3802 - val_accuracy: 0.9611\n",
      "Epoch 69/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0206 - accuracy: 0.9944 - val_loss: 0.3479 - val_accuracy: 0.9624\n",
      "Epoch 70/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0155 - accuracy: 0.9957 - val_loss: 0.3623 - val_accuracy: 0.9613\n",
      "Epoch 71/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0603 - accuracy: 0.9866 - val_loss: 0.3558 - val_accuracy: 0.9591\n",
      "Epoch 72/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0181 - accuracy: 0.9947 - val_loss: 0.3607 - val_accuracy: 0.9582\n",
      "Epoch 73/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0207 - accuracy: 0.9942 - val_loss: 0.3261 - val_accuracy: 0.9619\n",
      "Epoch 74/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0078 - accuracy: 0.9975 - val_loss: 0.3195 - val_accuracy: 0.9651\n",
      "Epoch 75/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0016 - accuracy: 0.9994 - val_loss: 0.3265 - val_accuracy: 0.9654\n",
      "Epoch 76/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 0.0024 - accuracy: 0.9994 - val_loss: 0.3452 - val_accuracy: 0.9644\n",
      "Epoch 77/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0095 - accuracy: 0.9973 - val_loss: 0.3634 - val_accuracy: 0.9631\n",
      "Epoch 78/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0082 - accuracy: 0.9975 - val_loss: 0.3806 - val_accuracy: 0.9617\n",
      "Epoch 79/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0307 - accuracy: 0.9925 - val_loss: 0.3700 - val_accuracy: 0.9623\n",
      "Epoch 80/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0570 - accuracy: 0.9888 - val_loss: 0.3691 - val_accuracy: 0.9579\n",
      "Epoch 81/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0186 - accuracy: 0.9947 - val_loss: 0.3189 - val_accuracy: 0.9648\n",
      "Epoch 82/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0090 - accuracy: 0.9973 - val_loss: 0.3398 - val_accuracy: 0.9631\n",
      "Epoch 83/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0073 - accuracy: 0.9976 - val_loss: 0.3434 - val_accuracy: 0.9649\n",
      "Epoch 84/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0038 - accuracy: 0.9990 - val_loss: 0.3651 - val_accuracy: 0.9611\n",
      "Epoch 85/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0371 - accuracy: 0.9914 - val_loss: 0.3686 - val_accuracy: 0.9599\n",
      "Epoch 86/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0141 - accuracy: 0.9959 - val_loss: 0.3198 - val_accuracy: 0.9654\n",
      "Epoch 87/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0088 - accuracy: 0.9974 - val_loss: 0.3464 - val_accuracy: 0.9643\n",
      "Epoch 88/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0085 - accuracy: 0.9974 - val_loss: 0.3952 - val_accuracy: 0.9610\n",
      "Epoch 89/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0116 - accuracy: 0.9964 - val_loss: 0.3361 - val_accuracy: 0.9649\n",
      "Epoch 90/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0104 - accuracy: 0.9968 - val_loss: 0.3658 - val_accuracy: 0.9627\n",
      "Epoch 91/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0114 - accuracy: 0.9968 - val_loss: 0.3620 - val_accuracy: 0.9631\n",
      "Epoch 92/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0115 - accuracy: 0.9970 - val_loss: 0.3813 - val_accuracy: 0.9608\n",
      "Epoch 93/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0122 - accuracy: 0.9965 - val_loss: 0.4224 - val_accuracy: 0.9582\n",
      "Epoch 94/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0111 - accuracy: 0.9971 - val_loss: 0.3735 - val_accuracy: 0.9622\n",
      "Epoch 95/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0247 - accuracy: 0.9934 - val_loss: 0.3896 - val_accuracy: 0.9602\n",
      "Epoch 96/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0210 - accuracy: 0.9944 - val_loss: 0.3734 - val_accuracy: 0.9623\n",
      "Epoch 97/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0184 - accuracy: 0.9954 - val_loss: 0.3680 - val_accuracy: 0.9621\n",
      "Epoch 98/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0345 - accuracy: 0.9921 - val_loss: 0.3813 - val_accuracy: 0.9593\n",
      "Epoch 99/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 0.0135 - accuracy: 0.9964 - val_loss: 0.3885 - val_accuracy: 0.9606\n",
      "Epoch 100/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0080 - accuracy: 0.9976 - val_loss: 0.3320 - val_accuracy: 0.9656\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=256, epochs=100, \n",
    "                    validation_split=0.3, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 训练集和验证集准确率可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xd4VVW+//H3Nye9kkYNJfQuJYAIiooioGPBBo4z6qjcsYxlqt6Zq46j93rvz3EcZyyDim1EBCyDDooNRBSQIr33FEIaCekn55z1+2MdIAkpB0gI7Hxfz5OHc3ZdO5t8zjprr722GGNQSinVOgS1dAGUUkqdPhr6SinVimjoK6VUK6Khr5RSrYiGvlJKtSIa+kop1Ypo6CulVCuioa+UUq2Ihr5SSrUiwS1dgNqSkpJMt27dWroYSil1Vlm9enWeMSa5seXOuNDv1q0bq1atauliKKXUWUVE9gWynDbvKKVUK6Khr5RSrYiGvlJKtSJnXJt+XaqqqsjIyKCioqKli+IY4eHhpKSkEBIS0tJFUUqdRo2GvojMBK4AcowxA+uYL8BfgclAGXCrMWaNf94twB/8iz5hjHnjZAqZkZFBTEwM3bp1w+5OnQpjDPn5+WRkZJCamtrSxVFKnUaBNO+8DkxsYP4koJf/ZzrwIoCIJACPAqOAkcCjIhJ/MoWsqKggMTFRA7+JiAiJiYn6zUmpVqjR0DfGLAEKGljkKuBNYy0H2ohIB+Ay4HNjTIEx5hDwOQ1/eDRIA79p6e9TqdapKdr0OwHp1d5n+KfVN10p5WA+n+HrHbms3V+IK0hwBQltY8K4fHAHIkOb5zJizuEK4iJDCAt2Ncv2neSMuJArItOxTUN06dKlhUtTt8LCQmbNmsXdd999QutNnjyZWbNm0aZNm3qXeeSRR7jgggu45JJLTrWYSrWY4ooq3l2ZzlvL97Evv+y4+U8u2MKPR3Xh+uGdCRKh1O0hMtRF18SogPdRUeUlt7iSglI3ucWVrNiTz+JtuezIKWHSwPa8ePPwRtc9IjYihLiIM6MjgzGGbQeLKSqrYlT3xGbdV1OEfibQudr7FP+0TODCWtMX17UBY8wMYAZAWlraGfmk9sLCQl544YXjQt/j8RAcXP+vccGCBY1u+/HHHz/l8qmmVeX1seNgCR3bhNMmMrSli9Mkqrw+jAFXkFBe5WXR1hw+3ZjN8t35XNKvHb+b1JeEqJM71u/3FPDA7B/IKqogrWs8v57Qh8sGtMcVJHh8PjZkFPHyN7t5YfEunl+0q8a6/3ftYG4Y0bmeLVvlbi8zluzmpa93UV7lPTo91BXEyNQEerWLZsGGbBZty+GiPm2Pzi+t9PDFloMs3JTN4m25lLm9NbbbIzmKYV3i6ZYURUGpm/ySSorKq47O9xqocHspq/Lg8RoSokJpFxtOXEQI2UUV7Cso41Cpm5+P684t5514R5Ovt+cyb3UGy3blkVfipm/7GD594IIT2saJaorQnw/cKyKzsRdti4wxB0RkIfDf1S7eTgAeboL9tYiHHnqIXbt2MWTIEEJCQggPDyc+Pp6tW7eyfft2rr76atLT06moqOD+++9n+vTpwLFhJUpKSpg0aRJjx47lu+++o1OnTvzrX/8iIiKCW2+9lSuuuILrrruObt26ccstt/DRRx9RVVXF3Llz6du3L7m5udx0001kZWUxevRoPv/8c1avXk1SUlIL/2acp9zt5ZbXvuf7PfZSVnxkCB3iIvAZczQ4o8KCiQkPpm1MGPeN70X35OiT2pcxhi0HilmyI5c9uaXklVSSV+pmSEocD0/uR3hI3c0Vh0rd5BRX0qd9TI3pHq+PtemFFFd4qKjyUlDmZl16IT/sL2RnbgmmVpUqKTqM4V3jeW9NBp9tzuY3l/Ul2CUs3pbDtzvzmX5Bd+65qGeNdfbklZJxqIwOcRG0iw3j1aV7eO7LHXROiOS9u0YzvGtCjeVdQS7SuiWQ1i2BvXmlfLsrj/BgF1FhLv65fD8Pf7CBpJhQLu7brs5j/ff6Azz5781kFVUwaWB7LurbloTIUBKiQ+nTLoaosGAqPV62Zn/DY/M3MfqBRMJDXGQVlnPTy8vZm19GckwY1wztxDkpbTiSyznFlazZd4gvthzkUFkVkaEukqLDiI0IJsi/kIgQERJE25hwXEFCfkklK/cWUFhWRdvYMLomRBITFsxjH21m84HD/OnqgUebmHIOV7Auo4gNGYVsyS5mTI9Efjq6G0FBdtuzv7fHnhgVxtieSZzXM4kxPZv/7zmQLpvvYGvsSSKSge2REwJgjHkJWIDtrrkT22XzNv+8AhH5E7DSv6nHjTENXRAOyB8/2sTmrMOnupka+neM5dEfDWhwmaeeeoqNGzeydu1aFi9ezOWXX87GjRuPdnmcOXMmCQkJlJeXM2LECK699loSE2t+TduxYwfvvPMOL7/8MjfccAPvvfceN99883H7SkpKYs2aNbzwwgs8/fTTvPLKK/zxj3/k4osv5uGHH+bTTz/l1VdfbbpfwFnI7fERGtz09xa6PT7uens1K/cW8NuJfQgOEvbklXHwcAXBQUKIKwjE1iCLKzx8uTWHzzcf5IlrBnLN0JRGt2+MIb2gnOV78lmxu4BvduSS429ySI4Js6ETHswby/axNr2Ql34ynA5xEUfX9/oMs77fz9MLt1FcUcUzNwzh6qH2UllFlZd73l7Dl1tzauwzPjKEoV3imTSwPWEhLrw+gwCjuicyvGs8riBhW3Yxf/hwA//5wQYA2sWGkRgdyguLdnLTyC7E+78BlFR6uP6lZeSVVNbYx5ShnXj86oFEhzUcKd2SouiWdKw5Z2yvZKbOWMbdb6/hnTvPZWiXmh383ludwa/mrqN/h1ieuXEI59bT9BEW7OJPVw3kx6+s4KWvd3Hd8BSmvbycwtIqXr9tBBf0Sj4atnWdk4oqHxGhJ3c9wOczPPvFdp77aifbsotJSYhk7f5CMgvLAQgS6BAXweebD/Ll1hz+fP05zF+XxRP/3sKFfZJ58cfDT3rfJ6PR0DfGTGtkvgHuqWfeTGDmyRXtzDZy5Mgafdyfe+45PvjgAwDS09PZsWPHcaGfmprKkCFDABg+fDh79+6tc9tTpkw5usz7778PwNKlS49uf+LEicTHn1Tv1yZ1qNRNRKir3tpoUyuuqOKTjdl8sCaT5XvyubB3Mo9fNZDOCZHHLbstu5hXvtnN6B6JTBlWM4z/uXwfmYXl3DamG21jwo9O9/oMD767lsXbcvnvawZx06jGry9lFZZz/+wfePDddXyzPY9L+7ejbWw4SdGhVHl9HK7wUFRWxeYDh1mfUci69CKyD9uusglRoYzukciFvZMZ1zuZtrHHyvLZpmx+OWcdP/rbUu4b3wsRobLKy4drM9mYeZjzeiTi8RkenLMWj88weVB77nxzFd/uzOfhSX0ZmZpAeIiLmPBgOrWJaLTZoU/7GOb8x2iW7swjPjKUAR1j2X6whMueXcKrS/fw68v6APDS4l3klVTy5+vPwRUkZBaW07NtNJcNaN/o76ou0WHBvHbrSK598Tt+9vpKXrttJEM62+tfW7MP8/sPNzAqNYG37xhFsKvhD/kxPZO4YnAHXli8i7mrMiiuqOKfd4zinM71X08Df23+FEI3KEj45YQ+9OsQy0PvbyCvxM3QLm342dhUhnSOo3+HOMJDgpj1/X7+9PFmLnp6MaVuL5MHtefZG4c2S+WlIWfEhdwT0ViN/HSJijpWW1m8eDFffPEFy5YtIzIykgsvvLDOPvBhYWFHX7tcLsrLy+vc9pHlXC4XHo+niUt+ar7ccpAvthxkxZ4CdueWEhXqYvKgDlwzrBPnpiYeV5vyeH1Uec0p/VFlF1UwY8luZn2/j4oqH90SI5k2sgsf/pDJpX/5mvvH9+a8Hol4jaG00sOsFfv5ZGM2AO//kEmHuAhG97AfwAs2HOAPH24EYObSPfx4VFdGdItnxZ4CluzIZXduKb+f3C+gwAfo2CaCd+48l+e+3MHfF+3k/R8y6102NSmKUd0TSOsaz6juifRqG11vGE8Y0J4P74li+lureeRfm45Obx8bzt+mDeWKwR2oqPJx55ur+M28dbyweCd780r58/XncO3wxr9x1EVEOL/XsZF5+7SP4fJBHXj9u73ccX4qZW4vL3+zmyvP6XjS+6hLckwYb/5sJDe/uoIb/rGM/7lmEJcOaMdd/1xDbHgIf7tpaKOBf8QfLu/Poq05lFR6ePuOcxmUEtdk5WzMpEEdmDiwfb3n9MejujIqNZGH319Pn/YxPPajAQEfV1M660K/pcTExFBcXFznvKKiIuLj44mMjGTr1q0sX768yfc/ZswY5syZw+9+9zs+++wzDh061OT7aEhReRX/9eFG5q/LIiY8mBHdErh2WAr78ktZsCGbuasz6NQmghtHdOb6tBSCRJi1Yj/vfL+fglI3w7rEc0HvJMb1bsvATrFH/zD255fx1Kdb+HpbLuf5a2pjeyaRfbiCnTklLN+dz3urM/Eaw9VDOnHzuV0Y0rkNIsI9F/Xksfmb+N9Pt9Yoa0x4MPeN78X1w1O49bXvuWfWGj76xViKyqr41Zx1DO3ShqemDGbGkt28sWwvM7/dQ0SIixGpCdx9YU+uO8FAC3YF8csJffjZ2FQyC8vJOVxJXkkl4SEuosODiQ0PpmdyDHGRJ9ZTpGfbGBY+cAHZRRWEh7gICwkiKjQYl/+DNSLUxSu3pPHzf65m6Y48/jZtGJcP7nBC+2jML8b35N8bDjBz6R4yDpVjgN9O7NOk+wDb7DP/3rHc8/YafjV3HV2/iiTjUDnv3HlujW9jjWkfF868u84jOiy4zm+Aza2xb1Q920Yz9+fnnabS1E1DP0CJiYmMGTOGgQMHEhERQbt2xy46TZw4kZdeeol+/frRp08fzj333Cbf/6OPPsq0adN46623GD16NO3btycmJqbxFU9RudvLst15/OGDjeQUV/LrCb35+bgeNWoof7xyIJ9tzmbuqgye+Xw7z36xnSARPD7DuN7J9O0Qw7c783j6s+08/dl2OsaFc2n/doS4gnhz2T5cQcKl/duxYk8+n28+WGP/oa4grh2ewt0X9jjuj7hTmwhe/mkaa/Yf4lCpG5e/zX1QShyx4TZgZ/w0jav//i13vrGKovIq4iJC+MfNw2kbG86fbziHBy7pxcHDFQxKiTvlPt5tIkNpExnKgI6ntJkaQlxBDYZXeIiLV28ZQUGpm+SYsHqXO1l928cyaWB7Xv5mD+VVXn4+rgcp8c0TpglRobx5+0ie/PcWXv9uL3+4vB8jUxMaX7GWfh1im6F0ziGm9uX8FpaWlmZqP0Rly5Yt9OvXr4VKdGaorKzE5XIRHBzMsmXLuOuuu1i7du0pbXPN+o38+rM84iJD6JkcTbekKNweH4fK3OSVVLL9YAm7c0vwGds08eyNQxptH92fX8a81elU+QxTR3Su0Qc7r6SSxdtyWbgpmyXbc3F7fVw7LIVfT+hD+7hwfD7Dmv2HWL3vECnxkfRsG023pMhTDuOvth7k9jdWEeIKYu5/jG70GFRNWw4cZtJfvyEhKpTFv7nw6Adqc8oprjihGr4CEVltjElrdDkN/bPDjh07uOGGG/D5fISGhvLCCy8wYsSIgNat8vpwe3xEhLgIChKMMeQUV7J+42b+/H0J8ZGh7MwtOXrjSpvIEBIiQ+meHE3/jrH07xDLuN7JTdrDoMzt4XC5h/Zxp+cP+5MNB4iLCOG809Alzole+WY3PZKjuahv28YXVi0i0NDX5p2zgM9nCEvoxAefL6VDXDhh1XrLVHl9dn6tHjRVXh+5xZWUVNr+2gBBIkSFBSPA4QrbL/nDe8Yc7X1TWukhLDjotFxcigwNbrZb8usyaVDTtnW3Nnec372li6CaiIb+GaDc7bU3/WD7DEeEuI6GuMfrY29+GWVuDy4RtueUkBwdRlSYi4JSN4fLPYhA73YxR7t+GWPIOFROSaWHqFAX7ePCCQt2+fuWV+H2GNrHhRNSHFqju2VUI32slVJnP/0rbyHG370wx18bry0qNJg2kSHklbhxe310TYwkMjSY7KIKcooroBiCg4TE6FAKSt1kFZbTNTESEeFwhQ33DnERNS7u2XFG7J2lQSLkn8bjVUqdGTT0W4DH62NfQRmllR6CXUF0iIsgKsyF/8ZviiurOFRaRWZhOa4goXtS1NFaeOeESBKiQvF4fcREhBAkQohLOFBUweGKKqLDQjhQWE54iIuk6LrHUQnSYZWVarU09E8zj9fH7rxS3B4fHdtEkBAZetwNTRGhLpKjwyhzewlxCaG1eq/UboZJjA7jUFkVWYUVxEZ4cXt99Eio/8YfpVTrpQ9GbybR0dEUlVexZssurrx6ClVeH1XVAr9rYiTXXXEZa9asrnN98V90feHvf6Os7NgwtZMnT6awsLDGskEipMRHUOX1kV9SSXxkqLbPK6XqpKHfxNweL/vyS/EZ2JdfSkhMEk/8fSZbDhxme3Yxbo8dRiAmwL7Ozz77bI3QX7BgQZ1j80eGBpMcE0awK+i0dYNUSp19NPQD9NBDD/H8888fff/YY4/xxBNPMH78eIYNG8agQYOYM+99duaUUlJhe9R0T4oiqqqAaZeNpX1cOMGmiscenM6IoYO55ppraoy9c9ddd5GWlsaAAQN49NFHATuIW1ZWFhdddBEXXXQRYIdqzsvLA+CZZ55h4MCBDBw4kGeffZYOcRGEV+QzeOAA7rzzTgYMGMCECRPqHeNHKdX6nH1tAJ88BNkbmnab7QfBpKcaXOTGG2/kgQce4J577ICic+bMYeHChdx3333Exsaya38WF10wloXLfqB7cjQCRIeHkCeCCLSNCeefL79AbHQUW7ZsYf369QwbNuzo9p988kkSEhLwer2MHz+e9evXc9999/HMM8+waNGi48bNX716Na+99horVqzAGMOoUaMYN24c8fHxAQ/hrJRqfbSmH6ChQ4eSk5NDVlYW69atIz4+nsTktvzyN7+jb/+BXD7pMnIOHiDGV1rvUMNLliw5Gr6DBw9m8ODBR+fNmTOHYcOGMXToUDZt2sTmzZsbLM/SpUu55ppriIqKIjo6milTpvDNN98AgQ/hrJRqfc6+mn4jNfLmdP311zNv3jyys7O5/KopPPPCq6RnHWTewiUkx0UyanBfPFXuE97unj17ePrpp1m5ciXx8fHceuutdQ7NHKhAh3BWSrU+WtM/ATfeeCOzZ8/m3TlzGTX+cqoqSujepSMDUuLZvHoZ+/bta3D9Cy64gFmzZgGwceNG1q9fD8Dhw4eJiooiLi6OgwcP8sknnxxdp74hnc8//3w+/PBDysrKKC0t5YMPPuD8889vwqNVSjnR2VfTb0H9+/fnUOFhEtq2p0/3Lgy+8zauvPJKBg8eTFpaGn379m1w/bvuuovbbruNfv360a9fP4YPHw7AOeecw9ChQ+nbty+dO3dmzJgxR9eZPn06EydOpGPHjixatOjo9GHDhnHrrbcycuRIAO644w6GDh2qTTmq5Rw+APu/g7ICKC+EoCDoMAQ6DoXIEx8i+ajSfIiIt9ursb8siEoGVwM94bb+G/J2QOr50P4ccNUReVXlUHIQ4rudfBmbwvIXobIELvg1NOM9NjrK5gnIKiwnr6SSpOgwOsSFn/U3P50pv9ezls8H7mIIP31PZzojGQNrZ8GnD0FlPc+vTuwFPS+B3pdB1zEQXPfd4sfZvwJenwxxKTD0JzDoeshYCatfh73fQMoImDoLousY/XP3YnhrChg74CBhsTDgapjwxLFzdjgLZt1gO4f0+xGM+53t2HG6rXsXPphuy3D9m8d/wAVAh1ZuYrnFlRwoKndM4MOZ8Xs96xzaBxvmwv7lkP69DbkJT8B5956+Mhhja7A5m+G8X0DIsQenU3zQ1rb7XdVwcJTm2eA7Ukv2+SB7Pez7FmLa22COCeC5tyU58NH9sG2BXWfCnyA2BSLa2Br0gbWQucZud8834K2E0GjoOR76TIYuo21ZDu2BqjIYdAOE+O8zqaqAl8ba7SSk2pA/ok0X6HsFrHoNopLgpnehXbVHqRbshhkXQUwHmPo2ZP0AuxfB2nfsB8i1r9jf29s3QGUxDJkG62bb85kywu6zONt+YIycDufebY+pNp8Xtn0CnUfW/OCpKoctH9vthUTafXUba8ta2/aF8M406DYGbpp77PhPkIZ+Eyoqd7Mvv4y4iBC6JEQ6IvCh5X+vZzRvlf3KL0EgLijYZb9+b/0YjA+S+0LnUTYYdiyEC/8Txv22Wb+WA1CwBz75Lez4zL5P7gfXvQpt+8P6d+GT30FFoQ3EKTMg9NhDbMjdDps/hE0fQs4mCAqGhO42QA+sg9LcmvtK6GFr1uf+3Dav1JaxCmb/GMoPwSWPwqi7Gv6gcZfCniWw/VPY9imUZB+/TPcLYeo7EBoJXzwGS/8CP/kAelwM+btgy0fQfiB0v9juK+sHG5iVxTD2QehxkT2mmRPt+btzkf3AOCL9e3jvdijKhOBw+8H34zm2dl9+CJa/BLu+suEc3c7+TrZ+bJc77xc2/I/8Tt1l8P6ddr4r1P6uht5sj/H7GVBWa0jD8Di45I8w7BZbdp8Pdn0J794MbfvBLR9B2Mk/Dc9xod+3b98WCdvSSg978mw3zO5JUceNk3O2MsawdevWszP0t3wMX/3J/pGfM/XUt+ephNytkL3R1nYzV9uv+55aPajC20DabTDiDltbBPB6YP4vYN0sOO8+uPTxwILf44a87ZC+AvZ9Z/+N6wxjH4BeE47fhrsMvv0rfPusDeuLfg+JPeFf90BFEXQaBvuXQcpIG5Bf/y90OAdu/CdkroIV/7DzATqfa5tZKottGQ7ttaHTYzx0H2c/yPZ9C7sW2VAKjYGRd8I50yCxBwS5bI35o/vtt4Fp79SsZQfC57PfArJ+gNiOtj09YyXMv8/WiC98CN64EobcBFf9veFtHc6C9+6wZQb7IQ32w6L7uOOXryiy9/sc2ms/MGMbeb7lgfWw6L9h+yf2m8P4R2xT1eyb7Affxb+337DWvm2/rQD0ngij74Gk3v5rBjn2/+zeb+w5SuoFOz6H0hzb9PWzT+v+FnACHBX6e/bsISYmhsTExNMa/CUVHvbmlxLiCqJHclSLPLm+ORhjyM/Pp7i4mNTU1MZXOFOUF9p243XvQEgUeMrhhjdtOyjYZo/9y/w18X4QlWinFWfbcPNW2RprZDwUZdg2392LbS3X5x/eOiTKhmWnYTZUwX7FD4uDvpfbGmhtPh988htY+Yr9Y7/q+WN/wO5S2PwvGzClufaPP3+n/Tmyz+j20OVc2wxStB/aDbJh166/PY49S+CLR+FwJgyYApc9eSyoSnJt8O9ZYsNo1H/YUN72Ccy73R9CBtp0tR9Wg65rPOSqy94I3zxtvx1gIDjC1pxzNkPqBXD9G6d2kba29XNt27Yx9gPl7uV1N6vUpfigDdW9S+0Hx6Drmq5cYJv0Fv6nrRS4Qu23wCkvQ/8r7fzyQ/YbTKfhkNz7+PWNsU1In/3eNgv1HA+9LoM+kyD81J/r66jQr6qqIiMj45T6rp+oiiov+aVugoOEpOgwXA6p4R8RHh5OSkoKISEN9HzwVMLur+3X3faD7NfXQC/ANaay2P4EGkB7v4X3p0PxATj/V7YW9fb1trZ40xwbPAt/X7PdN6qtra3Xd3FRXLb9tut59vjaD7JNA0En8VhIY2xt+vP/sh8sl//ZBub3M6C8ABCITLQfBgk9oG1fG+idhtl9itgPpQ3zbJNG3raa2+9wDkz8X+g6uu59eyqPbws+sB5WvGQ/FHtNOLnjOqJgtw29g5vsT8ehcNF/Ntxz5mRt+gD+/Su4+iXoPaHpt38qfD7Y9D788JZt0usy6iS24b+wfCrnow6OCv3TbdHWHKa/tYre7WJ46/ZRJEQ1UdCd6dylNiiy1ti2z51f2t4pQcG2VhrTEUbfDd0vsjWTsBgbbNs+se3asR1h8p/rruWAbdLY9aW9ELp1gQ3kcb+1PSbq+wPwemDJ/4Ml/wfxqbZmlWK7ulJ+CF673NaavW4b/OMegsTukLMVcrfYmmlyH/t1OjjCrlNeYIO565gmqWHVkL3B1rCPhHafybbZp/PIwP/IjbHfCHK3QM4W2y1xwJST6tFx1jKm+a+POEyThr6ITAT+CriAV4wxT9Wa3xWYCSQDBcDNxpgM/zwvcGSwnP3GmCsb2ldLh36528uFTy8iISqM2dPP9T9tykGyfoBvn7MhKUGAsf2rC/fb9sUjYlOg58XQ90f2a/zepbY9uXpN+ghXmP06nbnaNieM+x2Mvte+riiyzSdbP7a9FCoPQ0QCDLgG3CX24mO3822Yh0TYi2/F2bad9nCmvWCZvsK2J0/+f8df6Co+aC/MdRxqvwEE2hTQnNxltrbaabit0St1GjRZ6IuIC9gOXApkACuBacaYzdWWmQt8bIx5Q0QuBm4zxvzEP6/EGBMdaMFbOvSfX7ST/7dwG3N/PpoR3ZqwrfJ027vUNjcMvtG2GSKw/AXbIyI81l6QOtKmHNPeXkRs09X2jOg4DGLa1b3d7I22J0tFEVQctj0/elwMYdG2drrgN7aHSG2RibYc/a60yx9pFlg7y36Vryo7fh2w7d2X/rFpLtgq5WCBhn4gd+SOBHYaY3b7NzwbuAqoPiJYf+CX/teLgDr+6s98+SWVvLh4FxP6tzu7A78oA979iW3K2DIf4rpAm862d0PfK+DKv538xbf2A+1PXaLbwg1v2F4JmWvsh0t4nO2ZkTKy7rshh9wEndJsjT+ijQ35mHYQ28k2F1Xvg66UOmWBhH4nIL3a+wyg9tWLdcAUbBPQNUCMiCQaY/KBcBFZBXiAp4wxZ+wHwt++2kl5lZffTjyLv5J73DD3VntR8O7lttfK9zNsE8vkp20PjuZuK+11qf0JVHJvGP9fzVcepdRRTTX2zq+Bv4vIrcASIBPwX6KmqzEmU0S6A1+JyAZjzK7qK4vIdGA6QJcuXZqoSCdmb14p/1y+j6kjOtOzbcCtUSfHXQarZtqLiT3HH7vzsTQPstbaGnFiT3vhtKGLdzlbYf1s267dewLUd+NVAAAWiUlEQVT0vBS+esL2d77+ddue3LbvsS5lSqlWL5DQzwQ6V3uf4p92lDEmC1vTR0SigWuNMYX+eZn+f3eLyGJgKLCr1vozgBlg2/RP5kBO1dOfbSM0OIj7L+nVvDs6uBnm/cz2zDiibX870FLR/prLBkfYnhsh4fbuwbAYO35IeJy9mejAWtvtMCzG3hzkCrO3uY+6y14oVUqpWgIJ/ZVALxFJxYb9VOCm6guISBJQYIzxAQ9je/IgIvFAmTGm0r/MGOD/mrD8TWJbdjH/3nCAuy/sQduYJny+7IF18OE9NqQ7nGPbuJf+xQb3ze/Z27x3fgl7vobkOHvXY8eh9uaigl32tvOyAnuRs6rcdqksyrC30EcmwcSnYOB19htD+nJ7p6q72N4VqpRSdWg09I0xHhG5F1iI7bI50xizSUQeB1YZY+YDFwL/IyIG27xzj3/1fsA/RMSHHbv/qeq9fs4Uz321g6jQYO4Y2/3EV87baZtYcrbAhQ8fu8hZlAmzbrT9jUPCbXOOp9z2XLnmH8cGZ2o/yN56X1tdt483pNtY+6OUUg0IqE3fGLMAWFBr2iPVXs8D5tWx3ndAC4xTGrjtB4tZ4K/lxwd6E5YxdtCob/5s288lyI4cuOsruPpFOy7HOzfaJpvbF9pxSbweezdpbKfWdZONUuqM0uofovLclzuIDHEFXsvP22HHf9n5hb2d/tI/2eEJRGw3ybm32AGUCnbZ4QGODETlCrbdJpVSqgW16tDfcdC25d81rpFavs9rx+Je964ddyMkEi77H9sGX33skVs/tjcnrXnDdo88kW6LSil1GrTq0P/7op22ln9+A7X8zfNtkJdk2wuyaT+DC34L0cnHLxscBlc+Bxf/V93zlVKqhbXa0C+t9PDpxmxuHNG5/gHVcrbYkR2TesHk/7PDoAbyVBsNfKXUGarVhv6ibTlUenxcPqhD3Qu4S2HOLXZMmR/PDezRcUopdYZrtaH/yYZskqLDSKtvjJ1//9oOYfDTDzXwlVKO0SpDv9zt5autOVw3POXYw1E8bnuHa85mSF9p73Ad95B9ZqdSSjlEqwz9xdtyKK/yMmmQvwbv88E/pxwbKz4kCgZPtQ/4UEopB2mVob9gYzaJUaGMPNK0s+YNG/jjH4GB19qhiPUGKqWUA7W60K+o8vLVloNcOaSTfdB58UH70Olu58PYX+oj2pRSjtbqqrNfb8+l1O091mtn4cN2MLMrntXAV0o5XqsL/U82HCA+MoRR3RNgxxew8T04/9eQ1LOli6aUUs2uVYW+z2dYtC2X8f3aERIk8Nnv7Tg5dY1yqZRSDtSqQn9rdjFF5VWc1yPRXrjN3Qrn/9IOn6CUUq1Aqwr9FXvyARjVPRFWvmIfPqJPmFJKtSKtKvSX786nc0IEnYIK7VOmhv4EQiJaulhKKXXatJrQ9/kM3+8pYFRqIqx+3T6SMO1nLV0spZQ6rVpN6O/IKeFQWRWju8Xa0O91KSSktnSxlFLqtGo1ob98t23PH+f73o6NP+KOFi6RUkqdfq0m9FfsyadTmwiSNr8JbbrY59gqpVQr0ypC3xjDit0FXN/hIOxbCiOnQ5CrpYullFKnXasI/Z05JeSXurmubC6Et4Hht7Z0kZRSqkW0itBfvqeAHpJJysEvbS0/LKali6SUUi2iVYyyuXx3Pg9GfIIJikBG/UdLF0cppVpMq6jp79+znYm+Jciwn0JUUksXRymlWkxAoS8iE0Vkm4jsFJGH6pjfVUS+FJH1IrJYRFKqzbtFRHb4f25pysIHoszt4aryDwnCwHn3nu7dK6XUGaXR0BcRF/A8MAnoD0wTkf61FnsaeNMYMxh4HPgf/7oJwKPAKGAk8KiIxDdd8RuXWVDKDa6vyUqZaLtqKqVUKxZITX8ksNMYs9sY4wZmA1fVWqY/8JX/9aJq8y8DPjfGFBhjDgGfAxNPvdiBy0vfSqyU4e027nTuVimlzkiBhH4nIL3a+wz/tOrWAVP8r68BYkQkMcB1EZHpIrJKRFbl5uYGWvaAVKX/AEBM6vAm3a5SSp2NmupC7q+BcSLyAzAOyAS8ga5sjJlhjEkzxqQlJyc3UZGskNwNuE0w8V0HNel2lVLqbBRI6GcCnau9T/FPO8oYk2WMmWKMGQr83j+tMJB1m1ubwi3scXVF9EEpSikVUOivBHqJSKqIhAJTgfnVFxCRJBE5sq2HgZn+1wuBCSIS77+AO8E/7fQwhk4V2zkQ0fu07VIppc5kjYa+McYD3IsN6y3AHGPMJhF5XESu9C92IbBNRLYD7YAn/esWAH/CfnCsBB73Tzs9ijKINcUUtand2UgppVqngO7INcYsABbUmvZItdfzgHn1rDuTYzX/06o8fQ0RgKfd4JbYvVJKnXEcfUdu6d7VeI0QkaIXcZVSChwe+iZrHTtNJzomJ7Z0UZRS6ozg6NCPLNjERtONlHh9+LlSSoGTQ7/4IFGVuWyT7iRGhbZ0aZRS6ozg3NDPXg9AXnRfRKSFC6OUUmcG54b+gbUAVCRqd02llDrCwaG/jn20JzGxaYd1UEqps5ljQ9+XtZ4NXr2Iq5RS1Tkz9N2lBBXtY6uvMynxkS1dGqWUOmM4M/SLMgBIN8l0TtCavlJKHeHQ0LdD+GeaJK3pK6VUNc4M/UIb+odC2hEfGdLChVFKqTOHM0O/KB0vQYS06ah99JVSqhqHhn4GeUFJdEyIaemSKKXUGcWZoV+YToYvkc7aXVMppWpwZOibonT2eRNJitZHJCqlVHXOC32vBw5nkWUSCQtx3uEppdSpcF4qlmQjxkumSSLU5bzDU0qpU+G8VCw81kc/NNjVwoVRSqkzi/NCv6h66Dvv8JRS6lQ4LxX9oZ9lEjX0lVKqFuelYmE6nrB4ygnXNn2llKrFealYlIE7uhOA9t5RSqlanJeKRelURHUEIExr+kopVUNAqSgiE0Vkm4jsFJGH6pjfRUQWicgPIrJeRCb7p3cTkXIRWev/eampD6AGY6Aog7IIG/rapq+UUjUFN7aAiLiA54FLgQxgpYjMN8ZsrrbYH4A5xpgXRaQ/sADo5p+3yxgzpGmLXY/yQ+AuoSy8A6Chr5RStQWSiiOBncaY3cYYNzAbuKrWMgaI9b+OA7KarognwN9zpzi8PaChr5RStQWSip2A9GrvM/zTqnsMuFlEMrC1/F9Um5fqb/b5WkTOP5XCNsr/xKzDYf7Q1zZ9pZSqoalScRrwujEmBZgMvCUiQcABoIsxZijwS2CWiMTWXllEpovIKhFZlZube/Kl8N+NWxRqQz8sRO/IVUqp6gIJ/Uygc7X3Kf5p1d0OzAEwxiwDwoEkY0ylMSbfP301sAvoXXsHxpgZxpg0Y0xacnLyiR/FEUXpEBxBcVAcoDV9pZSqLZBUXAn0EpFUEQkFpgLzay2zHxgPICL9sKGfKyLJ/gvBiEh3oBewu6kKf5yidIhLodJrAG3TV0qp2hrtvWOM8YjIvcBCwAXMNMZsEpHHgVXGmPnAr4CXReRB7EXdW40xRkQuAB4XkSrAB/zcGFPQbEdTlAFxKbi9PgDCNPSVUqqGRkMfwBizAHuBtvq0R6q93gyMqWO994D3TrGMgStMhz4Tqayyoa/NO0opVZNzUrGqAkpzIK4zbq+P4CAhKEgfiq6UUtU5J/QrD0OX0ZDcF7fHp007SilVh4Cad84K0W3hZ58C4N6+US/iKqVUHRyZjG6PT0NfKaXq4MhkdHs19JVSqi6OTMZKj1d77iilVB0cmYy2eUeHYFBKqdocGfqV2ntHKaXq5Mhk1Au5SilVN0cmo9urNX2llKqLI5PR7fHphVyllKqDI5OxUpt3lFKqTo5MRm3TV0qpujkyGXXsHaWUqpsjk1HvyFVKqbo5MhnthVy9OUsppWpzbuhrTV8ppY7juGT0+Yw27yilVD0cl4z6fFyllKqf45JRQ18ppernuGR0e/wPRdfQV0qp4zguGY+Gvg7DoJRSx3FcMmpNXyml6ue4ZKzU0FdKqXo5Lhm1eUcppeoXUDKKyEQR2SYiO0XkoTrmdxGRRSLyg4isF5HJ1eY97F9vm4hc1pSFr4vb6wUgLETvyFVKqdqCG1tARFzA88ClQAawUkTmG2M2V1vsD8AcY8yLItIfWAB087+eCgwAOgJfiEhvY4y3qQ/kiEqt6SulVL0CScaRwE5jzG5jjBuYDVxVaxkDxPpfxwFZ/tdXAbONMZXGmD3ATv/2mo1eyFVKqfoFkoydgPRq7zP806p7DLhZRDKwtfxfnMC6TepI6OvNWUopdbymSsZpwOvGmBRgMvCWiAS8bRGZLiKrRGRVbm7uKRVEe+8opVT9AknGTKBztfcp/mnV3Q7MATDGLAPCgaQA18UYM8MYk2aMSUtOTg689HXQ3jtKKVW/QJJxJdBLRFJFJBR7YXZ+rWX2A+MBRKQfNvRz/ctNFZEwEUkFegHfN1Xh63J07J0QDX2llKqt0d47xhiPiNwLLARcwExjzCYReRxYZYyZD/wKeFlEHsRe1L3VGGOATSIyB9gMeIB7mrPnDmhNXymlGtJo6AMYYxZgL9BWn/ZItdebgTH1rPsk8OQplPGEaO8dpZSqn+OS8Ujzjoa+Ukodz3HJWFllW4+0eUcppY7nuGSs9PoIdQUhIi1dFKWUOuM4LvTdHp/emKWUUvVwXDq6PfpQdKWUqo/j0lFDXyml6ue4dHR7NfSVUqo+jktHt8enPXeUUqoejkvHSm3eUUqpejkuHbX3jlJK1c9x6agXcpVSqn6OS8dKr4/QYH0+rlJK1cVxoa8XcpVSqn6OS0e3x6tt+kopVQ/HpaP23lFKqfo5Lh21945SStXPcemod+QqpVT9HJeOeiFXKaXq57h01H76SilVP0elo9dn8PiMhr5SStXDUemoD0VXSqmGOSodj4R+mN6Rq5RSdXJU6Fd6/Q9F15q+UkrVyVHpeLSmr713lFKqTo5KR23TV0qphgWUjiIyUUS2ichOEXmojvl/EZG1/p/tIlJYbZ632rz5TVn42txeDX2llGpIcGMLiIgLeB64FMgAVorIfGPM5iPLGGMerLb8L4Ch1TZRbowZ0nRFrl9llT/0tXlHKaXqFEg6jgR2GmN2G2PcwGzgqgaWnwa80xSFO1FHavphIRr6SilVl0DSsROQXu19hn/acUSkK5AKfFVtcriIrBKR5SJy9UmXNABH2/S1pq+UUnVqtHnnBE0F5hljvNWmdTXGZIpId+ArEdlgjNlVfSURmQ5MB+jSpctJ71wv5CqlVMMCScdMoHO19yn+aXWZSq2mHWNMpv/f3cBiarb3H1lmhjEmzRiTlpycHECR6lapoa+UUg0KJB1XAr1EJFVEQrHBflwvHBHpC8QDy6pNixeRMP/rJGAMsLn2uk3laJu+hr5SStWp0eYdY4xHRO4FFgIuYKYxZpOIPA6sMsYc+QCYCsw2xphqq/cD/iEiPuwHzFPVe/00tcoq/x25Lh2GQSml6hJQm74xZgGwoNa0R2q9f6yO9b4DBp1C+U6I9t5RSqmGOSodtfeOUko1zFHpqL13lFKqYY5KRw19pZRqmKPS0e31IQLBQdLSRVFKqTOSo0K/0v9QdBENfaWUqoujQt/t8WkffaWUaoCjErLS4yNUH5WolFL1clToa01fKaUa5qiEdHt92nNHKaUa4KiEdHu8emOWUko1wFEJadv0HXVISinVpByVkNqmr5RSDXNUQrq1pq+UUg1yVELqhVyllGqYoxLS7b8jVymlVN0clZDavKOUUg1zVEJq7x2llGqYoxKy0uMjTIdhUEqpejkq9N0er3bZVEqpBjgqIbX3jlJKNcxRCam9d5RSqmGOSUiP14fP6KMSlVKqIY5JyEp9Pq5SSjXKMQl55KHoeiFXKaXq55iEDAoSLh/cge7J0S1dFKWUOmMFFPoiMlFEtonIThF5qI75fxGRtf6f7SJSWG3eLSKyw/9zS1MWvrq4iBCev2kY43onN9culFLqrBfc2AIi4gKeBy4FMoCVIjLfGLP5yDLGmAerLf8LYKj/dQLwKJAGGGC1f91DTXoUSimlAhJITX8ksNMYs9sY4wZmA1c1sPw04B3/68uAz40xBf6g/xyYeCoFVkopdfICCf1OQHq19xn+accRka5AKvDViawrItNFZJWIrMrNzQ2k3EoppU5CU1/InQrMM8Z4T2QlY8wMY0yaMSYtOVnb5JVSqrkEEvqZQOdq71P80+oylWNNOye6rlJKqWYWSOivBHqJSKqIhGKDfX7thUSkLxAPLKs2eSEwQUTiRSQemOCfppRSqgU02nvHGOMRkXuxYe0CZhpjNonI48AqY8yRD4CpwGxjjKm2boGI/An7wQHwuDGmoGkPQSmlVKCkWkafEdLS0syqVatauhhKKXVWEZHVxpi0Rpc700JfRHKBfaewiSQgr4mKc7ZojccMrfO4W+MxQ+s87hM95q7GmEZ7wpxxoX+qRGRVIJ92TtIajxla53G3xmOG1nnczXXMjhl7RymlVOM09JVSqhVxYujPaOkCtIDWeMzQOo+7NR4ztM7jbpZjdlybvlJKqfo5saavlFKqHo4J/cbG/HcKEeksIotEZLOIbBKR+/3TE0Tkc/9zCz733wHtKCLiEpEfRORj//tUEVnhP+fv+u8YdxQRaSMi80Rkq4hsEZHRTj/XIvKg///2RhF5R0TCnXiuRWSmiOSIyMZq0+o8t2I95z/+9SIy7GT364jQrzbm/ySgPzBNRPq3bKmajQf4lTGmP3AucI//WB8CvjTG9AK+9L93mvuBLdXe/y/wF2NMT+AQcHuLlKp5/RX41BjTFzgHe/yOPdci0gm4D0gzxgzEjgIwFWee69c5fqj5+s7tJKCX/2c68OLJ7tQRoc+Jj/l/1jLGHDDGrPG/LsaGQCfs8b7hX+wN4OqWKWHzEJEU4HLgFf97AS4G5vkXceIxxwEXAK8CGGPcxphCHH6uscPDRIhIMBAJHMCB59oYswSoPSxNfef2KuBNYy0H2ohIh5PZr1NCP+Ax/51ERLphn1K2AmhnjDngn5UNtGuhYjWXZ4HfAj7/+0Sg0Bjj8b934jlPBXKB1/zNWq+ISBQOPtfGmEzgaWA/NuyLgNU4/1wfUd+5bbKMc0rotzoiEg28BzxgjDlcfZ5/0DvHdMsSkSuAHGPM6pYuy2kWDAwDXjTGDAVKqdWU48BzHY+t1aYCHYEoWunT9prr3Dol9FvVuP0iEoIN/LeNMe/7Jx888nXP/29OS5WvGYwBrhSRvdimu4uxbd1t/E0A4MxzngFkGGNW+N/Pw34IOPlcXwLsMcbkGmOqgPex59/p5/qI+s5tk2WcU0I/oDH/ncDflv0qsMUY80y1WfOBW/yvbwH+dbrL1lyMMQ8bY1KMMd2w5/YrY8yPgUXAdf7FHHXMAMaYbCBdRPr4J40HNuPgc41t1jlXRCL9/9ePHLOjz3U19Z3b+cBP/b14zgWKqjUDnRhjjCN+gMnAdmAX8PuWLk8zHudY7Fe+9cBa/89kbBv3l8AO4AsgoaXL2kzHfyHwsf91d+B7YCcwFwhr6fI1w/EOAVb5z/eH2AcVOfpcA38EtgIbgbeAMCeea+xTBg8AVdhvdbfXd24BwfZQ3AVswPZuOqn96h25SinVijileUcppVQANPSVUqoV0dBXSqlWRENfKaVaEQ19pZRqRTT0lVKqFdHQV0qpVkRDXymlWpH/D+hyUJvHm2GRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(history.history['accuracy'])\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.legend(['training', 'validation'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. 模型验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 2s 5ms/step - loss: 0.2887 - accuracy: 0.9670\n"
     ]
    }
   ],
   "source": [
    "result = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上是采用 Keras 默认初始化程序的训练结果。对于 Keras 而言，每一层都有其自己的默认值以初始化权重。对于大多数层，例如 Dense，卷积和 RNN 层，默认的内核初始化程序是 `glorot_uniform`，默认的偏置初始化器是 `zeros`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. 定义权重初始化函数\n",
    "\n",
    "定义一个通用函数，在后面我们只需要修改输入值，便能更换初始化方法对模型进行重训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weight_model(kernel):\n",
    "    model = keras.Sequential([\n",
    "        layers.Dense(64, activation='relu', kernel_initializer=kernel, input_shape=(784,)),\n",
    "        layers.Dense(64, activation='relu', kernel_initializer=kernel),\n",
    "        layers.Dense(64, activation='relu', kernel_initializer=kernel),\n",
    "        layers.Dense(10, activation='softmax')\n",
    "    ])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. 正态分布随机权重初始化\n",
    "\n",
    "将各层的默认初始化器，统一修改为正态分布随机权重初始化，重新训练网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 64)                50240     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 59,210\n",
      "Trainable params: 59,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "weight_model(\"random_normal\")\n",
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.1 模型训练\n",
    "\n",
    "> **预计训练时间 5 分钟，请耐心等待**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "165/165 [==============================] - 8s 20ms/step - loss: 0.0115 - accuracy: 0.9965 - val_loss: 0.4029 - val_accuracy: 0.9655\n",
      "Epoch 2/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0156 - accuracy: 0.9962 - val_loss: 0.4152 - val_accuracy: 0.9647\n",
      "Epoch 3/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0165 - accuracy: 0.9959 - val_loss: 0.4361 - val_accuracy: 0.9606\n",
      "Epoch 4/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0209 - accuracy: 0.9948 - val_loss: 0.4308 - val_accuracy: 0.9608\n",
      "Epoch 5/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0170 - accuracy: 0.9958 - val_loss: 0.4330 - val_accuracy: 0.9612\n",
      "Epoch 6/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0113 - accuracy: 0.9969 - val_loss: 0.4332 - val_accuracy: 0.9630\n",
      "Epoch 7/100\n",
      "165/165 [==============================] - 3s 19ms/step - loss: 0.0091 - accuracy: 0.9977 - val_loss: 0.3903 - val_accuracy: 0.9647\n",
      "Epoch 8/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0106 - accuracy: 0.9972 - val_loss: 0.3901 - val_accuracy: 0.9639\n",
      "Epoch 9/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0087 - accuracy: 0.9974 - val_loss: 0.3924 - val_accuracy: 0.9631\n",
      "Epoch 10/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0036 - accuracy: 0.9988 - val_loss: 0.4064 - val_accuracy: 0.9662\n",
      "Epoch 11/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0068 - accuracy: 0.9983 - val_loss: 0.4380 - val_accuracy: 0.9631\n",
      "Epoch 12/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0462 - accuracy: 0.9900 - val_loss: 0.4134 - val_accuracy: 0.9616\n",
      "Epoch 13/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0219 - accuracy: 0.9944 - val_loss: 0.4103 - val_accuracy: 0.9649\n",
      "Epoch 14/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0101 - accuracy: 0.9971 - val_loss: 0.3759 - val_accuracy: 0.9657\n",
      "Epoch 15/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0136 - accuracy: 0.9968 - val_loss: 0.3947 - val_accuracy: 0.9648\n",
      "Epoch 16/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0058 - accuracy: 0.9986 - val_loss: 0.3817 - val_accuracy: 0.9666\n",
      "Epoch 17/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0022 - accuracy: 0.9994 - val_loss: 0.3858 - val_accuracy: 0.9683\n",
      "Epoch 18/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0016 - accuracy: 0.9995 - val_loss: 0.3813 - val_accuracy: 0.9677\n",
      "Epoch 19/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 6.5363e-04 - accuracy: 0.9997 - val_loss: 0.3834 - val_accuracy: 0.9680\n",
      "Epoch 20/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 0.0012 - accuracy: 0.9997 - val_loss: 0.3826 - val_accuracy: 0.9673\n",
      "Epoch 21/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 0.0068 - accuracy: 0.9980 - val_loss: 0.5442 - val_accuracy: 0.9579\n",
      "Epoch 22/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0529 - accuracy: 0.9903 - val_loss: 0.4714 - val_accuracy: 0.9564\n",
      "Epoch 23/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 0.0308 - accuracy: 0.9930 - val_loss: 0.3950 - val_accuracy: 0.9621\n",
      "Epoch 24/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 0.0159 - accuracy: 0.9960 - val_loss: 0.3976 - val_accuracy: 0.9646\n",
      "Epoch 25/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0060 - accuracy: 0.9980 - val_loss: 0.3883 - val_accuracy: 0.9650\n",
      "Epoch 26/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 0.0030 - accuracy: 0.9990 - val_loss: 0.3846 - val_accuracy: 0.9663\n",
      "Epoch 27/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 0.0108 - accuracy: 0.9972 - val_loss: 0.3960 - val_accuracy: 0.9626\n",
      "Epoch 28/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 0.0099 - accuracy: 0.9974 - val_loss: 0.4460 - val_accuracy: 0.9599\n",
      "Epoch 29/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 0.0138 - accuracy: 0.9966 - val_loss: 0.4803 - val_accuracy: 0.9630\n",
      "Epoch 30/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0187 - accuracy: 0.9961 - val_loss: 0.4317 - val_accuracy: 0.9641\n",
      "Epoch 31/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0137 - accuracy: 0.9964 - val_loss: 0.3748 - val_accuracy: 0.9649\n",
      "Epoch 32/100\n",
      "165/165 [==============================] - 3s 21ms/step - loss: 0.0071 - accuracy: 0.9981 - val_loss: 0.4136 - val_accuracy: 0.9621\n",
      "Epoch 33/100\n",
      "165/165 [==============================] - 3s 18ms/step - loss: 0.0048 - accuracy: 0.9986 - val_loss: 0.4038 - val_accuracy: 0.9657\n",
      "Epoch 34/100\n",
      "165/165 [==============================] - 3s 21ms/step - loss: 0.0047 - accuracy: 0.9984 - val_loss: 0.4132 - val_accuracy: 0.9660\n",
      "Epoch 35/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0031 - accuracy: 0.9991 - val_loss: 0.4005 - val_accuracy: 0.9674\n",
      "Epoch 36/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0039 - accuracy: 0.9990 - val_loss: 0.4191 - val_accuracy: 0.9653\n",
      "Epoch 37/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0101 - accuracy: 0.9975 - val_loss: 0.4127 - val_accuracy: 0.9647\n",
      "Epoch 38/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0201 - accuracy: 0.9959 - val_loss: 0.4964 - val_accuracy: 0.9594\n",
      "Epoch 39/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0282 - accuracy: 0.9935 - val_loss: 0.4020 - val_accuracy: 0.9639\n",
      "Epoch 40/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0132 - accuracy: 0.9972 - val_loss: 0.3829 - val_accuracy: 0.9649\n",
      "Epoch 41/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0060 - accuracy: 0.9984 - val_loss: 0.3832 - val_accuracy: 0.9654\n",
      "Epoch 42/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0037 - accuracy: 0.9988 - val_loss: 0.3759 - val_accuracy: 0.9670\n",
      "Epoch 43/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0035 - accuracy: 0.9991 - val_loss: 0.3955 - val_accuracy: 0.9672\n",
      "Epoch 44/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0036 - accuracy: 0.9989 - val_loss: 0.4166 - val_accuracy: 0.9661\n",
      "Epoch 45/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0127 - accuracy: 0.9972 - val_loss: 0.4987 - val_accuracy: 0.9606\n",
      "Epoch 46/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0213 - accuracy: 0.9953 - val_loss: 0.4261 - val_accuracy: 0.9621\n",
      "Epoch 47/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0440 - accuracy: 0.9909 - val_loss: 0.3743 - val_accuracy: 0.9626\n",
      "Epoch 48/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0222 - accuracy: 0.9947 - val_loss: 0.4159 - val_accuracy: 0.9642\n",
      "Epoch 49/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0091 - accuracy: 0.9975 - val_loss: 0.3841 - val_accuracy: 0.9635\n",
      "Epoch 50/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0066 - accuracy: 0.9983 - val_loss: 0.3718 - val_accuracy: 0.9662\n",
      "Epoch 51/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0031 - accuracy: 0.9991 - val_loss: 0.3811 - val_accuracy: 0.9666\n",
      "Epoch 52/100\n",
      "165/165 [==============================] - 3s 18ms/step - loss: 9.3895e-04 - accuracy: 0.9998 - val_loss: 0.3850 - val_accuracy: 0.9670\n",
      "Epoch 53/100\n",
      "165/165 [==============================] - 3s 21ms/step - loss: 0.0020 - accuracy: 0.9997 - val_loss: 0.4135 - val_accuracy: 0.9656\n",
      "Epoch 54/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0037 - accuracy: 0.9990 - val_loss: 0.4373 - val_accuracy: 0.9641\n",
      "Epoch 55/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0099 - accuracy: 0.9978 - val_loss: 0.4491 - val_accuracy: 0.9627\n",
      "Epoch 56/100\n",
      "165/165 [==============================] - 3s 21ms/step - loss: 0.0166 - accuracy: 0.9965 - val_loss: 0.4176 - val_accuracy: 0.9636\n",
      "Epoch 57/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "165/165 [==============================] - 3s 20ms/step - loss: 0.0187 - accuracy: 0.9955 - val_loss: 0.4626 - val_accuracy: 0.9620\n",
      "Epoch 58/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0175 - accuracy: 0.9958 - val_loss: 0.4642 - val_accuracy: 0.9609\n",
      "Epoch 59/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0096 - accuracy: 0.9976 - val_loss: 0.4030 - val_accuracy: 0.9665\n",
      "Epoch 60/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0137 - accuracy: 0.9972 - val_loss: 0.4337 - val_accuracy: 0.9638\n",
      "Epoch 61/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0154 - accuracy: 0.9965 - val_loss: 0.4258 - val_accuracy: 0.9634\n",
      "Epoch 62/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0058 - accuracy: 0.9985 - val_loss: 0.4116 - val_accuracy: 0.9667\n",
      "Epoch 63/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0032 - accuracy: 0.9990 - val_loss: 0.4070 - val_accuracy: 0.9655\n",
      "Epoch 64/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0014 - accuracy: 0.9997 - val_loss: 0.4029 - val_accuracy: 0.9663\n",
      "Epoch 65/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 7.4875e-04 - accuracy: 0.9999 - val_loss: 0.3892 - val_accuracy: 0.9701\n",
      "Epoch 66/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 8.2879e-05 - accuracy: 1.0000 - val_loss: 0.3916 - val_accuracy: 0.9692\n",
      "Epoch 67/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 4.5646e-05 - accuracy: 1.0000 - val_loss: 0.3922 - val_accuracy: 0.9692\n",
      "Epoch 68/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 3.1154e-05 - accuracy: 1.0000 - val_loss: 0.3940 - val_accuracy: 0.9691\n",
      "Epoch 69/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.6262e-05 - accuracy: 1.0000 - val_loss: 0.3959 - val_accuracy: 0.9692\n",
      "Epoch 70/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 2.2873e-05 - accuracy: 1.0000 - val_loss: 0.3975 - val_accuracy: 0.9695\n",
      "Epoch 71/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 2.0264e-05 - accuracy: 1.0000 - val_loss: 0.3990 - val_accuracy: 0.9696\n",
      "Epoch 72/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 1.7999e-05 - accuracy: 1.0000 - val_loss: 0.4009 - val_accuracy: 0.9696\n",
      "Epoch 73/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 1.6023e-05 - accuracy: 1.0000 - val_loss: 0.4022 - val_accuracy: 0.9696\n",
      "Epoch 74/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 1.4481e-05 - accuracy: 1.0000 - val_loss: 0.4035 - val_accuracy: 0.9694\n",
      "Epoch 75/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 1.3114e-05 - accuracy: 1.0000 - val_loss: 0.4048 - val_accuracy: 0.9693\n",
      "Epoch 76/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 1.1942e-05 - accuracy: 1.0000 - val_loss: 0.4063 - val_accuracy: 0.9694\n",
      "Epoch 77/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 1.0865e-05 - accuracy: 1.0000 - val_loss: 0.4077 - val_accuracy: 0.9695\n",
      "Epoch 78/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 9.9670e-06 - accuracy: 1.0000 - val_loss: 0.4091 - val_accuracy: 0.9697\n",
      "Epoch 79/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 9.1202e-06 - accuracy: 1.0000 - val_loss: 0.4105 - val_accuracy: 0.9696\n",
      "Epoch 80/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 8.3677e-06 - accuracy: 1.0000 - val_loss: 0.4119 - val_accuracy: 0.9695\n",
      "Epoch 81/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 7.6879e-06 - accuracy: 1.0000 - val_loss: 0.4134 - val_accuracy: 0.9696\n",
      "Epoch 82/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 7.0784e-06 - accuracy: 1.0000 - val_loss: 0.4149 - val_accuracy: 0.9695\n",
      "Epoch 83/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 6.5081e-06 - accuracy: 1.0000 - val_loss: 0.4163 - val_accuracy: 0.9696\n",
      "Epoch 84/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 5.9931e-06 - accuracy: 1.0000 - val_loss: 0.4179 - val_accuracy: 0.9694\n",
      "Epoch 85/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 5.5325e-06 - accuracy: 1.0000 - val_loss: 0.4193 - val_accuracy: 0.9694\n",
      "Epoch 86/100\n",
      "165/165 [==============================] - 3s 18ms/step - loss: 5.0896e-06 - accuracy: 1.0000 - val_loss: 0.4209 - val_accuracy: 0.9693\n",
      "Epoch 87/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 4.6915e-06 - accuracy: 1.0000 - val_loss: 0.4224 - val_accuracy: 0.9693\n",
      "Epoch 88/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 4.3367e-06 - accuracy: 1.0000 - val_loss: 0.4239 - val_accuracy: 0.9694\n",
      "Epoch 89/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 4.0080e-06 - accuracy: 1.0000 - val_loss: 0.4256 - val_accuracy: 0.9694\n",
      "Epoch 90/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 3.6804e-06 - accuracy: 1.0000 - val_loss: 0.4270 - val_accuracy: 0.9693\n",
      "Epoch 91/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 3.3895e-06 - accuracy: 1.0000 - val_loss: 0.4285 - val_accuracy: 0.9692\n",
      "Epoch 92/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 3.1295e-06 - accuracy: 1.0000 - val_loss: 0.4301 - val_accuracy: 0.9692\n",
      "Epoch 93/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.8863e-06 - accuracy: 1.0000 - val_loss: 0.4317 - val_accuracy: 0.9691\n",
      "Epoch 94/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.6629e-06 - accuracy: 1.0000 - val_loss: 0.4332 - val_accuracy: 0.9693\n",
      "Epoch 95/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.4670e-06 - accuracy: 1.0000 - val_loss: 0.4349 - val_accuracy: 0.9692\n",
      "Epoch 96/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 2.2746e-06 - accuracy: 1.0000 - val_loss: 0.4364 - val_accuracy: 0.9692\n",
      "Epoch 97/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 2.0987e-06 - accuracy: 1.0000 - val_loss: 0.4380 - val_accuracy: 0.9692\n",
      "Epoch 98/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 1.9386e-06 - accuracy: 1.0000 - val_loss: 0.4398 - val_accuracy: 0.9692\n",
      "Epoch 99/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 1.7870e-06 - accuracy: 1.0000 - val_loss: 0.4416 - val_accuracy: 0.9692\n",
      "Epoch 100/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 1.6560e-06 - accuracy: 1.0000 - val_loss: 0.4431 - val_accuracy: 0.9692\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=256, epochs=100, \n",
    "                    validation_split=0.3, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.2 训练集和验证集准确率可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xd4HNXZ+P3v0ao3q1u2Zcly71UuYINtTDGmd9OCk4DfEBIgIQSSPIEn/EISnhBCCCU0Aw7F9BKwQ7XB4Crj3rslS7Jkyepty3n/OLvSqqy0klZW2ftzXb5WOzM7c0Zr3XPmPmWU1hohhBD+IaCrCyCEEOL0kaAvhBB+RIK+EEL4EQn6QgjhRyToCyGEH5GgL4QQfkSCvhBC+BEJ+kII4Uck6AshhB8J7OoCNJaQkKAHDRrU1cUQQogeZdOmTSe11omtbdftgv6gQYPIzMzs6mIIIUSPopQ66s12kt4RQgg/IkFfCCH8iAR9IYTwI90up98cq9VKdnY21dXVXV2UXiM0NJSUlBSCgoK6uihCiNOoRwT97OxsoqKiGDRoEEqpri5Oj6e1prCwkOzsbNLT07u6OEKI06jV9I5SaolSKl8ptcPDeqWUekIpdUAptU0pNdlt3S1Kqf3Of7e0t5DV1dXEx8dLwPcRpRTx8fFy5ySEH/Imp/8yML+F9RcCw5z/FgPPACil4oAHgenANOBBpVRsewsqAd+35PcphH9qNb2jtf5GKTWohU0uA5Zq89zFdUqpGKVUP2AO8LnWughAKfU55uLxRkcLLYTwjtXu4PDJCvbklXG4oAK7w9HVRRItSO4Txg3TUzv1GL7I6Q8AstzeZzuXeVrehFJqMeYugdTUzj3h9iouLub111/npz/9aZs+t2DBAl5//XViYmI8bvPAAw9w9tlnc+655za7XmtNtdWB1e7AEqAIDFAEWQIICJDaelcqqqjljQ3HCFCKmPAgkqNDOXt4IpYu/l725pXx9b58vt5XwMYjp6i11Qd6ucHr3iYOjOkRQb/DtNbPAc8BZGRkdMsntRcXF/P00083Cfo2m43AQM+/xuXLl7e674ceeqjJMrtDU2uzU1pto7jSSo3N3mB9WJCFoUmRkqbpIifLa7jphfXsyStrsPzajBQeuWp8l30v//ffPTy96iAAI5Oj+MGMNMYO6MOI5CiGJEYSHCi9tP2dL4L+cWCg2/sU57LjmBSP+/JVPjhel7j//vs5ePAgEydOJCgoiNDQUGJjY9mzZw/79u3j8ssvJysri+rqau666y4WL14M1E8rUV5ezoUXXsisWbNYs2YNAwYM4MMPPyQsLIxFixZx8cUXc9VVV5GaNohLrr6elZ+vwGa18ui/XmbsmNFQVc7tty4iNzeHSRnTWPXVl6xZv5FBA5K7+DfjfwrKarjh+XVknark1R9PZ0paLMVVtfx77VGeXnWQ2IhgfnPhqNNerue/OcTTqw5ybUYKvzhvOP36hJ32MojuzxdB/yPgZ0qpZZhG2xKtda5S6lPgT26Nt+cDv+nowf7wn53syint6G4aGN0/mgcvGdPiNn/5y1/YsWMHW7ZsYdWqVVx00UXs2LGjrsvjkiVLiIuLo6qqiqlTp3LVVVcRHx/fYB/79+/njTfe4Pnnn+faa6/l3Xff5aabbqpbn1tSjUNr+iYl8vWa9fz7xef4cOmzXLbkRX72h/s579x5/OY3v2H58hW88e9XKK6o7fC52+wOAi2np/aXX1bNpiOnGNUvmrT4cJ/Xhosra/lkey79+4Rx1rCETjmvgrIaFj63lpziapYsmsqZQxIACAsO494LRlBSZeXZrw8RHxHM4rOH+Pz4nryzKZuHl+/movH9+POV47s8xSS6r1aDvlLqDUyNPUEplY3pkRMEoLX+F7AcWAAcACqBHzrXFSml/h+w0bmrh1yNur3BtGnTSElNq3v/xBNP8P777wOQlZXF/v37mwT99PR0Jk6cCMCUKVM4cuRI3bryahsny2sIUIpbb15IUlQos86YzoqPPwLg22+/rdv/ggUXEhMTS2mNDavdQVA7g1u11c7oBz/ltVunM3VQXLv24a3DJyu44fl15JaYbqJ9woI4d1Rf/u/qjgeo3bmlvPzdET7YcpwaZ/46OTqUq6YM4MezBhMXEdzh8oP5fd22NJOc4mpe/uFUpg9u+P0qpXjosrEUV1n50/I9DEmMZN6ovj45tid2h+bVdUd56ONdzBqawGPXTpCAL1rkTe+d61tZr4E7PKxbAixpX9Ga11qN/HQJCgljd24pSVGh7P5+LV988QVr164lPDycOXPmNNsHPiQkpO5ni8VCVVUVYGrbhRW1RIYEYglQddtZLBZsNluzx1fKNPCeqqglKTq0zeW32R2cqrRSa3Pw6rqjPg36pdVWSiqtDIwLB+BgQTk3PL8Oq13z4i0ZFJTV8N3BQt79PpvZIxK5dEL/dh/nr//dy6vrjxISGMCVk1O4cXoq2acqeXNjFs+sOsjevDJeuGVqh89Ja819725jS1Yx/7ppSpOA72IJUDx27QT25Jby8PLdzB6e2Gl3UpuOnuKBD3ewM6eUs4Yl8K+bphASaOmUY4neo1s05PYEUVFRlJWZRjuHQ1Pr7EmTX1bNvqwTxMTEEB4ezp49e1i3bp3X+3VoTUWtnUALpDqDZHNmzpzJW2+9xX333cdnn33GqVOniAi2UFRRS2JUSJtSJVprjhdX4XBozhwSz3935FFSZaVPWMenZFi5J5973t5KUUUtqXHhnDUsgc92nUBrzRu3zWBEchQA12YMZE9uKU99dYCLx/Vrc0+kz3bm8fsPd5BfVsOiMwdx97zh9Ak35R87oA/zx/bj0U/38vSqA+QUV9E/pmP57Se/OsCHW3K494IRzB/bcjtKSKCF++aPZPG/N/FmZhY3Tk9rcfv2eGtjFr9+dxvJ0aE8ecMkLhrXTxr1hVekKd9L8fHxzJw5k7Fjx3LPvffi0Jq0uHBSYsPImHUOpZU1jBo1ivvvv58ZM2Z4vd+yKisOhyY2PLjFGuGDDz7IZ599xtixY3n77bdJTk4mNTmeWruDsurm7wY8KamyUlJlJToskPvmj6TG5uDjbTlt2kdjtTYHD3+yix++vJGkqBB+f/FohveN5L3vjxOgaBDwAQICFHfMHcreE2V8sftEm46VeaSIxf/eRFxECO//dCYPXjKmLuC7u27qQDSwbGNW0520wYrtufzt831cMWkAP53jXZ7+vNF9mToolr9/vp+KmrZ9P954ec0RxvSP5st7ZnPx+P4S8IX3tNbd6t+UKVN0Y7t27WqyrKs4HA69J7dU78sr1Q6HQ2ut9amKGr0165SuqLG2eX+HCsr1rpySun15Ul1dra1Ws/81a9boCRMmaIfDoXfllOhDBeVeH6/WZtc7jhfr/SfK9M6dO7XD4dDnP/a1vuzJb9tcdne3v5qp0+77WP/u/W26qtZWX26rTdvszZ+b1WbXZz3ylb7kn6tbPX8Xh8OhL33yWz3t4c+9+n3/4MX1evrDX2irze7diTSyK6dEj/yfFfryp75tcF7e2HS0SKfd97H+++d723VsTw4VlOu0+z7WL6w+5NP9ip4NyNRexFip6bdRabWNGpu9QUol2FlDt9nbNsSg1uagrNpKbHhwqzW1Y8eOMXXqVCZMmMCdd97J888/j1KK6NAgKttQkzxVWYvdoUmJDUMphVKKazJS2JJVzIH8stZ30IysokqWb8/j9jlD+OPl4wgNqs8rhwRaPDYsBloCuH3OELZll7B6/0mvjvWfbblszSrmnvNHEB7cenbyhump5JVWs3JvgXcn46aoopbblmYSHRbIszdNaXBe3picGsuCcck8980h8st8N8/R8u25ACwYJ911RdtJ0G9Ea01uSRWHCsqbDIgCOFlWQ7AloEH+22IxQc3maFvQP1VpulzGRbSeSx82bBibN29m69atbNy4kalTTeNkkEVh1xqHl8curbISHmxpEMAunzSAwADF25nZbSq/y5sbswhQcPOMtueur5w8gH59QnnyqwOtbltttfPIij2M6hfNVZNTvNr/vJFJJEWF8MaGY20ql9Xu4I7Xvie/rIZnb85oV2M5wL0XjKTW5uAfX+z3+jNPrzrAAx/uoKTK2uz6j7flMiUtVvrhi3aRoN9IYUUtBWU1VNTYOXCinGJnYLY5HBSW11BRayOhUcNpYID5NbZlXhPt7HkTGRJIcAd6XAS24YJTa3NQWWsnulGDbUJkCHNHJvHe5uNsOlrEM6sOcvurm9h0tPUetja7g7c3ZTF7eGK7GktDAi3cdtZgNhwpYktWcYvbvrzmCMeLq/ifi0Z53S0x0BLAdVMHsmpvPseLq7wu1ytrjrD2UCF/vmIcEwd6nkKjNekJEVw/LZVlG7M4VFDe6vYlVVYe/2I/S9ce5YK/f8OqvfkN1h8sKGd3bikXjevX7jIJ/yZB301ZtZXc4iqiQ4MYkRxJSJCFY0WV7DtRxu6cMo4XVxESGEBseMN+3wHK9NFuLfBW1tqw2c2FobzGRq3d0eE+5K4Ljs2LC46r5tgntOmdxTVTUigoq+GqZ9byyH/3sHJvPouXbiL7VGWL+1y5t4ATpTUsnNb++UKuyUghItjC0rVHWiz7U18dYO6IRGYOTWjT/l0Num+2oba/cm++uaOY4t0dRUt+Pm8oIYEB/O2zfa1u+8m2XNMofsVYokIDWfTSRh74cAd25/+t5dtcqR0J+qJ9JOg7VVvtHCusJCTIwsC4cIIDLQxOjCApKhSLUiRGhTAkMZLhfaOa1DKVMpOgtZTTr7HaOZBfzu68Mo4WVpBfVoMlQDWpdbeVqyzetCeUVlkJDbIQ0kxuet6ovjxw8WieumEyG393Lp/ceRa1NgeLl26iqrZpmsvljQ3HSIoK4ZyRSe0+h6jQIK6cnMLHW3MpLK9pdpvvj56irMbGbWcPbvP+U2LDOXtYIu9+f9yrNFitzcGmo6eYnu6bsQtJUaHcOiudT7ab9oiWvPt9NkOTIrlhWiof3zmLH89KZ+nao9z7zlbsDs0n23PJSIsluU/70k1CSNDH9JU/VlSJUopB8RF1gTRAKZL7hDIkKZLkPqFEhAR6bHANDFB1tbHmlDsbW2PCgiivsVFRYyM2PJiADna1C/IyvWO1O6iotXnsi28JUPxoVjoXje9Xd4F74vpJ7M4r5d53tmI6BzSUW1LFqr35XJOR0u5RwS43n5FGrd3Bm5nNd6/clWum3hg7oE+79n/l5AEcL65iw5HWU1bbj5dQbXUwY7DvBqzddrYZGfzIf/c0+7sEOHKygk1HT3HV5BSUUoQEWvj9xaP55XnDee/74yx6aQN78sq4aLzU8kX79Zqgr7XmaGEFpVVWj39UnhSW11BttZMSG9buWQgtAQ3TO5GRkQDk5ORw9dVXU15jI9gSQEpsGKOSo02u9/ILyczMbHG/jz/+OJWV9SmWBQsWUFxcX1u0eJneKXWldtpwZzF3ZBK/vmAkH2/L5dfvbKOsumHD4lsbs3FouC6j41PBDu8bxYzBcby27lizF89duaUMjAsjupnUlDfOH51MZEgg733femP1+sOFAD4dpRwVGsTP5g5lzcFCvt7XfE+i977PRim4fFLDEcp3zhvGvReMqOvhdOFYCfqi/XpN0K+1O6iy2jlSWMGB/HJKvAz+tTY7J0priA4N6lCqJdAS0Gzg7d+/P2+//TYVNfa6O4WAAEVUaBDe1PEbB/3ly5c3mJvfEqAIUC2nlsDkxEMCLYS08aL2k9mDuWPuEN79Ppv5j69m9f4C1h0q5KevbeKJr/Zz1rAEUuM9jyRui1vOGMTx4iq+2pPfZN3u3FJGJUe3e99hwRbmj01mxfY8qq2e01UA6w8VMSwpkvjIkBa3a6sbZ6SSnhDBb9/bTkllwwuow6F5b/NxZg1NaLZXzh1zh/KnK8Zx17xhktoRHdJrgn5IoIXhfaNIiQ3H4az17z1RxonS6gYPkXCntSan2PSfbq3nyf33389TTz1V9/5///d/+eMf/8i8efOYPHkyF8yaxmfLP27yuSNHjjB27DhsDgcBDisLFy5k1KhRXHHFFXVz7wDcfvvtZGRkMGbMGB588EHATOKWk5PD3LlzmTt3LmCmaj550tT4HnvsMcaOHcsV887g2af/WXe8UaNGcdtttzFmzBjOP/98ysorqKixEx3mOT3liVKKey8YyTu3n0lIUAA3v7iBhc+t47sDhdw6K52/XzexTftryXmj+5IcHdqkQbey1sbhkxWM6tf+oA9w5aQBlNXY+HyX5xHANruDzCNFTPdhasclJNDC49dNJL+sht99sL1BpWTDkSKyT1Vx5eRmnzMEmDEHvzhvuM/LJfxLz5t7Z8X9kLe92VUBQBwQi8bu0Fjt5rUWIDCgbhCVi93hIMHqIKn/eIIv/WuLh73uuuu4++67ueMOM7fcW2+9xaeffsqdd95JdHQ0uw9nc8Hcs7njloVYGh3H4fzjfu2l5wkPD2f37t1s27aNyZPrniHPww8/TFxcHHa7nXnz5rFt2zbuvPNOHnvsMVauXElCQsMeK5s2beKll15i/fr1HMgv45oL53L5hecRGxvbZArn1998mzMuuLxDc+tMTo1l+Z1n8eq6o0SHBnHJhP6EBft2cq9ASwA3TE/lsc/3kVVUWTdh2968MrQ2U2B3xIzB8fTvE8p732dziYdJ3nblllJRa2d6evMTqnXUhIEx/OK84fz1072cMzKJKyenUF5j46XvDhMRbOGCMTLgSnSuXlPTd6dQBAYEEBZkITzYjAi12h11wRdAo6mxOQgIgLCg1n8NkyZNIj8/n5ycHLZu3UpsbCzJycn89re/Zfz48Vx72UXk5+VyPDevyWcdWhMcGMCa776tmz9//PjxjB8/vm6bt956i8mTJzNp0iR27tzJrl27WizPt99+yxVXXEFERAR9oqI4d8ElrF69Gmg6hfOBQ4exBCjC2jiitLHQIAu3njWYa6cO9HnAd3HNuOneP313rhkpPLqDNf2AAMVlkwbwzf6TFJQ130to/SHT0OurnjvN+cnsIUwbFMcDH+7knre2Mu3hL/h05wmun5bq1ShjITqi5/0Pu/Avbdo8ALDYHBw8UUZUaCBp8REAZBdVUlxpZUhSBMrLP7RrrrmGd955h7y8PK677jpee+01CgoK2LRpE5U2GD18SIP8O5gUkkNDZAvHOHz4MI8++igbN24kNjaWRYsWNTs1syeBAQr35ovGUzhX19QSHtz21E5XGJQQQVp8OKv2FnDzGYMAk8+PCgkkJbbjI1CvnDSAZ1Yd5KOtOfx4VnqT9esPF5KeENHuEbjesAQoHrtuAgv+sZr/7sjlkvH9uXbqQCantn8QmBDe6pU1/caCAwNIigqhpMpKWbWZYfJUZS1J0SFtqlldd911LFu2jHfeeYdrrrmGkpISkpKSCAoK4rtvviYnOwtbo8bjGqsdrTURoYGcffbZvP766wDs2LGDbdu2AVBaWmpq7H36cOLECVasWFH3efcpnd2dddZZfPDBB1RWVlJbXcnnyz9m1qxZTbZzaI3NoQnvpJp5Z5g9PJE1BwvrpsHYlVvKqH7RPrloDesbxZj+0Xy6o+kdmd2h2XC4qFNr+S4pseGs/NUcNvzuXB65ejxT0mJ7xEVZ9Hx+EfQBEqJCCAkMIKe4muOnqggLspAY1bbeGWPGjKGsrIwBAwbQr18/brzxRjIzMxk3bhzLXn+V9KHDm3Q3rHAObIoMDuT222+nvLycUaNG8cADDzBlyhQAJkyYwKRJkxg5ciQ33HADM2fOrPv84sWLmT9/fl1DrsvkyZNZtGgR06ZN46JzZ3Pl9TczfkLTRlWrsxE7oocF/Sqrncwjp3A4NHtySxnVL6r1D3ppfEoM+5uZXG5PXiml1bZOacRtTnxkCBEhPe9mW/RsfvM/LkAp+vUJ40hhBUop0uMi2jUwavv2+kbkhIQE1q5dC5heH7tyS+u625WXm3lW4pNT+OTrDQQFBhAUGMayZcua3e/LL7/c7PKf//zn/PznP6977/6IxV/+8pf88pe/pLiylmNFldgcmkGDBrFjx466bX58x10UlFYT1oNyxTMGxxNsCeDrfQWkxIZRUWvvcM8dd0MSIzhVaaWoorbBNBiufP60TmrEFaI78JuaPkB0WBBJUaGkxIZ1uFGzMUuAQqEaTLqmtaaixkZESOfWsgMDPI/KrayxERrkeXrj7igiJJCp6bGs2pvPbudI3I723HE3ONG06zSeAG1LVjH9+oQyoINP2RKiO/OroA+Q3Ce0yYRpvqCUajIqt9pqx641kZ18Cx9YN59/w/EIWmsqa+2E98AUwuzhiew7Uc6Xu/MJUGbErq8MTjCjpQ8VVDRYvjevzKd3FEJ0Rz0m6Ld1aoWuEGhpODK2xplPb+vDN9rK4qGmX22149C62Xx+d/99zhlhJnD7YMtxBidG+vR3mBIbRpBFcfBkfU2/1ubgYEE5I5N9d3ERojvqEUE/NDSUwsLCbh+oGk+6Vm1zoKDd8/m05biKpjNtuhqRG/fc0VpTWFhIaGj3Hc4/LCmSfn1Csdp1h/vnNxZoCSAtPqJBTf9gQTk2h27wHF8heqMecd+fkpJCdnY2BQVtf+Td6VRUUYvV7qC2MLTufa3Nwd6yzg+uBSVVlAVZOOWWuiqqqKXG5uBgM8cPDQ0lJaXjc8V3FqUUs4cnsmxjVqekXIYkRnAgv76mvzfP9OYZ2YH5fYToCXpE0A8KCiI9velAmu7mgQ938OGWE2x98HwALvzHapKjQ3jph5M6/dh3//0b0uLDee4HE+qWzXrkK8an9OHpG0d1+vE7w5wRSSzbmMW4dk6n3JLBiZF8tScfq91BkCWAPXllBFlUXSOvEL1Vj0jv9BSx4cGUVFnNlA8OzeGT5QxOjDwtx46PDKaworbu/YnSarJPVTE5Nfa0HL8znD+6L0t/NI2ZQ33fhXJwQgRWuyaryIyg3ptXypDEyA4/F0CI7k7+h/tQfKRJrZyqrCW3tJpqq+O01RwTIkMaPHVq09FTAExJ67lBPyBAcfbwxE4Zqeq6GLvy+nvzyqQRV/iFHpHe6SlcA32KKmrJLzUBeMhprOmfLK+v6W84XERoUABj+vs+NdIbDHH11T9ZTkllHDkl1YyQfL7wA1LT96G6oF9eWzfw53TW9MtrbHUPCFl3qJCMtLhO7znUU8WEBxMXEcyhAvPcBUBq+sIvSETwofgIM5dPYUUtBwsqiAoNJNHHT1/yfOzgumOfqqhlT16ZT5/x2hsNSTTdNvfmmVG/I304v48Q3ZUEfR9yT+8ccjbinq6ZExOcF5fC8pq6h39PHyxzyLRkcEIkh06WsyevjOjQQJI7cTplIboLCfo+FBtunkxVWFHLoYIKhiScvu5/rkbkk+U1rDtUSGhQAONTJJ/fksGJEZwsr2XD4SJGJvtm6mYhujsJ+j4UaAkgJjyI7FOV5JZUMyTp9DTiQn1N/2R5LesOFTElLZaQwJ4znXJXcPXg2Z9fLiNxhd+QoO9jcRHBdd0lB3dBTf9gQTl78kqZIdMDt8q9kV3y+cJfSND3sfiIYI4WmgE/p7OmHx4cSHiwhRXb89AaZgyRoN+a1LjwummppeeO8BcS9H3M1ZgboCAtPvy0Hjs+MphjRZWSz/dSkCWA1DjzHfly6mYhujMJ+j4W5+y2mRIbftpz6q4uo5LP997wvlGkxYcTFRrU1UUR4rTwKugrpeYrpfYqpQ4ope5vZn2aUupLpdQ2pdQqpVSK27pHlFI7nP+u82XhuyNXf/khXTBxl6sxV/L53nvw0tG8eEtGVxdDiNOm1aCvlLIATwEXAqOB65VSoxtt9iiwVGs9HngI+LPzsxcBk4GJwHTgV0qpXj3WPdYZ9E/XRGvuEpyNuZLP916/PmEMTZLUjvAf3tT0pwEHtNaHtNa1wDLgskbbjAa+cv680m39aOAbrbVNa10BbAPmd7zY3Vd9Tf/0B/3BiRHERQRLPl8I4ZE3QX8AkOX2Ptu5zN1W4Ernz1cAUUqpeOfy+UqpcKVUAjAXGNixIndvA50Ng758kLe3fjQznZX3zJF8vhDCI1/Nsvkr4Eml1CLgG+A4YNdaf6aUmgqsAQqAtYC98YeVUouBxQCpqak+KlLXmJIWy5f3zO6Smn6gJYA+4dI2L4TwzJsIcZyGtfMU57I6WuscrfWVWutJwO+cy4qdrw9rrSdqrc8DFLCv8QG01s9prTO01hmJiYntPJXuoysCvhBCeMOboL8RGKaUSldKBQMLgY/cN1BKJSilXPv6DbDEudziTPOglBoPjAc+81XhhRBCtE2r6R2ttU0p9TPgU8ACLNFa71RKPQRkaq0/AuYAf1ZKaUx65w7nx4OA1c6JrEqBm7TWNt+fhhBCCG8orXVXl6GBjIwMnZmZ2dXFEEKIHkUptUlr3eqgE2n1E0IIPyJBXwgh/IgEfSGE8CMS9IUQwo9I0BdCCD8iQV8IIfyIBH0hhPAjEvSFEMKPSNAXQgg/IkFfCCH8iAR9IYTwIxL0hRDCj0jQF0IIPyJBXwgh/IgEfSGE8CMS9IUQwo9I0BdCCD8iQV8IIfyIBH0hhPAjEvSFEMKPSNAXQgg/IkFfCCH8iAR9IYTwIxL0hRDCj0jQF0IIPyJBXwgh/IgEfSGE8CMS9IUQwo9I0BdCCD8iQV8IIfyIBH0hhPAjEvSFEMKPSNAXQgg/IkFfCCH8iAR9IYTwIxL0hRDCj3gV9JVS85VSe5VSB5RS9zezPk0p9aVSaptSapVSKsVt3f8ppXYqpXYrpZ5QSilfnoAQQgjvtRr0lVIW4CngQmA0cL1SanSjzR4FlmqtxwMPAX92fvZMYCYwHhgLTAVm+6z0Qggh2sSbmv404IDW+pDWuhZYBlzWaJvRwFfOn1e6rddAKBAMhABBwImOFloIIUT7eBP0BwBZbu+zncvcbQWudP58BRCllIrXWq/FXARynf8+1Vrv7liRhRBCtJevGnJ/BcxWSm3GpG+OA3al1FBgFJCCuVCco5Q6q/GHlVKLlVKZSqnMgoICHxVJCCFEY94E/ePAQLf3Kc5ldbTWOVrrK7XWk4DfOZcVY2r967TW5Vo2Z54tAAAfj0lEQVTrcmAFcEbjA2itn9NaZ2itMxITE9t5KkIIIVrjTdDfCAxTSqUrpYKBhcBH7hsopRKUUq59/QZY4vz5GOYOIFApFYS5C5D0jhBCdJFWg77W2gb8DPgUE7Df0lrvVEo9pJS61LnZHGCvUmof0Bd42Ln8HeAgsB2T99+qtf6Pb09BCCGEt5TWuqvL0EBGRobOzMzs6mIIIUSPopTapLXOaG07GZErhBB+RIK+EEL4EQn6QgjhRyToCyGEH5GgL4QQfkSCvhBC+BEJ+kII4Uck6AshhB+RoC+EEH5Egr4QQvgRCfpCCOFHJOgLIYQfkaAvhBB+RIK+EEL4EQn6QgjhRyToCyGEH5GgL4QQfkSCvhBC+BEJ+kII4Uck6AshhB+RoC+EEH5Egr4QQvgRCfpCCOFHJOgLIYQfkaAvhBB+RIK+EEL4EQn6QgjhRyToCyGEH5GgL4QQfkSCvhBC+BEJ+kII4Uck6AshhB+RoC+EEH5Egr4QQvgRCfpCCOFHJOgLIYQfkaAvhBB+xKugr5Sar5Taq5Q6oJS6v5n1aUqpL5VS25RSq5RSKc7lc5VSW9z+VSulLvf1SQghhPBOq0FfKWUBngIuBEYD1yulRjfa7FFgqdZ6PPAQ8GcArfVKrfVErfVE4BygEvjMh+UXQgjRBt7U9KcBB7TWh7TWtcAy4LJG24wGvnL+vLKZ9QBXAyu01pXtLawQog0qi2Dln6CmrKtLIroRb4L+ACDL7X22c5m7rcCVzp+vAKKUUvGNtlkIvNHcAZRSi5VSmUqpzIKCAi+KJIRo1a4P4OtH4P2fgMPR1aXxjtZdXYJeL9BH+/kV8KRSahHwDXAcsLtWKqX6AeOAT5v7sNb6OeA5gIyMDPnWhfCFEzvN656P4Zv/gzlNmuNaVlsJpTlQeRIqTrq9FoIKgPB4iEiAPimQOBKi+oFSLe/TVgOnjsDR7+DId3ByH8QOMp9PGA6RSWafoTFQU2qOV3UKQiIhPAHC40y5Kp3lCE+ApJEQFtue35Bf8iboHwcGur1PcS6ro7XOwVnTV0pFAldprYvdNrkWeF9rbe1YcYUQXsvbAalnQGw6rPoz9B0Doy5p/XPHN8HGJbDjXbBVNV0fFAFosDbK1IZEm8CdOBISR5hAfXI/FOyBosMmSNeU1m8fmQx9R0P+LnNh0h24G4nsay4+4QnmohEeX39Rcl9WWQRHvzUXnLIct3UJEOH8THQKDJzq3YWkpgyy1kP8UIhJa/2i1w14E/Q3AsOUUumYYL8QuMF9A6VUAlCktXYAvwGWNNrH9c7lQojTQWtT05+wEM7/o6lRv3sbDJhigltEEsQPMcE5Nt0E5iPfwqFVcGKHCewTroPUM53B0C1wBoWZY7hq3MXHzOfz95jX/Z/BllfNNkHh5kIwYDJEJJr9RPczF6O4wfVB0lZjLgwVBWafVcUQGm22D4uF2nJnrb/IlC0iHsLioDzfHLNgrwniZbmQt93sw17r+fcTP9Qcv7IITh2GikKodW/7UNB3rCl3YIhzUYA5ZkQCBFhg32dw4Auw15j10SmQOsOck6usgWHOi088BIY2/z3VVtTfRcUNhqtf7Mg336pWg77W2qaU+hkmNWMBlmitdyqlHgIytdYfAXOAPyulNCa9c4fr80qpQZg7ha99XnohRPOKj5og1ncMBIXCwtfg8wdNgM7fDeWroLqk4WcsIZCSARf+1QT80D4tHyM4HIJTISYVBs1quK6yyATq6BQI8KLpMDDEpGkY2ZazNIaf33SZ1vUXisrC+vRUUDiknQlRyU0/Y6022xYdgqNrzB2B+x2Iw97wTiWqP2T8EIaeZy4cR76FY+vAEmguVpF9wVpl0lnHMz1fhFwXsfB4c7fSyZTuZg0nGRkZOjMzs6uLIUTPtvtjePNGuPVLE8gb09oEwoI9UHTQ1HwHZJgLhPDMbjUXhtoKc4fkzQXtNFFKbdJaN/NlN+SrhlwhRHdyYiegIGlU8+uVgshE8y/9rNNatB7NEtT8XUIP0n0uU8LY8wlkbezqUoie7sR2kx8OjujqkohuRoJ+d1J2At76Abw0Hza+0HDdqSOm+5wQ3jix0+TzhWhE0jvdyeZ/g8MGabPgk3tMl7uh50Lmi3DwK9M49KNPIWFoV5dUdGc15aYnzITru7okohuSmn534bDDplcgfTbc8hHMvBs2vWQa4/L3wFn3mO2WXgbFWS3vS/i3/N2ANl0OhWhEavrdxYEvoeQYnP//TB/g8/4AQ+eZvtBDzzXdwEZfDi9fDP++HH64woxe7En2fw4HV8L8P3V1SXq3E9vNq6R3RDOkpt9dZC4x/XpHXlS/LP1sGDHfBHyAfuPhxrdMbv+D27umnO1lt8Hye2HdU1B4sHP2f/x7WP9c5+y/J8nbYUbHxqR2dUlENyRBvzsozoL9n8Kkm02XsJakzoAxVzhv4XuQ3R+aASxgeij5isMO798OjwyC5+fCinvh7VvMRaAzaQ17lsN7i81ozu7E1YjbA6YEEKefBP3u4PulJohMucW77cNizSRUPYXW8O3fIX4Y9B3n26C/dzlsfR2GXwBXvQgXP26G4Tfu/eRLJ3aZFNuy62Hbm7DmH513LE+Kj8HeFU2XOxzOoC/5fNE8CfpdzW4zQX/Y+d7fjofFmsmurNWdWzZfOfilCcSz7jYTfmWtN3Om+MK6f0GfVLjiWRh3NUxZBEPmwVd/hLI83xzDxVoFn/0P/Gsm5GyB+Y+YdpYNL5hpB3wlZ4v5fbXkm7/CGwshZ3PD5SXH6qdfEKIZEvS72qGVUJ4Hk2/2/jPhcea1o7V9uxW2vWVeO9O3j5t5SsZd62yz0M3XUtsqd5uZH2XabfXtHkrBgr+aeU4+/V3Hj+FydA08MxPW/NOk4e7cDDN+ArPvA2sFrHvGd8f6+G6TsmrJcWew/+z3DeegP/69eU0e57vyiF5Fgn57HVplgkBNecf2s+V1U3MfdoH3n3FN+VrVwdrlhufhvdtMiqSzZG2EI6vhzJ9BYLCpgcak+SbFs/5fZrKqxhfM+CEw6xew4x3zPXXU3hXw0oVmDMUPPoRLn6i/8PYdbe5e1j/bdAKz9jp1xPTA8XT3YK0y0xHHpJrf7T7nYyrK8uDT35rfr6R3hAcS9Ntr14dmCtrjm9q/j6piE/zGXm0CorfCfFDTrymD1Y+an/N2tH8/nlQVw6pH4LWrzEVqsrO9QikYebEJxh15jF95Pmx/GyZe3/y857N+Yaby3fxq+4/hsuNdMxXx7Wtg8Jym68++F2pKTM+hjqopr/9ej3zb/DZ5O0Db4byHzERpn//eTAD25k1QXQrXvyETpwmPelfQ91VNyxuu+XGyvZwn5+QBeGaWmf3QZdcHZi7uiW0cOekKch3JI699yswWGNrHXLx8aef78Ph4WPUnM7p40SfmgRouIy8y533gy/YfI/Mlk8KZ/pPm1weFmod5FB9r/zHApE4OrzaTkrmfg7t+E2D4fNMdtaPPoy1xG3h3ZHXz27jy+CnTTOA/uQ+enW3+L17xjOTzRYt6T9AvPAj/zPBNza41NWWQ73wUnTc1/eoS09PjxHb44Kf1gWjrMvOAif6T23b8uvROO2v6FSdNbnrUpabR09c1/W//DlF94f9bDde/3jQIDZxu7lbam+IpLzC9c4aeBwnDPG8Xk9bxoF94wLS5DGplJsqz7zXfx/dLO3Y812jrsDhzsWlOzmZz5xHdH0YsMA86Kdxv2hdGX9ax44ter/cE/Zg0M43sx7/o/FkqczabBytE9Te1q5aeSeCwmycWFR2Cy54yn3tvsan5H1tr5kdpa3/qjjbkrn7M9P455/eQPNb0+PDVXZLdasYQDJ9vBpM1xxIIIy6Eff+Fgn1t23/uNnhujrnwzv51y9vGDDRPUrLVtO0Y7g5/Y17Tz255u5QMSJtpGnQ7Mkag+Kh5HX8tFOw2F7jGcjZD/0nm/41ScMW/YMGjMLuNz8AVfqn3BH1LIFzzsqn9vHkTlOZ23rGyNpjXabeax7u5/lCbs/JhM/Bq/l9g0k1w0aMm2L92NaBg/HVtP35QOFiC29eQm70JNj4PE2+AxOGm3zzUP0S7TfvKNI2O7gr2mLRLvwktf3bmXeZpSUsu8P4ivfN9sz0afrQCBk5reXtXF9iSbO/235wjq83FPW5w69ue8TOTntn1QfuPV5JlvtuxV9Uf311NOZzca4K+S2ya6cHUjR7oIbqv3vW/JDwOFr5haoFv3tj+fux2W8u1w+yNJi0z9Dznew9P+srfA6v/BpN/AFNvNcvGXwfjrjGjUwfPhj4D2l4+pcztf1tq+tZq87i8F881s3XO+a1Z7kq9tDXoH1sPS+bD8ka17dxt5jXZQy3fJXEE/Pgz06bwyiXmeaMt2fcpvL3I9Eq5bWXDoOeJK+i3N8WjtWlMTT/Lu7ux4fNNw+qaJ1q++2tJcZZ5ZF7/yRAc1TTo5203d4venL8QzehdQR9MF7ornzO59g3t7E3xn7tMQ2Rzc7hobYJ+ylRIGm1q3Z4ac7PWmddZv6gPGkrBRX8zk6jNvKt95QOT1/e2Ibc4C/41C757HCbeCD9dW3+xie5v9tXaYCB3JcfN3ZTDau5aHPb6dXnbze8kfkjr+4kbbAJ/4nBzkfZ0PmUnTFtI33Fwy39Me4E3Ohr0C/aYO7nW8vkuAQGmtp+71XPPm9aUZEGfgebONe2Mpnl9VyNu/4nt27/we70v6AOMutj8weduaftny/Nh2zLTeLf0sqapgaJDptdLylTzh9l/kueafs5mU5ONTW+4PLQP3PQuDDmn7eVzCY8z3SK98f0r5jmoN70Hlz0JYTH165QytWdve/BYq2DZDeZ11i/Ng6Ld7xLytpn9BVi8219kkrnrsNc2f5F1OOCDn5guiVe90LauiFH9QVnaH/RdAbctjxOcsNDcSa19sn3HLD5m2iLAXGwK9zdMVeZsNufVwx/ZJ7pO7wz6YLrrFext++dcDzK56kXTuLn08oaNaa5avSufnJJhAl1z6aCcLdBvYudMfBUW631O/8CX5qHXQ+c1v77vWNP46l5j9+Q/d5ua7FXPQ8YPzbJja82rw2Fq+m0dDRqbZl6baxtZ/4x5gMwFD0PSyLbt1xJo7mi8DfqbX4U3rnfrJ/+NmeIhdpD3xwwKM/n1ff9t+/8/azWUnzDHhPqLjftdg6sRV4h26sVBfwSc3N+2nhQOO2x62dSwxl0NN7xpavqvXWVqmmAacYOjzEUFTI3fXlufy3ax1ZgacGf9gXo76VpFoQkUngI+mB481krztKWW7PvU3AXN/rXpfROTalIRR9eY9cVHTM3fU68dT1xpmMaNwkWH4Yv/hREXQcaP2rbPun172W3TVgNf/MGMTn7lUvN7c+Xz22rqrRAYBt890bbPlR53ltlZ008eb+4KD39t3leXmpq/BH3RAb046I8yA4Ba6lnT2MGvTIBwBZi0M+HaV0xA//AOZz5/AwyYXJ++GJBhXhvn9U/sNDnvzgz6lUWtNxgeWglo0x/fE9eQ/RMt5PWtVWY+/IThcNav6pennmFq+lp734jbWHCEGT3bOOhnbTAX1Hm/b//dUp+B3gX9He9BRb5pZynYa7qFVp3yPp/vLiLBTA2x7c2m6cGW5jlyldN1EQywmM4Cm181Deaui6sEfdEBvTjoO2viBXu8/0zmEhN8Rl5cv2z4BXDu/5rugl/90QRz966C0f0gOgWON8rr1zW4ddIfaHicuahZq1re7uBXEBpjLlSeJI40ue+WevCsfsxcQC/6W8MpI9LOMCmJokMmtaMspoG7rWLSml6gC/eDCoA4LxqFPe431dlXv9bzNlrDuqfN7+HcP5hpDCqcs4C2p6YPpkFXO2Dt0/XLCg/C30aaCeia4wr6fQbWL7vkHzBtMWx41jR2gzTiig7pxUF/uHn1NuiXHDd52Ek3N50HZ+ZdZn6c1Y+aP+SURv3DUzKa1vRzNptulZ319CJvJl3T2gT9wXNablgNCjUjWz2NzD15wPT8GXdt00FKqWea16NrTNtG4oj2zfsSOwhONQ76B8zFoC3zEjUWkwpoKG2hr/6xtabs039i7iiGzjMTq81/xHSfbI/YNJMi3PSyuSOzVpmHu1SeNF06m+tOXJJlLnLR/euXhUTCgv8zj8d0TaQWkdC+MglBbw76IVGmBu5tY1pLDzJRCi79p3PAkTJB3l3KVFNLK82pX5azpX7UZGfwZtK1/F2mlttSPt+l71jPNf0V90JgKJz/x6brEkdAeLwJnLnb2p7acYlNM6kQ9zaYkwdanmbBG95021z3tLmIug+US51hpk7uiJl3mWmXNzwPK+4zd0Jn3ml6f21/u+n2xVmmZ05zT09LOxN+thFu/aJjZRJ+r/cGfTAByZvHCjocsOU104XSU0+N4HDT5fHm9+unQXAZ7pwWedMr5tU19W1n5l69mXTNNaFZS/l8l75jzHQMjbuBlp0wdwtn3tl8/3ilTF5/339NN9e2NuK6xKSZmSNdNXKHw3QzjR/avv3V7beVoH/qqJkDaPIt5jv2pb5jzICt1X8z3WZn/dJMkNZ3rJmuoXF7TElWfSNucwIspneQEB3Qu4N+0igzA2FrXRGPrDZ/cBNvaHm7iAQYMrfp8oRh5o974/Mm4Lumvu3MoO/N/DsHvzR5am9G/SZ7mI4h2znlxODZnj+beoapvULHavpQn+IpyzE9ijoa9KMHeO6rb7fBV/8PUKabZWeY9QvT9pI2C+b+zlwkp//ETNjXeLRt8TF5mLnodL076CeOAFt16703ti6DkGjnU53a6cyfm8C39Y3Ob8SF1nP6tZVwdK13tXyonyuncdtE1gYICGo5mKedUf9zcjsf3uG6w3I15hYeMK8dDfqWQBP4G/8fqCwyXXG3vw1n3dP+3H1rUmeY9oGFr9U/3WvcNSYl5v60LbvNpAf7tFDTF8IHennQd/XgaSGvX1NuHogy5vKO3TqnzTRBfs2TZgoI19S3naW16ZWPrjE1zKFejvqNTDK/L9eski7ZmeaC0FLjbPIE8wSrmNTmH2jijegUUyN31fRP7jevHc3pg0mZuAf9gn3wwjw48h1c+iSc48PHKjZn8JyGo6CDQk234L0rTK8nMHc22t5yekcIH+jdQT/Bix48u/9jGtsmtJLaaY1SprZfdBB2vte5jbhgLlCBYZ5z+q55/l29a7yRfrZpkHV1b7RbzV1La7NZWgJh0o2mh1N71Y2eddX0D5oLSVS/9u/TJSa1fp56hwPeutlMyrfok7Y9m9iXMn5scvSrHzPvXeWTmr7oZL076IfFmKDRUk1/6xsmtZA6o+PHG3WZGUJvrz09A2jCYj3Pv1OSBZF929Y4mT7b5NFdYw7ytoOtyvROas2Cv8K5D3p/rObEpNUP0CrcbyZt88WFMybV1KRttbBvhakEzP8LpE7v+L7bK7qfye1v/re546gbmJXWdWUSfqF3B31wzsHjoQdPSbZJZ7TnQSbNsQTCjNvNz6cj6IfHec7pl2S1PU89aCag6lM8ronkvAn6vhCbVp/eKfRBd02XmFQzvqI029SsY9Jg9OW+2XdHzHU+xPw/d5qLHHRe24IQTn4S9PeZ2/rGti4DdPseZOLJ1Fvhimdh2Hm+26cnLc2/U5Ld9lRBWKzJ3x9yzvWSvcHcKZ2uQBQ7yIyErTplar4dbcR1cfWI2brM3MXMvLO+UbUrBUeYEbeFB0yjbkSSPNBcdDo/CPojTM7efURmdQl89j+w6i9mbpW4dM+fb6vAYDO9rrdTC3eEpzn1tXYG/XYE68HOB2zXVpieOylTO7dtwl3MIPN66GtTM4/3YU0fzPQHEYnmmQLdxZC5pjzWSmnEFaeFHwR9tx48ZXmmRvXPKaaXzfjr4OolXVu+jvBU068oMF1V29PnO/1sM1Hcro9Mo2prjbi+5Oqrf8A56tSbB7F4I3qAmd7AXmPSb91tgNP5fzS1/IQRXV0S4Qe8usdVSs0H/gFYgBe01n9ptD4NWAIkAkXATVrrbOe6VOAFYCCggQVa6yO+OoFWJTr/kD6600xJgDaDiW58u+fPVujK6WvdsDZe0oGeIKlnmH753zp7lZyufD7UN2K6RhL7Kr1jCTLTG1SXmF4z3U14HNz+Xfe7GIleqdWgr5SyAE8B5wHZwEal1Eda611umz0KLNVav6KUOgf4M+DqC7cUeFhr/blSKhJoJrneicLjYOB000VvyiIYfVnbH8bRXYXFmge+1JabuYZc6rr/tSO9ExxhAv2xNSb49zuNMzpGJpluqGU5EJkModG+2/cZd5jfkXt/+e4kMqmrSyD8hDc1/WnAAa31IQCl1DLgMsA96I8Gfun8eSXwgXPb0UCg1vpzAK11uY/K3TY/buWh2z2Va9K1yqKGQd81h3t7c8TpZ5ug32/86W1YVMqkeAr2+K6W73LGT327PyF6KG9y+gOALLf32c5l7rYCVzp/vgKIUkrFA8OBYqXUe0qpzUqpvzrvHIQveBqVW5IFwZFmHv32cM2z03gK6dPBleJJ8HHQF0IAvmvI/RUwWym1GZgNHAfsmDuJs5zrpwKDgUWNP6yUWqyUylRKZRYUFDReLTypm3StUQ+e4iyTz29vr5uUqTB+oemFdLq5GnN9XdMXQgDeBf3jmEZYlxTnsjpa6xyt9ZVa60nA75zLijF3BVu01oe01jZM2qfJI5y01s9prTO01hmJiYntPBU/1FJNvyN96y1BcOWzXfOEJldN31fdNYUQDXgT9DcCw5RS6UqpYGAh8JH7BkqpBKWUa1+/wfTkcX02RinliuTn0LAtQHSEe07fXWvzsndnA6ebi5k8ElCITtFq0HfW0H8GfArsBt7SWu9USj2klLrUudkcYK9Sah/QF3jY+Vk7JrXzpVJqO6CA531+Fv6qrqbvNv9OTbmp+ffUibsGToX7jkBUcleXRIheyat++lrr5cDyRssecPv5HeAdD5/9HGjnkzVEiwKDTYOte3rH1XOnpwZ9IUSn6v0jcnu7sNiGDbkd7a4phOjVJOj3dGExjWr6zil6ZbZGIUQzJOj3dGFxDRtyi7MgINA3Dx8RQvQ6EvR7usaTrpVkm8c0no5ZPoUQPY4E/Z6uT4qZDbPcOaitJEsacYUQHknQ7+mmLDLPsl3/L/O+PQ9PEUL4DQn6PV3CMBh1CWx83qR5SnOkEVcI4ZEE/d5g1t1mrviVfwJtl+6aQgiPJOj3BgOmQPps2PiCeS/pHSGEBxL0e4tZvzDPlQUJ+kIIjyTo9xaD59Q/5Upy+kIID7yae0f0AErBxY/BwZUQHN7VpRFCdFMS9HuTAVPMPyGE8EDSO0II4Uck6AshhB+RoC+EEH5Egr4QQvgRCfpCCOFHJOgLIYQfkaAvhBB+RIK+EEL4EaW17uoyNKCUKgCOdmAXCcBJHxWnp/DHcwb/PG9/PGfwz/Nu6zmnaa0TW9uo2wX9jlJKZWqtM7q6HKeTP54z+Od5++M5g3+ed2eds6R3hBDCj0jQF0IIP9Ibg/5zXV2ALuCP5wz+ed7+eM7gn+fdKefc63L6QgghPOuNNX0hhBAe9Jqgr5Sar5Taq5Q6oJS6v6vL01mUUgOVUiuVUruUUjuVUnc5l8cppT5XSu13vsZ2dVl9TSllUUptVkp97HyfrpRa7/zO31RKBXd1GX1NKRWjlHpHKbVHKbVbKXVGb/+ulVK/cP7f3qGUekMpFdobv2ul1BKlVL5Saofbsma/W2U84Tz/bUqpye09bq8I+kopC/AUcCEwGrheKTW6a0vVaWzAPVrr0cAM4A7nud4PfKm1HgZ86Xzf29wF7HZ7/wjwd631UOAU8OMuKVXn+gfwX631SGAC5vx77XetlBoA3AlkaK3HAhZgIb3zu34ZmN9omafv9kJgmPPfYuCZ9h60VwR9YBpwQGt9SGtdCywDLuviMnUKrXWu1vp7589lmCAwAHO+rzg3ewW4vGtK2DmUUinARcALzvcKOAd4x7lJbzznPsDZwIsAWutarXUxvfy7xjzRL0wpFQiEA7n0wu9aa/0NUNRosafv9jJgqTbWATFKqX7tOW5vCfoDgCy399nOZb2aUmoQMAlYD/TVWuc6V+UBfbuoWJ3lceDXgMP5Ph4o1lrbnO9743eeDhQALznTWi8opSLoxd+11vo48ChwDBPsS4BN9P7v2sXTd+uzGNdbgr7fUUpFAu8Cd2utS93XadMlq9d0y1JKXQzka603dXVZTrNAYDLwjNZ6ElBBo1ROL/yuYzG12nSgPxBB0xSIX+is77a3BP3jwEC39ynOZb2SUioIE/Bf01q/51x8wnW753zN76rydYKZwKVKqSOY1N05mFx3jDMFAL3zO88GsrXW653v38FcBHrzd30ucFhrXaC1tgLvYb7/3v5du3j6bn0W43pL0N8IDHO28AdjGn4+6uIydQpnLvtFYLfW+jG3VR8Btzh/vgX48HSXrbNorX+jtU7RWg/CfLdfaa1vBFYCVzs361XnDKC1zgOylFIjnIvmAbvoxd81Jq0zQykV7vy/7jrnXv1du/H03X4E/MDZi2cGUOKWBmobrXWv+AcsAPYBB4HfdXV5OvE8Z2Fu+bYBW5z/FmBy3F8C+4EvgLiuLmsnnf8c4GPnz4OBDcAB4G0gpKvL1wnnOxHIdH7fHwCxvf27Bv4A7AF2AP8GQnrjdw28gWm3sGLu6n7s6bsFFKaH4kFgO6Z3U7uOKyNyhRDCj/SW9I4QQggvSNAXQgg/IkFfCCH8iAR9IYTwIxL0hRDCj0jQF0IIPyJBXwgh/IgEfSGE8CP/P1Z0ZnvBNTetAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(history.history['accuracy'])\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.legend(['training', 'validation'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Xavier 权重初始值\n",
    "\n",
    "将各层的默认初始化器，统一修改为 Xavier 权重初始值，重新训练网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 64)                50240     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 59,210\n",
      "Trainable params: 59,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "weight_model(\"glorot_normal\")\n",
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9.1 模型训练\n",
    "\n",
    "> **预计训练时间 5 分钟，请耐心等待**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "165/165 [==============================] - 4s 14ms/step - loss: 0.0100 - accuracy: 0.9979 - val_loss: 0.5831 - val_accuracy: 0.9641\n",
      "Epoch 2/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0136 - accuracy: 0.9976 - val_loss: 0.6444 - val_accuracy: 0.9609\n",
      "Epoch 3/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0092 - accuracy: 0.9980 - val_loss: 0.5558 - val_accuracy: 0.9643\n",
      "Epoch 4/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0063 - accuracy: 0.9986 - val_loss: 0.4899 - val_accuracy: 0.9659\n",
      "Epoch 5/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0081 - accuracy: 0.9982 - val_loss: 0.5335 - val_accuracy: 0.9627\n",
      "Epoch 6/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0107 - accuracy: 0.9975 - val_loss: 0.4939 - val_accuracy: 0.9652\n",
      "Epoch 7/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0079 - accuracy: 0.9981 - val_loss: 0.5234 - val_accuracy: 0.9627\n",
      "Epoch 8/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0046 - accuracy: 0.9988 - val_loss: 0.5223 - val_accuracy: 0.9646\n",
      "Epoch 9/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0049 - accuracy: 0.9986 - val_loss: 0.5019 - val_accuracy: 0.9663\n",
      "Epoch 10/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0087 - accuracy: 0.9982 - val_loss: 0.5363 - val_accuracy: 0.9653\n",
      "Epoch 11/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0112 - accuracy: 0.9976 - val_loss: 0.5147 - val_accuracy: 0.9640\n",
      "Epoch 12/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0313 - accuracy: 0.9940 - val_loss: 0.5144 - val_accuracy: 0.9636\n",
      "Epoch 13/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0121 - accuracy: 0.9969 - val_loss: 0.4602 - val_accuracy: 0.9671\n",
      "Epoch 14/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0116 - accuracy: 0.9974 - val_loss: 0.5268 - val_accuracy: 0.9635\n",
      "Epoch 15/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0071 - accuracy: 0.9984 - val_loss: 0.4618 - val_accuracy: 0.9647\n",
      "Epoch 16/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0092 - accuracy: 0.9983 - val_loss: 0.4431 - val_accuracy: 0.9659\n",
      "Epoch 17/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0018 - accuracy: 0.9994 - val_loss: 0.4917 - val_accuracy: 0.9662\n",
      "Epoch 18/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 9.1572e-04 - accuracy: 0.9998 - val_loss: 0.4834 - val_accuracy: 0.9665\n",
      "Epoch 19/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 9.3759e-04 - accuracy: 0.9998 - val_loss: 0.5130 - val_accuracy: 0.9661\n",
      "Epoch 20/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 0.0059 - accuracy: 0.9985 - val_loss: 0.5009 - val_accuracy: 0.9647\n",
      "Epoch 21/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0028 - accuracy: 0.9995 - val_loss: 0.4851 - val_accuracy: 0.9668\n",
      "Epoch 22/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0071 - accuracy: 0.9984 - val_loss: 0.5675 - val_accuracy: 0.9627\n",
      "Epoch 23/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0162 - accuracy: 0.9965 - val_loss: 0.5191 - val_accuracy: 0.9624\n",
      "Epoch 24/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0212 - accuracy: 0.9953 - val_loss: 0.5037 - val_accuracy: 0.9624\n",
      "Epoch 25/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0202 - accuracy: 0.9956 - val_loss: 0.4753 - val_accuracy: 0.9658\n",
      "Epoch 26/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0122 - accuracy: 0.9972 - val_loss: 0.4465 - val_accuracy: 0.9669\n",
      "Epoch 27/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0054 - accuracy: 0.9985 - val_loss: 0.4309 - val_accuracy: 0.9668\n",
      "Epoch 28/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 0.0040 - accuracy: 0.9989 - val_loss: 0.4728 - val_accuracy: 0.9671\n",
      "Epoch 29/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0048 - accuracy: 0.9987 - val_loss: 0.4843 - val_accuracy: 0.9651\n",
      "Epoch 30/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0033 - accuracy: 0.9989 - val_loss: 0.4674 - val_accuracy: 0.9673\n",
      "Epoch 31/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0096 - accuracy: 0.9977 - val_loss: 0.5210 - val_accuracy: 0.9632\n",
      "Epoch 32/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0191 - accuracy: 0.9962 - val_loss: 0.4948 - val_accuracy: 0.9655\n",
      "Epoch 33/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0152 - accuracy: 0.9967 - val_loss: 0.5137 - val_accuracy: 0.9653\n",
      "Epoch 34/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0064 - accuracy: 0.9982 - val_loss: 0.5027 - val_accuracy: 0.9675\n",
      "Epoch 35/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0031 - accuracy: 0.9990 - val_loss: 0.4789 - val_accuracy: 0.9679\n",
      "Epoch 36/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0038 - accuracy: 0.9990 - val_loss: 0.4873 - val_accuracy: 0.9680\n",
      "Epoch 37/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0055 - accuracy: 0.9986 - val_loss: 0.4971 - val_accuracy: 0.9671\n",
      "Epoch 38/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0211 - accuracy: 0.9960 - val_loss: 0.5188 - val_accuracy: 0.9641\n",
      "Epoch 39/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0128 - accuracy: 0.9975 - val_loss: 0.5247 - val_accuracy: 0.9666\n",
      "Epoch 40/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0057 - accuracy: 0.9985 - val_loss: 0.5112 - val_accuracy: 0.9656\n",
      "Epoch 41/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0088 - accuracy: 0.9979 - val_loss: 0.5092 - val_accuracy: 0.9655\n",
      "Epoch 42/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0049 - accuracy: 0.9989 - val_loss: 0.5024 - val_accuracy: 0.9663\n",
      "Epoch 43/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0057 - accuracy: 0.9986 - val_loss: 0.5255 - val_accuracy: 0.9656\n",
      "Epoch 44/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0091 - accuracy: 0.9980 - val_loss: 0.5082 - val_accuracy: 0.9645\n",
      "Epoch 45/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0069 - accuracy: 0.9983 - val_loss: 0.5063 - val_accuracy: 0.9671\n",
      "Epoch 46/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0104 - accuracy: 0.9977 - val_loss: 0.5476 - val_accuracy: 0.9653\n",
      "Epoch 47/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0223 - accuracy: 0.9961 - val_loss: 0.5519 - val_accuracy: 0.9637\n",
      "Epoch 48/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0179 - accuracy: 0.9967 - val_loss: 0.4843 - val_accuracy: 0.9667\n",
      "Epoch 49/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0056 - accuracy: 0.9987 - val_loss: 0.4882 - val_accuracy: 0.9674\n",
      "Epoch 50/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0037 - accuracy: 0.9991 - val_loss: 0.4970 - val_accuracy: 0.9682\n",
      "Epoch 51/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0058 - accuracy: 0.9988 - val_loss: 0.5203 - val_accuracy: 0.9649\n",
      "Epoch 52/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0083 - accuracy: 0.9982 - val_loss: 0.4953 - val_accuracy: 0.9657\n",
      "Epoch 53/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0062 - accuracy: 0.9985 - val_loss: 0.5185 - val_accuracy: 0.9659\n",
      "Epoch 54/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0082 - accuracy: 0.9980 - val_loss: 0.5295 - val_accuracy: 0.9681\n",
      "Epoch 55/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0048 - accuracy: 0.9989 - val_loss: 0.5098 - val_accuracy: 0.9679\n",
      "Epoch 56/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0058 - accuracy: 0.9988 - val_loss: 0.5440 - val_accuracy: 0.9650\n",
      "Epoch 57/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0163 - accuracy: 0.9969 - val_loss: 0.4655 - val_accuracy: 0.9668\n",
      "Epoch 58/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0085 - accuracy: 0.9980 - val_loss: 0.5414 - val_accuracy: 0.9654\n",
      "Epoch 59/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0137 - accuracy: 0.9974 - val_loss: 0.5521 - val_accuracy: 0.9662\n",
      "Epoch 60/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0078 - accuracy: 0.9981 - val_loss: 0.4793 - val_accuracy: 0.9684\n",
      "Epoch 61/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0029 - accuracy: 0.9992 - val_loss: 0.5073 - val_accuracy: 0.9675\n",
      "Epoch 62/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0010 - accuracy: 0.9997 - val_loss: 0.4981 - val_accuracy: 0.9678\n",
      "Epoch 63/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0042 - accuracy: 0.9991 - val_loss: 0.5310 - val_accuracy: 0.9683\n",
      "Epoch 64/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0124 - accuracy: 0.9975 - val_loss: 0.5516 - val_accuracy: 0.9644\n",
      "Epoch 65/100\n",
      "165/165 [==============================] - 3s 18ms/step - loss: 0.0140 - accuracy: 0.9975 - val_loss: 0.4935 - val_accuracy: 0.9677\n",
      "Epoch 66/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0043 - accuracy: 0.9988 - val_loss: 0.4830 - val_accuracy: 0.9679\n",
      "Epoch 67/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0038 - accuracy: 0.9993 - val_loss: 0.5138 - val_accuracy: 0.9669\n",
      "Epoch 68/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0150 - accuracy: 0.9970 - val_loss: 0.4980 - val_accuracy: 0.9645\n",
      "Epoch 69/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0087 - accuracy: 0.9979 - val_loss: 0.4773 - val_accuracy: 0.9669\n",
      "Epoch 70/100\n",
      "165/165 [==============================] - 3s 15ms/step - loss: 0.0021 - accuracy: 0.9995 - val_loss: 0.5004 - val_accuracy: 0.9669\n",
      "Epoch 71/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0021 - accuracy: 0.9994 - val_loss: 0.4704 - val_accuracy: 0.9684\n",
      "Epoch 72/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0059 - accuracy: 0.9987 - val_loss: 0.5033 - val_accuracy: 0.9663\n",
      "Epoch 73/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0187 - accuracy: 0.9964 - val_loss: 0.5101 - val_accuracy: 0.9661\n",
      "Epoch 74/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0193 - accuracy: 0.9966 - val_loss: 0.5063 - val_accuracy: 0.9659\n",
      "Epoch 75/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 0.0169 - accuracy: 0.9969 - val_loss: 0.4568 - val_accuracy: 0.9678\n",
      "Epoch 76/100\n",
      "165/165 [==============================] - 2s 15ms/step - loss: 0.0140 - accuracy: 0.9977 - val_loss: 0.4618 - val_accuracy: 0.9672\n",
      "Epoch 77/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0034 - accuracy: 0.9992 - val_loss: 0.4618 - val_accuracy: 0.9689\n",
      "Epoch 78/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0033 - accuracy: 0.9994 - val_loss: 0.4771 - val_accuracy: 0.9694\n",
      "Epoch 79/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0022 - accuracy: 0.9996 - val_loss: 0.5179 - val_accuracy: 0.9669\n",
      "Epoch 80/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0097 - accuracy: 0.9984 - val_loss: 0.5167 - val_accuracy: 0.9669\n",
      "Epoch 81/100\n",
      "165/165 [==============================] - 3s 16ms/step - loss: 0.0040 - accuracy: 0.9990 - val_loss: 0.4958 - val_accuracy: 0.9666\n",
      "Epoch 82/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0065 - accuracy: 0.9986 - val_loss: 0.5034 - val_accuracy: 0.9651\n",
      "Epoch 83/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0052 - accuracy: 0.9989 - val_loss: 0.5158 - val_accuracy: 0.9674\n",
      "Epoch 84/100\n",
      "165/165 [==============================] - 3s 19ms/step - loss: 8.0663e-04 - accuracy: 0.9998 - val_loss: 0.5042 - val_accuracy: 0.9688\n",
      "Epoch 85/100\n",
      "165/165 [==============================] - 4s 22ms/step - loss: 0.0013 - accuracy: 0.9996 - val_loss: 0.5021 - val_accuracy: 0.9687\n",
      "Epoch 86/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0094 - accuracy: 0.9980 - val_loss: 0.5167 - val_accuracy: 0.9660\n",
      "Epoch 87/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0078 - accuracy: 0.9983 - val_loss: 0.5286 - val_accuracy: 0.9647\n",
      "Epoch 88/100\n",
      "165/165 [==============================] - 3s 17ms/step - loss: 0.0175 - accuracy: 0.9971 - val_loss: 0.5930 - val_accuracy: 0.9634\n",
      "Epoch 89/100\n",
      "165/165 [==============================] - 3s 19ms/step - loss: 0.0144 - accuracy: 0.9965 - val_loss: 0.5751 - val_accuracy: 0.9648\n",
      "Epoch 90/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0156 - accuracy: 0.9972 - val_loss: 0.5128 - val_accuracy: 0.9664\n",
      "Epoch 91/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0059 - accuracy: 0.9989 - val_loss: 0.4373 - val_accuracy: 0.9685\n",
      "Epoch 92/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 0.0022 - accuracy: 0.9993 - val_loss: 0.4754 - val_accuracy: 0.9675\n",
      "Epoch 93/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0040 - accuracy: 0.9990 - val_loss: 0.5093 - val_accuracy: 0.9662\n",
      "Epoch 94/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0029 - accuracy: 0.9992 - val_loss: 0.5133 - val_accuracy: 0.9686\n",
      "Epoch 95/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 0.0108 - accuracy: 0.9980 - val_loss: 0.5451 - val_accuracy: 0.9646\n",
      "Epoch 96/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 0.0246 - accuracy: 0.9957 - val_loss: 0.4709 - val_accuracy: 0.9663\n",
      "Epoch 97/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0069 - accuracy: 0.9985 - val_loss: 0.4952 - val_accuracy: 0.9652\n",
      "Epoch 98/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 0.0073 - accuracy: 0.9986 - val_loss: 0.4744 - val_accuracy: 0.9662\n",
      "Epoch 99/100\n",
      "165/165 [==============================] - 2s 14ms/step - loss: 0.0045 - accuracy: 0.9988 - val_loss: 0.5445 - val_accuracy: 0.9668\n",
      "Epoch 100/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 0.0059 - accuracy: 0.9990 - val_loss: 0.5299 - val_accuracy: 0.9667\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=256, epochs=100, \n",
    "                    validation_split=0.3, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9.2 训练集和验证集准确率可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzs3Xl8VNX5+PHPk30hCWQjkLCD7MgSFmXHDVcEV6rWnVal1vrVn1jb2qq01fpVa9X264KKdUdRtCqLgIAsEvYdwpoQIIFA9m2S8/vj3AmTZJJMQiBYn/frlVdm7j1z585k5j7nPGeJGGNQSiml/Jr7BJRSSp0dNCAopZQCNCAopZRyaEBQSikFaEBQSinl0ICglFIK0ICglFLKoQFBKaUUoAFBKaWUI6C5T6AhYmNjTceOHZv7NJRS6kdlzZo1R40xcfWV+1EFhI4dO5KSktLcp6GUUj8qIrLfl3KaMlJKKQVoQFBKKeXQgKCUUgrwsQ9BRGYAVwCZxpg+XvYL8HfgMqAQuM0Ys9bZdyvwO6foU8aYt53tg4C3gFDgK+DXphFrcZeVlZGenk5xcXFDH6q8CAkJISkpicDAwOY+FaXUGeZrp/JbwEvAzFr2Xwp0c36GAv8EhopINPA4kAwYYI2IzDHGHHfK3A2swgaE8cDXDX0B6enpRERE0LFjR2xcUo1ljOHYsWOkp6fTqVOn5j4dpdQZ5lPKyBizBMiuo8gEYKaxVgItRaQNcAkw3xiT7QSB+cB4Z1+kMWal0yqYCVzdmBdQXFxMTEyMBoMmICLExMRoa0upn6im6kNIBNI87qc72+ranu5le6NoMGg6+l4q9dN11ncqi8gUEUkRkZSsrKzmPp2zQll5BdkFJRzNKyEzt5ij+SXov0JVbvpZOCkrr4S5Ww5zvKC0uU/lR6GpAsJBoJ3H/SRnW13bk7xsr8EY86oxJtkYkxwXV+9EuzPuxIkTvPLKKw1+3GWXXcaJEyfqLPOHP/yBBQsWVN4vrzAcyS1mx+E80o8XkZFTxOHcYjJOFJFTVFbj8eUVFQ0+L/Xj9tHqNAY8OZ+3l+9r7lNpEkWl5czdcpj044UNfqyrvIIp76Twi3fWMPCp+Ux4aRlvfr9XA2Ydmmqm8hxgqoh8gO1UzjHGHBKRucCfRaSVU+5i4FFjTLaI5IrIMGyn8s+BfzTRuZxR7oBw7733VtnucrkICKj97f3qq6/qPfYTTzxReTu/xMWB7EJc5RVEhQYSHxlCoJ/gJ8KuzHwy80qICg2sTPnkFpWx/1gh7aNDiQoLauSrUz8WJa5y/vTFVt5bdYDo8CAen7OFsvIK7hrZuUHHMcZgDPj51Z46/GJDBm8s20vLsECiw4LonRjFHcObblBHiauc5buPMWd9BnO3HKawtJxBHVox65fnNeg5/m/JHtYdOMEj43tQ4ipnwbYj/OmLrQzq0Ip+SS2b5Fz/2/g67PR9YAwQKyLp2JFDgQDGmH9hRwldBqRih53e7uzLFpEngdXOoZ4wxrg7p+/l5LDTr2nECKOzwbRp09i9ezf9+/cnMDCQkJAQWrVqxfbt29m5cydXX301aWlpFBcX8+tf/5opU6YAJ5fhyM/P59JLL2XEiBEsX76cxMREPv/8c0JDQ7ntttu44ooruOaaa+jetQtXXz+Z7xfOpdzl4uOPP6ZHjx5kZWXxi5/dSNrBDIafdx7fLfqW1atTyC4PxmA4lFtMRGggfto38F8rt7iMn7/xA+vTTvDL0V144MJu/ObD9Tz1n21UGMOUUV18Os6WjBzue3ctQzvF8PS1/byWMcbw/IKd5BW7qDCGnYfz+HTdQYpKXUwd1+2UXsd3O7OYtSadxdszyStxERkSwIT+bQkPCuD1ZXtZsusoo8/xLUuwJSOHFxbs5PJ+bbhnjH39d4zoxOCnFvBxSnqTBYQNaSf4+7e7aBkWyI2D2zO4YytEhLziMjak5dAhJox20WFN8lxngk8BwRgzuZ79Brivln0zgBletqcANeY0nIo/fbGFrRm5TXlIerWN5PEre9e6/69//SubN29m/fr1LF68mMsvv5zNmzdXDtucMWMG0dHRFBUVMXjwYK655hpiYmKqHGPXrl28//77vPbaa1x//fV88skn3HzzzZX7swtKAUPHxAReXreOV155hWeffZbXX3+dP/3pT1x84QVMun0qy7/7lnfefpOs/BLKgwKJiwgmK6+E7IJSYlsEN8n7UeqqwFVRQVhQ0y2DlZVXwmOzNxEZGkiXuBZ0i2/ByHNiCQ7wb7LnaGqu8greXXWA//tuN11bR3D78I6M7hZXZ836dHl5YSob0k/w8s8Gcnm/NgC8OHkAD3y4nj9/tZ12rcK4tG+bOo/x6dp0Hv10EyWuCo4VlDJ9Yh8C/GtmlNceOMGerAKeuaYf1w9uhzGG33y4nmfn7eSc1hFc3DuhUa9hx+E8bp3xAzHhQVzerw0X927N8K72M1DqquDrzYd5bv5ORnWLrWwlbD+ci6vc0CcxqsqxSlzlPPjhBlqGBfHUhJOXmMiQQMb3SeDz9Qd57PKehAQ2/vN1LL+Ev83dwYcpaUSHBVHiquDTtQfpHBdOWJA/WzNyqTDQOTaceb8Z5fW99FVadiFztxzm9uGd8D/Nn68f1eJ2ZxNjDIWl5bjKq+bphwwZUmUM/4svvsjs2bMBSEtLY9euXTUCQqdOnejfvz8AgwYNYt++fZX7KioMR/JKEBEmX39tZZlPP/0UgGXLljF79mwiI4JJHjGWlq1acayglI4tg0iIDKGotJzM3BJahQXi73dqXUZztxzm8c+3kF1QypjucUzon0in2HDWHDhOyr5sggP8ePqafg1q1htjeOSTjSzbdZSWYYHMWmMHn3WOC+fPE/syrHNMPUeo3bur9rNyTzY/G9KeYZ2jmyylsWrPMR6fs4Xth/MY1KEV2w7lcvubq+kcF870q/tyXhffzvmdlfv5etMh2kSFktgqlPM6x/j8WLeDJ4p4c/k+Jg5IrAwGAIH+fvz9hv5sOZjDW8v31RkQnvlmO68s3s2wztFc2qcNj8/ZwsaDOQxs36pG2Vlr0ggN9Ocy57lEhL9e04+9Rwv4zYfr+fTe4XRPiGjQawB4e8U+ggP8WPDgaFqFV01xBgX48atxXZn26SYW7chkXI/WLNt1lLtmrqbEVcHNQzvw8PjuRIYEsu7AcZ6bv5MdR/J487bBNY513aB2fL4+g/lbj3DluW0bfJ5g+zUue3Epx/JLuXN4J+6/sBsBfsJ/Nh7ik7XpCMLUcd0IDvDjb3N38Om6g1yf3K7+AzuMMaQfL2Lxziw+X3eQlP3HARjSKfq0p7r+qwJCXTX5+uQXu8gpLqN1RLBP0fxIbgmZecVkZOZTWl7B0fwSyisM4eHhlWUWL17MggULWLFiBWFhYYwZM8brGP/g4JO1d39/f4qKiirv5xWX4SqvwF+EkJCQyjIul6vKMVqFBXEkr4SKCoMf0CYqBBEhISqE1Mx8svJLSYgMaejbAsDhnGL+8Plm5m09Qo+ECMb3SeCrTYeYt/VIZZmo0EByisoY2z2+3tqop3+vOsDC7Zk8fmUvbh/eidziMlbuPsaT/9nKja+u5PrkJP5wZW9aBDfso7o7K58/zdmKq6KCLzZk0LNNJFPHdq1y0WyMfUcL+Nnrq0iIDOGfNw1kfJ8EysoNX206xIvf7uLOt1fzwZRh9X5xl+7K4vefbaZjTBj7jhZwOLeYGcv2kvK7CxtUc31+/k4A/ufi7jX2Bfj7cf3gdjzzzQ72ZOXTOa5FjTJ7svL553e7uWZgEk9f05fcYhd//GIL3+86WiMgFJWW88WGQ1zWt02Vv0dIoD//d0syV720jLtnprDgwdEEBfhe+cgpKmP22oNM6N+2xgXc7ZpBSby8OJXn5u/ET4RfvLOGTrHhDO0UzcyV+5m39TAdY8JZtTebqNBA/nBFL8b2iK9xnPO7xJDYMpSP16Q3OiDM23qYI7klvHnb4CrPcV1yO67zuPAbY5i35TB/X7CLCf3b1tniTcsuZMmuLJbvPkbKvmyO5JYAcE7rFjx8SXeuOrftGUk9nfXDTs+EghIX+44VcCy/hNSsfIpKXXWWz8orJjOvmFZhQbRLiKYgL5+ME0WkHy+krLyichRDTk4OrVq1IiwsjO3bt7Ny5cp6z6WiwlDmqqDEVU5FhSG3yEVkSCC1VW6HDx/ORx99hJ+fsHnVEnJzThAXEUygE9TCggKICg3kaF4JZeUNH3VkjOG2N39gya4sHhnfgy9+NYI/XtWbFY9ewLt3DeX5G85lycNjWfO7C+kSF87/zt9JeYVvozhSM/OZ/p+tjDonjtvO7wjYZv3FvROY98Bofjm6Cx+vSee1JXvqPL8pM1P41fvrKHVVVG57bPYmggP9+O7hsfx1Ul/KKyq47721vP/DgQa/B57mbT1MeYXhw18M49K+bRARggL8uHpAIh9MGUZ0eBC3vbmaPVn5GGOYv/UI419Ywt0zU8gptCPBjuaX8OBHG+gW34Kvfz2K5Y9ewIzbBpNf4mL57qM+n8v2w7l8sjadW8/rQGLLUK9lrh2YhL+f8FFKutf9byzbS6C/H9Mu7UGAvx/R4UH0bhvJ0tSa5/HNlkPkl7i4Ljmpxr6EqBD+dFVvDmQXssap0frq45Q0isrK+fl5HWstE+jvx/3jurH5YC53vLWaLnEteO/uYfxpQh8+u3c4rSNDSD9exO8u78nyaeO4Y4T3mfZ+fsI1AxNZuiuLjBNFXsvUZ/a6g7SNCqm3P0NE+J+Lu3PwRBEfrk7zWmbxjkzGPbuYkc8s4rHZm1mz7zhDO8Xw5NV9mP+bUcx9YBT3je16xvohfvIBoajUxb6jBQT6+9ExJhxjYHdWAccLvY9bPpZfwqGcYlqGBpLUKpTenZIYPWoEN14ygv996vcUlZWzO6uAkrJyxo8fj8vlomfPnkybNo1hw4bVeS7GGLILS8kuLGXH4TxyisqowJAQVXvN/vHHH2fevHn06dOHeV9+RuuEBDokVE07JESGUGFMra+pLkt3HWX74Tyeurov94zpUhlo/P2E4V1jmTggifYxYQT4+/HgRd1JzcxnzgavI4irKHVV8MCH6wgN9OfZa2ummUKD/Jl2aQ96t43kh721T5L/bP1B5m09whcbMvjV+2spK69g9rqDrNyTzSPje9AuOowbh7Tny1+NZEz3OH47exNzNmQAUFxWzutL93Dfe2vJzPNtdva32zLpkRBBUquaX9D4yBDeuXMoAvx8xg/c8OpK7p6ZQmFpOYt3ZHLlS8vYkpHDQx9vIKeojBcnDyA0yNYaz+8SS0RwAHM3H6lx3No8880OWgQHcN/YrrWWiY8MYVyPeGatSa9RITiWX8KsNelcMzCRuIiTrdQRXeNYd+A4BSVVK0Yfp6TTPjqMoZ2ivT7XiG6x+PsJy1J9ny9UUWF4Z+V+kju0qtEXUN3EAYn0SIigT2IU7909lGinNXFuu5bMmTqC76eN466RnQmvpzV57aB2GGP7TfJLXLz1/V7ufXeN01dXt6y8EpbuOsqEAYk+9ReN7BbLkI7R/GNhKkWl5TX2/3vlAXKKyvjDFb1Y8OBoVjw6jhcnD+CWYR3o1jrijE8U/a9KGdXGGOP1jS0uK2fv0UL8/YROseEEBfgRGuTPgexC0rILMYbKDx3A8YJSDp4oIjIkkKTosMpjvvfee5XPc7ywjEM5Rew+WkDn2HC+/tr74Cl3P0FsbCybN28G4Gh+KT+76z4SIkMI9Pfj9RlvEhTgR0igf5V+heTkZBYvXgxAVFQUc+fOJSAggBUrVpCSklKZWnILDvQnNNCf3CIX8Q1M7762dA/xEcFc5UPz+tI+CfRqE8nz83dxRb+2lcHDm8/XH2TzwVz+edNA4utIZSV3iObD1WmUlVfUOF5ecRl//mo75yZFMaF/Ik98uZWp760lZd9xBrRvyc+GtK8sGxTgxz9vGsStb/7Agx+uZ/PBHD5ff5AjuSX4+wmb0nOYeccQOsaGVz+FSjmFZaTsP849o2sftdMpNpw3bx/M5FdXUlxWzpNX9+HGwe3YmJ7Dve+u4aqXvqe8wvDEhN70bBNZ5fzG9ohn/rYjTC+vqExbuj9T0dVSKfO3HmHh9kweGd+DlvUMK75xcDvmbz3Ct9syGd/nZKfvOyv3U+Kq4M4RVYemjugay7++280Pe7MrUyJp2YUs332MBy86p9aLVERIIP3btWRZ6jEevqTOU6r03c4s9h8r9Jryqi7A34/P7htOkL/fKXXet4+xQe31ZXv513d7yHcCX9uoUH53Ra86HztnQwblFYZJA3xbWEFEeOiS7lz/fyt4Z+W+KiO+jDGsO3CcMd3ja23RnGk/iRZCxoli9h0tqIzQxhibHsrMB6EyGIBtmnaKDadFcAAHjxeR60z4OlFYSvrxQloEB9A+OszrME4RITo8iC5xLcDAnqwCistq1gq8KXWVcyS3mMgQOzqoVXgQcRHBRIXWverogQMHGDx4MOeeey73338/r732mtdyUaGBFJa6GpQ22nE4j6W7jnLr+R19ygn7+Qn/c/E5HMgu5ONaUhRu/165n67xLapcoLwZ1KEVRWXlbDtUc/TYPxamkpVXwh+v6s0dIzrx2GU9mbvlCCeKyvjzxL41LhqhQf68cWsyvdpG8uqSPXSIDueDKcOY9cvzyCsu49p/LWfzwZxaz2XxzkzKKwzjetbMTXvql9SSRQ+N4buHx3LLsA4E+vsxqEMrvvjVCEZ2i2XSwERuGdahxuMu6Z1AdkFpZSciwNPf7GDI9AV8tu5kq2tTeg6//mAdfRIjuX14xzrPBWD0OXG0jgzmw9Un02XFZeXMXLGfC3vG0zW+at9CcsdWBAX4scwjbfTh6jREbC6/LiO6xrIp/URleqw+b6/YR1xEMON9HJ0UEujfJCO5bh/ekYISFxf0jOez+4Zz7aAkZq7cz+GculuKs9el0ycxkm6tfa9ZDekUzfldYnh3VdV0ZVp2EccKShnQ/uyZE/GTaCEEBggnilzsyswjKjQQY+zY7RbBAbSLDqtR8/QToUNMOHuP5nMgu5DYiGCycksICw6gQ0x4vR/IkEB/OseFs/doAXuyCugcF15nR6ExhoMn7AexbcvQBjUTu3Xrxrp16+otFxkayOHcYnKLyojxcQjqG8v2EBLoV6WmXZ9xPeIZ0L4l/1i4i2sHJXkNJJvSc9iQnsPjV/aq97Umd7Qdmyn7jlfpqE3NzGfGsr1cNyiJAU7n592jOjuT86hS+/YUERLIe3cPY3dmPv2Soiqff9Y95/PzN37gxldXsvB/RntttXy7LZOY8CD6+zDSw9vj4yNCeOv2IbU+Zkz3OIIC/Ji75TDDOsew72gBbyzbQ3CAHw98uJ6j+SVc0juBO95eTauwIGbcNtinDugAfz+uG9SOVxankpqZR6fYFnyyNp3sglLu9jJxLSTQnyEdo1m2ywaEDWkn+L8lu7msT5ta+yrcRnSL5e/f7mLFnqOM71N3B/66A8dZvCOLX1/QrUGd0E1hfJ827HgyofK7HBMexOfrD/LSol08dXVfr4/ZdSSPzQdz+X09rQhvLuzZmie+3Er68cLKdOO6NBv4z6aA8JNoIcRHhNC9dQTxESHkFbvIK3HRNiqUTrHhtaY1/P2EjjF2f2ZuMaFB/nSMCfd5HHBIoD+dY8OpcFojdTlRVEZecRkJkSGn7YsRHOBHcIAfucV1d5i7ZeWV8Nm6DK4b1K7WkR/eiAhTx3blUE4x327zng9/d9V+QgP9mTSw7tomYIdktgyt0VH51H+2Ehroz/8b36PK9usHVx3p4U2L4ADObdeySjDqEteCt++wHbuz19XsA3GVV7B4RyZje8SftrkG4cEBjOoWy7wtRzDG8NevtxPo78c3D4zisr4JPPWfbVz50jJKysp56/bBxEf4Pmrs+uR2GODC55bQ7bGvePzzLZybFMWQWvoDhneNZceRPFIz85n6/lriI0KYPrH+aUP927UkPMifpbvq7hwvKHHxmw/Xk9gytNnSJZ5/x3bRYdwwuB0f/JBGWrb3ZTI+XXcQfz/xKX1a3fCusQAsTz1WuW3dgROEBvrTvQGtjdPtJxEQwNaSEqJC6JEQQffWEcRGBNdbOw1w0ketI0PoFBvW4EkhwYH+hAT6U+zynqYxxnA0v4T07CLCggKIaXH6lpgQESJDA8kvcfm0xtE7K/dTVlHhU0qiujHd42kTFcL7XkZW5BaX8fn6DK46t2296TC3QR1akbI/u3L01u6sfBbvyGLKqM5VOkNPVdf4CAa2b8kna9NrrHeTsv84ucUuLqwnXXSqLu6dwMETRbyxbC/fbDnMPaO70C46jH9MHsit53WgpKyCV3+e3KCUBdi8+bt3DuXxK3sxdWxXfja0PU9M6FPrd2BkN3sBu+WNVRw6UcyLkwfU21cBNuU6rHMM33sZpeTpiS+2sj+7kP+9/lyfPwen26/GdcPfT3hhwa4a+1zlFXy+7iAju8U26jN3TusWxLYI5nuPUWTrDhynX1LUKU1aa2pnz5mcIQH+fg2qhQcF+NE6MqTRk7pCAv0oKSuvcYGpqLCTTzJOFBEREkCn2LDTPqIgMiQQYwx59bQSjDG8t2o/F/SI9zp2vT7+fsL1ye1YuiurRm3r0zXpFJWVc7OXHHptBndsxZHcEtKP22GCH65OI8BPuGGI75N9fDVpYBI7j+SzpdqM94XbMwn0F0Z0O70LLF7YszV+AtO/2kabqJDKtYj8/YQ/TejDhscvbvRkvfO7xnL78E48eHF3npjQh3Pb1Z6q6NUmkpZhgRzKKebhS7ozqEPNSWq1GdEtln3HCmutaX+z+TAfpqTxy9FdTmniYVNrHRnCLcM6MHtdOin7qo5se+o/28jIKeamob5/bj2JCOd3iWH57mMYYyguK2dLRm5luvNs8ZMLCGdacIA/rgqDq9rY/APZhRwvLKV1ZAgdYsJOeRaxL8KC/Anw8/O6Mqqn4rIKjuaXNvrDDzZ1I1Bl/LUxhn+vOsC5SVH0Tap7iKGnQR1sWiNlfzYlrnJmrUnnwp6tG5Qy8dUV/doQ5O/HJ2urdoov2HaEYZ1jGjxBrqGiw4MY0ikaY+CR8T0qh6W6nalcu5+fcPPQDkwamOi1n6EuI5z0SPVWQnmF4cuNGTz66Ub6JEbymwvPabLzbSr3je1K++gwbp3xQ2VQ+HD1Ad5avo87R3Tiol6tG33s4V1jyMorYVdmPlsycnBVmLOq/wA0IJx2IYH2LS5xRhu1aNGCCmPYsz+NR++7ndaRITVaBmPGjCElJaXO477wwgsUFp6sgfmynLZNGwVULkxWm4JSFwmRIYzycSExbxJbhjL6nDg+SkmrXN7j07UHSc3M56YGtA4AuidEEBEcQMq+48zfeoTsglImD/W9o7shWoYFcWGveOasz6gckZWyL5s9WQVc4GXm6+lw75iu3Hpeh0blqpvSQ5d057nr+ze4z6RrfAtaRwZXTm7LKy7jgx8OcOFz3zH1vXW0CgvixRsHnPGOZF+0Cg/igynn0ToyhJ/P+IHXl+7hd59tZmS3WB69tEf9B6jD+V1OBsp1B+x3dUAdrbTm8JMYZdSc3KNAil0VuJMvJWXlxLZO4N33P2z0cV944QVuvvlmwsLsiAVfltMGO9oou6CU/GIXkV5yt6WuCkrKKrg+OemUF9KaPKQ9U95Zw8LtmRzKKeaPX2xhcMdWDb7Q+fsJ/du3ZM3+4+w7VkBiy1BGOrXQ02HSgCS+2nSYJTuzSGwVyp1vp9A+OowJ/Rv9T/0aZNQ5cacUjJubiJ20uGDrEW578weWpx6jtLyCPomRvHLTQC7pnXDaF2k7FQlRIXwwZRiTX1vJU//ZRseYMF6aPPCUc/3tosNoHx3G96nHCA7wI7FlaJ1zcJrD2Reif2SmTZvGyy+/XHn/j3/8I0899RQXXHABAwcOZGD/c/lu/leVLQSAwtJyDqYdYPiQgQAUFRVx44030rNnTyZOnFhlLaN77rmH5ORkevfuzeOPPw7YBfMyMjIYO3YsY8eOBexy2keP2hrZc889R58+fejTpw8vvPACYCfC9ezZkwen3sOkC87jskvHV2lhuLlnM9c3UscX43rEEx8RzG9nb+bxOVu4sGdr3rlzaKNWmUzuEM2OI3l8n3qMGwe3O62rio7uHkd0eBD/990ebn79B0ID/Xn3rqENGm31U3dxr9bkFrvYnZXPz8/rwKxfnscXU0dwWd82Z3UwcIuPDOGDKedxy7AOzLhtMFFhTdPxPbxrDKv2HGPN/uNnXboI/ttaCF9Pg8ObmvaYCX3h0r/WuvuGG27ggQce4L777OrfH330EXPnzuX+++8nMjKSo0ePMmjwUC697MrKxxSVlePvB+6vxT//+U/CwsLYtm0bGzduZODAgZVlp0+fTnR0NOXl5VxwwQVs3LiR+++/n+eee45FixYRG1u1prxmzRrefPNNVq1ahTGGoUOHMnr0aFq1alW5zPbTL7zMzybfyLsffMTdd9xW+VhjDMcLSgkO9GuStVMC/P24YXA7/rEwlclD2vHkBO9LKvticMdWGGNbC00RrOoS6O/HVee25a3l+4gOD+Lfdw39Ua1pfza4pHcCP/z2AuJ8GM13toqLCObJq5t0hX7O7xLL+z+kkVfiOus6lOG/LSA0gwEDBpCZmUlGRgZZWVm0atWKhIQEfvOb37BkyRL8/Pw4cjiDg4cO0cWZEVpYWk6Ix8qHS5Ys4f777wegX79+9Ot38p+TfPTRR7z66qu4XC4OHTrE1q1bq+yvbtmyZUycOLFy1dVJkyaxdOlSrrrqqspltiuMoc+5/dmyc3eVZT3yS1yUllcQ3oT/62DquK4M6xzD+V1iTunC0L99SwL8hDHd4+tc26mp3Hp+R3YeyeO3l/Wka0PX+1CIyFmXDjkbnO+xvLm2EE63Omryp9N1113HrFmzOHz4MDfccAPvvvsuWVlZrFmzhsDAQNp36EBhUXHlSKOSsgpCA+t/6/fu3cuzzz7L6tWradWqFbfddpvX5bN95V5m20+EqLBgDh21/4u5ZViQs5xHKQF+QkBg02USgwP8KyflnIqwoABeuzVHQ6YlAAAgAElEQVSZc87QJJ5OseG8d3fdixEq1VAxLYLpkRDBnqwCerf1Ppu+Ofn0zReR8SKyQ0RSRWSal/0dRORbEdkoIotFJMlj39Mistn5ucFj+1sisldE1js//ZvmJZ15N9xwAx988AGzZs3iuuuuIycnh/j4eAIDA1m0aBFpB+waJu5+BIMhOOjkWz9q1KjKBfI2b97Mxo0bAcjNzSU8PJyoqCiOHDlSZaG8iIgI8vLyapzLyJEj+eyzzygsLKSgoIDZs2czcuTIGuVCA/0J8BMO5xZzvKCUnUfyyC0uIzr87G3ij+0eX+/SCUqd7aaM6szdozqdlf8RsN5qqoj4Ay8DFwHpwGoRmWOM2epR7FlgpjHmbREZB/wFuEVELgcGAv2BYGCxiHxtjHHP+nnYGDOrCV9Ps+jduzd5eXkkJibSpk0bbrrpJq688kr69u1LcnIy3XvY4WrFrgrcgz1DPTpW77nnHm6//XZ69uxJz549GTRoEADnnnsuAwYMoEePHrRr147hw4dXPmbKlCmMHz+etm3bsmjRosrtAwcO5LbbbmPIELtmzl133cWAAQOqrJYKtknfIjiAUlcFaccLCQn0p0NMOJEhARzPOA1vklIKwKclW5qLVJ9BW6OAyHnAH40xlzj3HwUwxvzFo8wWYLwxJk1s9TLHGBMpIg8DIcaYJ51ybwBzjTEfichbwJcNCQjJycmm+vj8bdu20bNnT18P0SyMMWw9lEtUaCAVxq7jUtvia2f6vI7mlxDk70dkaGBly+DH8J4qpXwnImuMMcn1lfMlZZQIeC5Kk+5s87QBmOTcnghEiEiMs328iISJSCwwFvAcIjLdSTM9LyJNtyjNWUZECAnwp6SsgqLS8iqtg+YkIsRFhBAVFnTWpomUUmdOU/UePgSMFpF1wGjgIFBujJkHfAUsB94HVgDuAfmPAj2AwUA08Ii3A4vIFBFJEZGUrCzf/xPT2SY40I+isnJKXOU1liNQSqmzgS8B4SBVa/VJzrZKxpgMY8wkY8wA4DFn2wnn93RjTH9jzEXYofc7ne2HjFUCvAl4XSjeGPOqMSbZGJMcF+d99mZ9aa+zQUigf+VyEWFncUD4MbyXSqnTw5eAsBroJiKdRCQIuBGY41lARGJFxH2sR4EZznZ/J3WEiPQD+gHznPttnN8CXA1sbswLCAkJ4dixY2f9hSzEY92WsyVlVJ0xhmPHjtX4F5xKqZ+GekcZGWNcIjIVmAv4AzOMMVtE5AkgxRgzBxgD/EVEDLAEuM95eCCw1MlP5wI3G2Pcay+/KyJx2FbDeuCXjXkBSUlJpKenc7ank8orDEdyignwE3blnb0X3JCQEJKSzt5REEqp06feUUZnE2+jjH4sjDEMnr6A87vE8uLkAc19OkqpnxBfRxn9d81UPouJCG/dPqRJ/8OXUko1JQ0IZ1CfRN//KYxSSp1puvy1UkopQAOCUkophwYEpZRSgAYEpZRSDg0ISimlAA0ISimlHBoQlFJKARoQlFJKOTQgKKWUAjQgKKWUcmhAUEopBWhAUEop5dCAoJRSCtCAoJRSyuFTQBCR8SKyQ0RSRWSal/0dRORbEdkoIotFJMlj39Mistn5ucFjeycRWeUc80Pn33MqpZRqJvUGBBHxB14GLgV6AZNFpFe1Ys8CM40x/YAngL84j70cGAj0B4YCD4lIpPOYp4HnjTFdgePAnaf+cpRSSjWWLy2EIUCqMWaPMaYU+ACYUK1ML2Chc3uRx/5ewBJjjMsYUwBsBMaL/SfL44BZTrm3gasb/zKUUkqdKl8CQiKQ5nE/3dnmaQMwybk9EYgQkRhn+3gRCRORWGAs0A6IAU4YY1x1HFMppdQZ1FSdyg8Bo0VkHTAaOAiUG2PmAV8By4H3gRVAeUMOLCJTRCRFRFKysrKa6HSVUkpV50tAOIit1bslOdsqGWMyjDGTjDEDgMecbSec39ONMf2NMRcBAuwEjgEtRSSgtmN6HPtVY0yyMSY5Li6uAS9NKaVUQ/gSEFYD3ZxRQUHAjcAczwIiEisi7mM9Csxwtvs7qSNEpB/QD5hnjDHYvoZrncfcCnx+qi9GKaVU49UbEJw8/1RgLrAN+MgYs0VEnhCRq5xiY4AdIrITaA1Md7YHAktFZCvwKnCzR7/BI8CDIpKK7VN4o4lek1JKqUYQW1n/cUhOTjYpKSnNfRpKKfWjIiJrjDHJ9ZXTmcpKKaUADQhKKaUcGhCUUkoBGhCUUko5NCAopZQCNCAopZRyaEBQSikFaEBQSinl0ICglFIK0ICglFLKoQFBKaUUoAFBKaWUQwOCUkopQAOCUkophwYEpZRSgAYEpZRSDg0ISimlAB8DgoiMF5EdIpIqItO87O8gIt+KyEYRWSwiSR77nhGRLSKyTUReFBFxti92jrne+YlvupellFKqoeoNCCLiD7wMXAr0AiaLSK9qxZ4FZhpj+gFPAH9xHns+MBzoB/QBBgOjPR53kzGmv/OTeaovRimlVOP50kIYAqQaY/YYY0qBD4AJ1cr0AhY6txd57DdACBAEBAOBwJFTPWmllFJNz5eAkAikedxPd7Z52gBMcm5PBCJEJMYYswIbIA45P3ONMds8Hvemky76vTuVVJ2ITBGRFBFJycrK8uF0lVJKNUZTdSo/BIwWkXXYlNBBoFxEugI9gSRsEBknIiOdx9xkjOkLjHR+bvF2YGPMq8aYZGNMclxcXBOdrlJKqep8CQgHgXYe95OcbZWMMRnGmEnGmAHAY862E9jWwkpjTL4xJh/4GjjP2X/Q+Z0HvIdNTSmllGomvgSE1UA3EekkIkHAjcAczwIiEisi7mM9Csxwbh/AthwCRCQQ23rY5tyPdR4bCFwBbD71l6OUUqqx6g0IxhgXMBWYC2wDPjLGbBGRJ0TkKqfYGGCHiOwEWgPTne2zgN3AJmw/wwZjzBfYDua5IrIRWI9tcbzWZK9KKaVUg4kxprnPwWfJyckmJSWluU9DKaV+VERkjTEmub5yOlNZKaUUoAFBKaWUQwOCUkopQAOCUkophwYEpZRSgAYEpZRSDg0ISimlAA0ISimlHBoQlFJKARoQlFJKOTQgKKWUAjQgKKWUcmhAUEopBWhAUEop5dCAoJRSCtCAoJRSyuFTQBCR8SKyQ0RSRWSal/0dRORbEdkoIotFJMlj3zMiskVEtonIiyIizvZBIrLJOWbldqWUUs2j3oAgIv7Ay8ClQC9gsoj0qlbsWWCmMaYf8ATwF+ex5wPDgX5AH2Aw9v8qA/wTuBvo5vyMP9UXo5RSqvF8aSEMAVKNMXuMMaXAB8CEamV6AQud24s89hsgBAjC/h/lQOCIiLQBIo0xK439H54zgatP6ZUopZQ6Jb4EhEQgzeN+urPN0wZgknN7IhAhIjHGmBXYAHHI+ZlrjNnmPD69nmMqpZQ6g5qqU/khYLSIrMOmhA4C5SLSFegJJGEv+ONEZGRDDiwiU0QkRURSsrKymuh0lVJKVedLQDgItPO4n+Rsq2SMyTDGTDLGDAAec7adwLYWVhpj8o0x+cDXwHnO45PqOqbHsV81xiQbY5Lj4uJ8fFlKKaUaypeAsBroJiKdRCQIuBGY41lARGJFxH2sR4EZzu0D2JZDgIgEYlsP24wxh4BcERnmjC76OfB5E7wepZRSjVRvQDDGuICpwFxgG/CRMWaLiDwhIlc5xcYAO0RkJ9AamO5snwXsBjZh+xk2GGO+cPbdC7wOpDplvm6SV6SUUqpRxA7y+XFITk42KSkpzX0aSin1oyIia4wxyfWV05nKSimlAA0ISimlHBoQlFJKARoQlFJKOTQgKKWUAjQgKKWUcmhAUEopBWhAUEop5dCAoJRSCtCAoJRSyqEBQSmlFKABQSmllEMDglJKKUADglJKKYcGBKWUUoAGBKWUUg4NCEoppQAfA4KIjBeRHSKSKiLTvOzvICLfishGEVksIknO9rEist7jp1hErnb2vSUiez329W/al6aUUqohAuorICL+wMvARUA6sFpE5hhjtnoUexaYaYx5W0TGAX8BbjHGLAL6O8eJxv7/5Hkej3vYGDOraV6KUkqpU+FLC2EIkGqM2WOMKQU+ACZUK9MLWOjcXuRlP8C1wNfGmMLGnqxSSqnTx5eAkAikedxPd7Z52gBMcm5PBCJEJKZamRuB96ttm+6kmZ4XkWBvTy4iU0QkRURSsrKyfDhdpZRSjdFUncoPAaNFZB0wGjgIlLt3ikgboC8w1+MxjwI9gMFANPCItwMbY141xiQbY5Lj4uKa6HSVUkpVV28fAvbi3s7jfpKzrZIxJgOnhSAiLYBrjDEnPIpcD8w2xpR5POaQc7NERN7EBhWllFLNxJcWwmqgm4h0EpEgbOpnjmcBEYkVEfexHgVmVDvGZKqli5xWAyIiwNXA5oafvlJKqaZSb0AwxriAqdh0zzbgI2PMFhF5QkSucoqNAXaIyE6gNTDd/XgR6YhtYXxX7dDvisgmYBMQCzx1Sq9EKaXUKRFjTHOfg8+Sk5NNSkpKc5+GUkr9qIjIGmNMcn3ldKayUkopQAOCUkophwYEpZRSgAYEpZRSDg0ISimlAA0ISimlHBoQlFJKARoQlFJKOTQgKKWUAjQgKKWUcmhAUEopBWhAUEop5dCAoJRSCtCAoJRSyqEBQSmlFKABQSmllMOngCAi40Vkh4ikisg0L/s7iMi3IrJRRBaLSJKzfayIrPf4KRaRq519nURklXPMD51/z6mUUqqZ1BsQRMQfeBm4FOgFTBaRXtWKPQvMNMb0A54A/gJgjFlkjOlvjOkPjAMKgXnOY54GnjfGdAWOA3c2wetRSinVSL60EIYAqcaYPcaYUuADYEK1Mr2Ahc7tRV72A1wLfG2MKRQRwQaIWc6+t4GrG3rySimlmo4vASERSPO4n+5s87QBmOTcnghEiEhMtTI3Au87t2OAE8YYVx3HVEopdQY1VafyQ8BoEVkHjAYOAuXunSLSBugLzG3ogUVkioikiEhKVlZWE52uUkqp6nwJCAeBdh73k5xtlYwxGcaYScaYAcBjzrYTHkWuB2YbY8qc+8eAliISUNsxPY79qjEm2RiTHBcX58PpKqWUagxfAsJqoJszKigIm/qZ41lARGJFxH2sR4EZ1Y4xmZPpIowxBtvXcK2z6Vbg84afvlLqv86u+fBcb0h5s7nP5Cen3oDg5PmnYtM924CPjDFbROQJEbnKKTYG2CEiO4HWwHT340WkI7aF8V21Qz8CPCgiqdg+hTdO6ZUopX7cXKUw9zF491rITYctnzb3Gf3kiK2s/zgkJyeblJSU5j4NpVRTKy+Dty6HtFUw+C6ocMGGD2HaAQg4hSlKrhI4kQatOoB/YNOd74+MiKwxxiTXV05nKiulmt+exTYYXP6/9qfrheAqgoy1p3bceb+DlwbBn9vCv0ba+2eqEvwjqmy7aUBQSjWeqxRyvI4HaZhNH0NIFAy4xd5vf779vW9p449ZdBzW/Rs6j4WhvwS/AFj+D8jec+rnu/EjOHGg7jLv3wifTz315zqDNCAodSpcJc3zvBnr4R+DIHN77WUKjsH8P0BO+uk7j0XT4eUhUJLX+GOUFsL2/0CvCRAQbLeFx0B8b9j3feOPu3YmlBXCxU/anwkv2+1pPzT+mADH98Ond8Mnd0FFhfcypQWQugAOrDy15zrDNCAo1VhHtsIzXWDDB74/Zu8Se6EqOn5qz735EziWCrN/YfPv1WVug9fGwvd/b9j5NYSrFNa9A6X5sKf6mJEG2PmNPUbf66pu7zjcppG8vb76lLtg1avQcSQk9LXb4npAcCSkn2JASJ1vf6etgg3veS+T9oPtBzm+t3Hn30w0IKjm5yqxF8kvHoDXL4SnO8KW2c19VnUrK4JZd0BpHuxf7ttjjIHP74M5v4Jnu8NHt8KhDY17/t2LICwWDq2Hpc9V3bfjG/s+uoptGuZYau3HKcyGbV9A9t6Gn8OOr6DwmL3tvkj6onqratMsiGgDHYZX3d5huK3hZ6xr+Llt/9KOVBr6y5Pb/PwgcRCkrW748TztWgAt20O7YbYFVphds8y+ZfZ3hcu2KH4kNCCo5nVgJfxrhL1IbvkU/IMhLAa+fBAKjjb32dVu7mOQtQ1aJMCRzb49JnuPzTsPuxeSb7cdqbPuaPhz52fCkU1w3r22Vr3kGXvRPHEAPrkb3r8BYrrA3YugzblwdFfNY6x+HV4bB890hg9vhveuh7Lihp3H2pkQmQTdL7NzB3zpRD2yBf7aAZb8zd4vOg675kGfa8DPv2pZd4BwX1wbYuU/oWUH6H5p1e3thkDmlsanuFwltpXX7WK4/Fl7/gufrFlu//cQGGZv1xWQzzIaEFTzKHfBVw/DjPG2tn3TLHhkP9z+H7jh3/YLO/e3zX2W3m37ElLegPN/Bb0n2vRMRXn9j9vtrP84+C649GkY/mt7sfBWw6zLnsX2d5dxcNnfIDwO3rsR/pEM2+bAiAfh9m8gKhFiz7EBwfNiXVYMX/0/KM6F0Y/AZc/C0Z2w9H99P4cTafb1DLgJzrkEcg/a96E+m2bZ0UMLn7Itm61zoKIM+l5bs2yLOJvmaWhAOLgW0lbC0F/UDDJJQ8BU2DKNsX85lBXYgJDQF4b8wk6g8zxeaSEcXGODHMAxLwH5LKUBQTWNzG2w/v36y7lt/xJ+eBUG3wn3roRuF4GI3RffE0Y+CBs/tM3zs8mx3Tbt06Y/jPsDJPSxaQ1fUi67F9laa3Rnez9xkP3d0KGVuxdCaDQknAuhrWDCS7am2uca+NUauPBxCHJqpzHdoCQHCjzWAcvaBqYcxv0Oxj4KQ+6GfjfCsudsDd4X69+1v/vfBF0vsrfrSxsZYwNWx5G2ZfPtn+xPTFf7fnrTwd2P4PK+35u1b0NgOAy4uea+JOc9b2w/wq75thXbcaS9P/a3EB5btZWQvhrKS6HnVfbv5K2FdpbSgNBcSgsa3kRvbsbYi9rBNTX3Lf6rvVC6Sn071u5vITgKxj8NwS1q7h/5P7Z2++VvoCT/1M67LhUVvtfQ8w7DOxNtrfO6N+2Eqda97b760kblZTbV0GXcycDXtj8gcLABOXL336DzGJsTBztm/7FDMPGfEJVUtXxsV/vb86Lkvui7O1sBLvmz7W+Yc3/9rZ2Kcmc45xg74Ssq0Y4I2lVPQMjaYVtEva+Gq/9lA1jhMehz7cn3pLqOI2yHc0P6WvYstucWElVzX2griO3e+H6E1Pm2s9sdcEMibT/F7oVw2PkM7P8exA/aD4PYbpoyUj7497X2Yne2Wvdv26zf8pm9mGz4AP45HN65Gj64uWoKoqIC9n5na50nfOhAMwZ2L4ZOI8E/wHuZgGC46h+QcwDWnMY1bdbNhOd7Q96RussV59i/WcFRuOnjk7X8uJ72y19fQEhPsR3QXcad3BYSZS8Y3gJsbTK3Qf7hqseBmqkRt5hu9vfRnSe3Hd5s89utOp7cFh5jg/PBFFjtZRWZvUttiu+7v9nPRU4aDLzl5P5uF8KBFTYNVZttXwACPa6wf/eJr8I1b8Dw+2t/jLsfYd5jdkz/51PtDOba+itOpMHxfTaQ1CZpsK3FN3Ti2PF99n3sdnHV7cl32PdzhTOsdd8y23cTEmnf/8YGhNqGtJ5GGhCaQ3mZ/eKlrTp9z5G103beHfaxw9NT5nbbybvkb/DxrfBSsh3eaCpsaiEvAw5vOln+8MaTwyh9+fAf220v9NUvatW1H2YvWqc6brwuqQtsymdrHWsrlpfB+z+zqZYb3jmZ6gEIDLFf+vpSLXsW2cDRaVTV7YmDbEDwvDh9/yK8OND7HAd3P0SXsXU/n1tUOwgIqfp3ObIZ4nvVDCJ9r7U59rUzax5n4VO2I3rRUza1FB5vL+xu3S62I2r21jH8dNsc26kbkWDv+wfY5wwKr/0xEa3t/ITj+yH1WzuyafYUWzHxlqZz9zd0Gln7MdsNhqLshk9Qc7eA3Ckyt7BoO6Fu08f2mOkpJwNZTBfIP1J3oKzOGFg4HaYn2H6hj26138X6Ki1NQANCc8jeY3OM2Xtsh2pD5Wfai2RdNZxtn0PxicblShf8EYJawIPbYMpiO6Hn5k/g3hV2gg/ALo9/beHu5AR7sa9PQy5qbQfYSVjVrXnr1Ma+g33/9q+wt+taSG3vEti/zFlS4YKa+xP61B94dy+0F//QllW3tx0IBZm2U9Z9Tmvfhuzddq6Bt+PEnlMzNVQbPz+bo3enjIyxAcGd6vIkYvtyjmyumkZzd5KeNxV+lwkPbIb7Vp2cRAbQbqgd479rXs3jgq1dH94IPa/07bw9XT8T/meb/XkoFS5/DtLXwCvn2daCp31Lbd4+3svrc0saYn83tKKRugBadbIX+eqG3WNbyLN/CeUlJ1sosU4LzdeOZWPsZL8lz9jvR1x3O4Js4VN2GPFppgGhqZw44HsTtLI2aao25etzdJfN8T7fB964CP49yX7RvHHXZny5QHva9z3s/BpGPACRbe0FecDNNk8tAi3i7UVsZ7WAEN/LfhF9aSHsXmhr/u60S13a9LetiYJjJ7e5R8l8fFvdQ1MPb7K1ytoc2w2FR+15HFhR+xIM7nHwvWr5L6+te9tzLM7xvr/ouL2gemsRuVsb7rRR5lb7HoqfHTpZfXTQ/uV2KYaGiOl68nOWd8ieT+s+3st2GA4Y+364pf9gRwJ1HGmDQMt2tlbsyT8QOo+2nztv4+63fWl/e7YqGsPPzw5EuG+VDcTfPFK1L27vUpvj96vj0taYCWplRc5w04u893dEd7KdyGmrAIH259nt7pSd5/ewJL/2z+3iv9rWwMCfw43vw43vwgMb7SJ/Ldv7fr6NpAGhKRxNhb/3h++e8a285/A8X4bqHd9vp8m/NNjm8vv/DC560tZwXjkPVv6r6oWjMNvmSKFhAcEYO9Emoi0Mvaf2cudcYpvFBUftl/HACtuJF9Ol/oBQXmZrcfWli9zaDrC/D3l0vGastbWwomz45lHvj8s9BG9fZZdS3rvEe5kDzoSyS/5sf2/9zHu5jHU2aFSv3bu1djpna0sb7V1q023eXnNCH/ALPDlscevnNhiM+a2tUXtOetu3zA7Z9PW9c4vtZvt2XCUnWzIJtQSExEE2xeS5ZMQ+j07SuvS/2Xa8/70fvHmZbcXlHrL7tn1h36foTg0799pEJdpRUkXHbSoKbOUo5wB0HFXnQxs1QW3blza12OPy2suc/yv7O6Hvyc9KdCf73nl26s+ZaufeVB8skTIDvvurrYBd8feqQS0kqvaO9yakAcGtvKz+xapqs+lj21xc+r++5SUzt9oasl+gvV2b4lxntcZk+4Ua/mv4zWa48gXbEXffKluj++aRqouA7VlkL0CRiTb14Kutn9u+jbG/PTmKwptuFwPGNqHTVtmmbOcxtiZa3+tPX21Hjfhay21zrv3tmTba71yshvwCNn1Uc3RLRblda8ZVbN/nWXdAbkbNYx9YaSfBnTMeEvrB5lrSRoc2nAxM3lSONPISEIyxaZSgiKp9D24Bwfbi7G4hbPnM/k3Pu8+OiFn5it2enwlfPmBn9NaVH/cm9hz7ecjee7LzO76X97KBIbbT1fPztP97+/6ERNb9PN3H29rsuN/b8/3i1/BcD1tpSVvVuHRRXTqOsoE6xfl/XO7+g7o6lN3cE9SKTtRfFuzgg5Yd6g42ScmQfKcdxusWEGxr9u6UUWG2DS55h052Qru3L/gjdBoNV/6j7hbOaaQBwW3Rn+HloXY4aEMYYwNCQj/bbP76kfpTR5nbbPnYc+penGzOVFj+kh2W96s1cNGfbMrGLSrJdnKGxdhWgtuu+fZi0nuirTX5MmkKYPFf7KiZ/j+ru1yb/tCitV2DZs9iu4pkh/NtCyH3YN3v4e6F3jtXaxPa0n7pPZcv2L/c5ogvftIOIaw+NPX7F+wF7dJnYPIHtrn/0a01h8TuX26b9iLQZ5INhtXTHflZdkRNXQEhsq19vz072vcutSNinu9j1/vpdlHt6/EnDrIB78hWOLrDdqIGhcGg22wnatZOO5u44ChMfr/uTlhvYtxDT3faoBXVvvbWDtiAdHiTvViWFdvWoC8XWbAXv1EPwdTV8Mvv4aIn7Gc2oo33yWenws/PvkcHVtjv1N6ldjmP+J71P7b7pTZIrn27/rLZe2wrc8At9V+or3jOpns8eY402jLbpt9a94XlL9rPF9jrT0menbDYTMEAfAwIIjJeRHaISKqITPOyv4OIfCsiG0VksYgkeexrLyLzRGSbiGx1/oMaIvKWiOwVkfXOTy0zU86A8jL7pS0rtF/KhshYZ2vhQ+6GMY/a2uD2/9RevqzIfsDie0F8j9pTRoc32xr7qIe9jy93CwyFQbfbC0f2HjtULXUBdLnABpzyUntBq0/RccjaDufeUPsQRjc/P3uBS11ox2UnDYbgCIh2Ots8WwnlLntxcQel3YsgMbnuC1J1nh3L5S44sMoGoMqhqenw5nj4epodobNwOvSeZJvecd3txK30H2D+708eM++wXXjMnQbpPdH+rr6G0iHneWubOAU2oLTuc7KFkLYaZk6ws3ATB8IVL8CVf6/j9Q20Q1KX/A2QkzXpwXfb+zMusTXsif+qOzDVxh0Qju2qvUPZU8cR2H6Elc4kqxLfA4KbiG35DP81/Pxz2yHsrTP2VPW/CfyD7GzhfUvtefqSWmk7wLZqV7xc/3ygde/aSkx9FaXaxHazqduKCjvZMq6nncdSVgTfPW2vASkz7PBVX4LZaVRvQBARf+Bl4FKgFzBZRKq3N58FZhpj+gFPAH/x2DcT+JsxpicwBMj02PewMaa/8+NlKMkZsvObkzM5DzdwsbFNs+wHsueVdqp8fC/4ZlrtteSs7YCxf/j4njbn6W1dlSXP2DTDeffWfw6D77IX8R9es+dfkGUv2O4voC/9CO5AWFtnY3XdLrEzYA9vsl8s8LjweDzf2gFsiakAABIBSURBVLdsvvSFvjb9lbHW9yGTbm0H2IXK8rPs6ysrsAEBoP1QO/rHP9jW9ub/3uaXr3j+5IWh90SbXlr1Lzs6BU52mrrX3W/V0dbUq4/scQcid+qqNq172/Rf0XH45E57Dg9stC245NvrTre4U0lbPrUtFvewzKhE21ooyoaxv7MTuhojJNLW0I9ssbns2voP3JKS7Wd631InPefRSXq2CY+1nblr37at04ak00Y8aIeEumdde1Pusvu7XmT/Ho0R08VWNvd/bwP7uTfYIDHoNjvH5rN77OTMMc2/VIsvLYQhQKoxZo8xphT4AJhQrUwvwBlLyCL3fidwBBhj5gMYY/KNMYVNcuZNae1M+4UJaQmHNlbdl7UD3r7Se66xotxeQLpdbFMG/oH24pSTBite8f5c7hZB694n87hZO6qWObLVtg6G/sIetz6RbexFb+07Th5cbAvBW429Nu7abX21R7cuY20fCJwMCO5RQ54dy7vm2/e2dR/7npgKO2KpIdy180PrT3ayugMC2FEnd38Lj6bD1DVw9+KaLZALfm/Hzn8z7eRw08AwaNPvZJnek2xHbpbHyK+MdbbJX1/+vLWzhMX7k22L5Zo3fG8FxXazw3yh5kV//F/tBK5RD/l2rNrEdIWd82xfV31/48BQ24rb/73Ny3t2kp6Nkm8/OSSzvg5lT51G2WD8/d9rXxojdYHN93tOwmso90ijRX8GBPpeb++PmWYrMhnrYPQ0OzmwmfkSEBIBz5xDurPN0wZgknN7IhAhIjHAOcAJEflURNaJyN+cFofbdCfN9LyIBNMccv5/e+ceZFV9H/DPd3dhBXdFWJY38nJRCK+FFVAxMGAnYlJfTVsTHKGxQ+voxGpsE8dO2jJJncZOmiY1D+Oj6hhTRcegaRMNwbEJBkVAHssjmIjy2t2kYGi0PJZv//j+Dvdw99699+7efd3z/czs3HvOuefc89vfub/v7/d9HrBOn7XMBofDaQJh+7OmP4z72ke88zOLGo3rRsddZkbKX3wz8yqhudEegsETzP0N2qqNXv2KDRCX3pZ/O+bdamqH1x6wGXVVrc00+52bnyto8w4TPtUj8/u+yuoQwl+VmuFWVtn50Qqh9aT9jy66GpY9DZ/bBTeH4KRCOGNY3mwD+ZBJqVl0nLJyS9WQ6YdVWQ1Lvmiqo+3P2gphTMPZev3pfwxSburDiENbQoqJHESD7LuvmVG+kDaWladUQemG1+rhNqPsrIfJ0Dp7PiDlFdUe4xeYMf291wtXF3U34y439WjV8JTffz6IWIqUo/uyp1vf/IQlD5x8VcfvL7qnd9fbCiZaaVQNg499yVYfcUN0D1Is68XdwEIR2QwsBA4ArUAFcEU4fgkwEVgRzrkHuDjsHwJ8PtOFRWSliGwUkY0tLS2ZPtI5tnzPZq31N5mht6nx7IIW0Yw07pcdse0ZGxDTH5YFd9kyP1PEZ/NOqJ1sUZqDx0PFgDQ31F3maTJ3ZVtf7/YYM8eCg7TV1EVgD3zNxDxVRjvMUFvIwLP0fvOTjg+qNRemPJsOvBk8ihbZdtUw81UvlHPOs+se2GQ/qvjqoBBmLbM+fulvTZd+Qdp1qoebsfGtp+wZONZkaoh89PbDpqSSni3oQEqSuSvhirvNQN0VRLPUigH5uX6Ov9x+Fx2xH3Q3IrYi++SjhQvOyUttYvazr7ZNFXGsCXb/F8z8VHaHgHyoHmkTM7BI/zgNn4GbVnfu+kUkH4FwABgb2x4T9p1BVQ+q6g2qWg/cG/YdxVYTW4K66RTwPDA7HD+kxnHgUUw11QZVfVBVG1S1oba2tsDm5eD0aZsBTPio/UhGzrQfQBTEc+p4yp8/XSCcOm5Gwyl/aEvsOBfMs1nL+m+09WxpakypisrKTTi0xATCq18xVcalHajFetlnsVwxMV/pIZNyu56ePm33la+6KKJ2cmqwj4jHIvzqFbufQt0kMzGq3hLifXikbSGVfCkrMy+OY4dssMvkVz/7ZrPB7PlRyqCcj0DoNwBueQlu/F5uo3wmpl5jaq2uYuhkex02Jb/7GzM3qAR7sf0gzsgZJsQKpazMBHhzY9tsrZseswnW7OWduzcR+11UDLB+7sXkIxDeAOpEZIKI9AduBNbEPyAiQ0UkutY9wCOxc88XkWgkXww0hnNGhlcBrgM6kHSnk7zzqi0X64Ob2IigT47cBw9uNt3k8Gm2L56P5O11ZlSNcp6ns+Aum11ujYXWf3jE8gDFfcCHTU2tEA5vN3XGvL/omD5xyifg7j1nG0BrJpkrZXtl/I7uM0NtoQIhE0MmWQbLD4+YQBhVn58dJBej6s1jCjq+QojOnXqdzebHXNL2+KQlNqPb9EQwKEvquch5j7Ny2xp6iijraS6DckT/gbbiHDmzsJVqX2TaH1nMzvpvpPa1njTPn0lLUv+7zjD/Vrjy70112YvJKRDCzP524MfATuBpVd0hIqtEJBJ3i4DdIrIHGA58OZzbiqmL1orINkCA74Zzngz7tgFDgS8VrVXpHH0vc/rcnS+YyifS2w6tMykeGZajAKgFd9qMMlotROdWDrJAkkxcuMQGkp9/LeVyGcUcxAVC7cWpdALr/tGu2V72x1zE4xTABmhtbT/orlCDcntEnkaH3rL/18RFnb8mpAzL543pfAj/dd+Elesyp90urzDV0t6XzZV36OTMn+trDLrA1CPZ0m9k4pMP24qn1CnvZwP2O/+dihjf9UP7XRZLtz/r0zD/L3N/rofJy4agqv+pqpNVdZKqRoP9F1V1TXi/WlXrwmf+PKiBonNfVtUZqjpdVVcETyVUdXHYN01Vb1LVrkt6v+Z2ywGUzuHt5kHR7xzbLiu3QTEyLO9bbz7Dkz9mxsZIbdR6Cnb/0CIzK/pn/k4RK/Ly270pI2UUlRz3NY6Ew+Yn7ZqX3V6cGXVEJlfQdJp2AJIychfj+zY9btkvJy7q/DUheAMJjLu08wbW/ue2L/zqb7IJwKEtHfP7742UlcGnv585OV82qkd03NWyrzF7ueU3eu3fbPuNh0yIpqe6LnGSEak8YrqpZeJqE1Xblx7CH3kaxQOgKqvtGlFmzH0/txl9rlD8KdfYsvuFOyyC+eBme+jiQWbDwiC8dpVFHM9vJ4dQR4hiEdqzIzRtNxtKMWbCg8dbEE/jGsuJM3Ze568J1gfXf9sC9bqaIRNSkdT5eBg5fZ9zzoM5y82hY89Ltlq45DMdswf1YRIiEGYEY3EswdT7+80GMHxq28/+3/tW4vHEsZS+etxlltrg1AlTF1UMMP1ie5SVw/IXrKj6hm/bSmHYlLNnuIPGmtqq9bippoqtYxxYY2qo9lYIzR0wKGejor+pdE6fNGNktPoqBjNvtMjj7mDOn9lroS6yTt9l3q3223xmhdmY6m/OeUqpkRCBEPyu47lmzqhv0gbCKFBpQ8gNdCYidr4ZmA9uNmFRd2X7CeAiKirhqvssp86AIW1d+KK0B1UjLOK42JxxPc0Si3DiAxMW7eWPL5RIbTRxUfGu2d185Hq47fXMCemc0mTQaMsbdvL3ZmjuBYFi3U2W+oUlRk2dSfzDWy3IBzLr88FUSJG9YPCElF945Hq3/utmbJpSoPvYRUvhr98GMiS+u/YBM/ymu68WiyGTsud+b9lp91SsFUL0ffykbwsEke5bjTi9hwV3WnqJQoJCS4hkCITyClMNxVcITY3msZIekt9vgHmWtOw8272xapgNdLteNP/sjhibsmUxLIZbW3vUXGh5ck4dP7vKFcRyGBVRIHzkelO75euu6Ti9hWEXwx09l1atp0mGyghscDq8LZWaurmxrf0gIlIbpfu7jwurhIkLe3dul3RqJpnXTKbqak07QsH1IhUuAfs/3fCdHk3j6zhO4STnFztiuqWT+N1B8zZq2Z29SMjoOZiLY1rkY5TqoNiFPrqaKMndK/e1rcXctD1ErybnUXAcJzPJUBnB2VHIg8eFIhVZ1CSzl1u2x/ScL1OvMePstCIX+uhqRs2ynClbnrIkXoMnWF1csOCxGX/Ss/fnOE6vIDkCYfhUQMywfCLEwLVbRjCDd0llNVz5d112i11GWbnVB/iDVRYf0PgDOB7ScIyeAzP+tGfvz3GcXkFyBEJlteXrP7zV3EfLKlIJv5JCZTXUL7M/x3GcNJIjEMDsCIe2WBRyTV32tBOO4zgJJFmWxBHTzdNm/xvZPYwcx3ESSsIEQjAsf/Cb7PYDx3GchJIwgRArHegCwXEc5yySJRCqR8DAofbeVUaO4zhnkSyBIGKrhP5VluvccRzHOUOyvIzAklcd+bVH5jqO46SR16goIleJyG4R2SsiX8hwfJyIrBWRrSLyioiMiR27QEReEpGdItIoIuPD/gkisiFc8z9CveauZ+JCmLOiW77KcRynL5FTIIhIOfAAsBSYCnxKRNIV8P8MPK6qM4BVwH2xY48D96vqFGAu0Bz2/xPwL6p6IXAEuKUzDXEcx3E6Rz4rhLnAXlX9VaiH/H3g2rTPTAV+Gt6vi44HwVGhqi8DqOr/quoHIiLAYmB1OOcxoIDq347jOE6xyUcgjAbei23vD/vivAXcEN5fD1SLSA0wGTgqIs+JyGYRuT+sOGqAo6p6qp1rAiAiK0Vko4hsbGlpya9VjuM4TsEUy7J6N7BQRDYDC4EDQCtmtL4iHL8EmAisKOTCqvqgqjaoakNtbW2RbtdxHMdJJx+BcAAYG9seE/adQVUPquoNqloP3Bv2HcVm/luCuukU8DwwG/gtcL6IVGS7puM4jtO95CMQ3gDqgldQf+BGYE38AyIyVESia90DPBI793wRiab2i4FGVVXM1hAVFlgO/KDjzXAcx3E6S06BEGb2twM/BnYCT6vqDhFZJSJRpflFwG4R2QMMB74czm3F1EVrRWQbIMB3wzmfB+4Skb2YTeHhorXKcRzHKRjReDnFXk5DQ4Nu3Lixp2/DcRynTyEib6pqQ87P9SWBICItwL4Onj4U+E0Rb6evkMR2J7HNkMx2e5vzY5yq5vTK6VMCoTOIyMZ8JGSpkcR2J7HNkMx2e5uLiyf0cRzHcQAXCI7jOE4gSQLhwZ6+gR4iie1OYpshme32NheRxNgQHMdxnPZJ0grBcRzHaYdECIRc9RxKAREZKyLrQs2JHSJyR9g/REReFpFfhtfBPX2vxUZEykPyxBfDds/U2uhGROR8EVktIrtCrZFLS72vReTO8GxvF5GnROScUuxrEXlERJpFZHtsX8a+FeProf1bRWR2Z7675AVCnvUcSoFTwOdUdSowH7gttPMLwFpVrQPWhu1S4w4sij4iCbU2/hX4kapeDMzE2l+yfS0io4HPAg2qOg0ox9LolGJf/ztwVdq+bH27FKgLfyuBb3Xmi0teIJBfPYc+j6oeUtVN4f0xbIAYjbX1sfCxkqs7EarzfRx4KGyXfK0NERkEfJSQ7kVVT4RkkiXd11j25AEhKeZA4BAl2Neq+irwP2m7s/XttVhxMlXVX2C540Z29LuTIBDyqedQUoQypfXABmC4qh4Khw5juaZKia8BfwOcDtt519row0wAWoBHg6rsIRE5lxLua1U9gFVmfBcTBO8Db1L6fR2RrW+LOr4lQSAkChGpAp4F/kpVfxc/FrLMloxbmYh8AmhW1Td7+l66mQosjfy3Qsr535OmHirBvh6MzYYnAKOAc2mrVkkEXdm3SRAIOes5lAoi0g8TBk+q6nNhd1O0hAyvzdnO74NcDlwjIu9gqsDFmG691Gtt7Af2q+qGsL0aExCl3NdXAr9W1RZVPQk8h/V/qfd1RLa+Ler4lgSBkLOeQykQdOcPAztV9auxQ2uwehNQYnUnVPUeVR2jquOxfv2pqi6jxGttqOph4D0RuSjsWgI0UsJ9jamK5ovIwPCsR20u6b6Oka1v1wA3B2+j+cD7MdVS4ahqyf8BVwN7gLeBe3v6frqojQuwZeRWYEv4uxrTqa8Ffgn8BBjS0/faRe1fBLwY3k8EXgf2As8AlT19f13Q3lnAxtDfzwODS72vgX8AdgHbgSeAylLsa+ApzE5yElsN3pKtb7EaMw+EsW0b5oXV4e/2SGXHcRwHSIbKyHEcx8kDFwiO4zgO4ALBcRzHCbhAcBzHcQAXCI7jOE7ABYLjOI4DuEBwHMdxAi4QHMdxHAD+HyefkMw4Z8bWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(history.history['accuracy'])\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.legend(['training', 'validation'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. HE 权重初始值\n",
    "\n",
    "将各层的默认初始化器，统一修改为 HE 权重初始值，重新训练网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 64)                50240     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 59,210\n",
      "Trainable params: 59,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "weight_model(\"he_normal\")\n",
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10.1 模型训练\n",
    "\n",
    "> **预计训练时间 5 分钟，请耐心等待**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "165/165 [==============================] - 4s 15ms/step - loss: 0.0059 - accuracy: 0.9988 - val_loss: 0.6589 - val_accuracy: 0.9657\n",
      "Epoch 2/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 0.0072 - accuracy: 0.9987 - val_loss: 0.6853 - val_accuracy: 0.9642\n",
      "Epoch 3/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 0.0141 - accuracy: 0.9978 - val_loss: 0.6494 - val_accuracy: 0.9645\n",
      "Epoch 4/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 0.0172 - accuracy: 0.9973 - val_loss: 0.5838 - val_accuracy: 0.9659\n",
      "Epoch 5/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 0.0068 - accuracy: 0.9984 - val_loss: 0.5391 - val_accuracy: 0.9671\n",
      "Epoch 6/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 0.0030 - accuracy: 0.9993 - val_loss: 0.6037 - val_accuracy: 0.9679\n",
      "Epoch 7/100\n",
      "165/165 [==============================] - 1s 7ms/step - loss: 0.0034 - accuracy: 0.9993 - val_loss: 0.5534 - val_accuracy: 0.9691\n",
      "Epoch 8/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 8.0923e-05 - accuracy: 1.0000 - val_loss: 0.5588 - val_accuracy: 0.9689\n",
      "Epoch 9/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 3.0670e-05 - accuracy: 1.0000 - val_loss: 0.5600 - val_accuracy: 0.9694\n",
      "Epoch 10/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.6667e-05 - accuracy: 1.0000 - val_loss: 0.5601 - val_accuracy: 0.9694\n",
      "Epoch 11/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 1.2818e-05 - accuracy: 1.0000 - val_loss: 0.5614 - val_accuracy: 0.9695\n",
      "Epoch 12/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.0839e-05 - accuracy: 1.0000 - val_loss: 0.5626 - val_accuracy: 0.9696\n",
      "Epoch 13/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 9.0117e-06 - accuracy: 1.0000 - val_loss: 0.5638 - val_accuracy: 0.9698\n",
      "Epoch 14/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 7.8695e-06 - accuracy: 1.0000 - val_loss: 0.5649 - val_accuracy: 0.9698\n",
      "Epoch 15/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 6.9659e-06 - accuracy: 1.0000 - val_loss: 0.5661 - val_accuracy: 0.9698\n",
      "Epoch 16/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 6.2001e-06 - accuracy: 1.0000 - val_loss: 0.5672 - val_accuracy: 0.9698\n",
      "Epoch 17/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 5.5604e-06 - accuracy: 1.0000 - val_loss: 0.5683 - val_accuracy: 0.9697\n",
      "Epoch 18/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 5.0214e-06 - accuracy: 1.0000 - val_loss: 0.5695 - val_accuracy: 0.9697\n",
      "Epoch 19/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 4.5244e-06 - accuracy: 1.0000 - val_loss: 0.5706 - val_accuracy: 0.9697\n",
      "Epoch 20/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 4.1049e-06 - accuracy: 1.0000 - val_loss: 0.5717 - val_accuracy: 0.9695\n",
      "Epoch 21/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 3.7365e-06 - accuracy: 1.0000 - val_loss: 0.5730 - val_accuracy: 0.9694\n",
      "Epoch 22/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 3.3799e-06 - accuracy: 1.0000 - val_loss: 0.5742 - val_accuracy: 0.9694\n",
      "Epoch 23/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 3.0784e-06 - accuracy: 1.0000 - val_loss: 0.5754 - val_accuracy: 0.9695\n",
      "Epoch 24/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 2.8105e-06 - accuracy: 1.0000 - val_loss: 0.5766 - val_accuracy: 0.9695\n",
      "Epoch 25/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 2.5664e-06 - accuracy: 1.0000 - val_loss: 0.5780 - val_accuracy: 0.9696\n",
      "Epoch 26/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 2.3420e-06 - accuracy: 1.0000 - val_loss: 0.5790 - val_accuracy: 0.9696\n",
      "Epoch 27/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 2.1422e-06 - accuracy: 1.0000 - val_loss: 0.5805 - val_accuracy: 0.9696\n",
      "Epoch 28/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.9694e-06 - accuracy: 1.0000 - val_loss: 0.5816 - val_accuracy: 0.9695\n",
      "Epoch 29/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.8072e-06 - accuracy: 1.0000 - val_loss: 0.5830 - val_accuracy: 0.9695\n",
      "Epoch 30/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 1.6589e-06 - accuracy: 1.0000 - val_loss: 0.5841 - val_accuracy: 0.9696\n",
      "Epoch 31/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.5228e-06 - accuracy: 1.0000 - val_loss: 0.5855 - val_accuracy: 0.9696\n",
      "Epoch 32/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.3907e-06 - accuracy: 1.0000 - val_loss: 0.5869 - val_accuracy: 0.9694\n",
      "Epoch 33/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.2752e-06 - accuracy: 1.0000 - val_loss: 0.5882 - val_accuracy: 0.9695\n",
      "Epoch 34/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 1.1706e-06 - accuracy: 1.0000 - val_loss: 0.5897 - val_accuracy: 0.9697\n",
      "Epoch 35/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 1.0731e-06 - accuracy: 1.0000 - val_loss: 0.5911 - val_accuracy: 0.9697\n",
      "Epoch 36/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 9.8248e-07 - accuracy: 1.0000 - val_loss: 0.5925 - val_accuracy: 0.9696\n",
      "Epoch 37/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 9.0448e-07 - accuracy: 1.0000 - val_loss: 0.5939 - val_accuracy: 0.9697\n",
      "Epoch 38/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 8.3234e-07 - accuracy: 1.0000 - val_loss: 0.5952 - val_accuracy: 0.9696\n",
      "Epoch 39/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 7.6772e-07 - accuracy: 1.0000 - val_loss: 0.5965 - val_accuracy: 0.9697\n",
      "Epoch 40/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 7.0755e-07 - accuracy: 1.0000 - val_loss: 0.5978 - val_accuracy: 0.9697\n",
      "Epoch 41/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 6.5337e-07 - accuracy: 1.0000 - val_loss: 0.5993 - val_accuracy: 0.9697\n",
      "Epoch 42/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 5.9886e-07 - accuracy: 1.0000 - val_loss: 0.6013 - val_accuracy: 0.9697\n",
      "Epoch 43/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 5.2963e-07 - accuracy: 1.0000 - val_loss: 0.6026 - val_accuracy: 0.9698\n",
      "Epoch 44/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 4.8922e-07 - accuracy: 1.0000 - val_loss: 0.6039 - val_accuracy: 0.9698\n",
      "Epoch 45/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 4.5076e-07 - accuracy: 1.0000 - val_loss: 0.6053 - val_accuracy: 0.9699\n",
      "Epoch 46/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 4.1727e-07 - accuracy: 1.0000 - val_loss: 0.6068 - val_accuracy: 0.9699\n",
      "Epoch 47/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 3.8663e-07 - accuracy: 1.0000 - val_loss: 0.6081 - val_accuracy: 0.9699\n",
      "Epoch 48/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 3.5748e-07 - accuracy: 1.0000 - val_loss: 0.6094 - val_accuracy: 0.9700\n",
      "Epoch 49/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 3.3050e-07 - accuracy: 1.0000 - val_loss: 0.6107 - val_accuracy: 0.9701\n",
      "Epoch 50/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 3.0611e-07 - accuracy: 1.0000 - val_loss: 0.6121 - val_accuracy: 0.9701\n",
      "Epoch 51/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.8286e-07 - accuracy: 1.0000 - val_loss: 0.6137 - val_accuracy: 0.9701\n",
      "Epoch 52/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.6126e-07 - accuracy: 1.0000 - val_loss: 0.6151 - val_accuracy: 0.9701\n",
      "Epoch 53/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 2.4245e-07 - accuracy: 1.0000 - val_loss: 0.6166 - val_accuracy: 0.9701\n",
      "Epoch 54/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 2.2388e-07 - accuracy: 1.0000 - val_loss: 0.6180 - val_accuracy: 0.9701\n",
      "Epoch 55/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 2.0734e-07 - accuracy: 1.0000 - val_loss: 0.6195 - val_accuracy: 0.9701\n",
      "Epoch 56/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "165/165 [==============================] - 2s 10ms/step - loss: 1.9160e-07 - accuracy: 1.0000 - val_loss: 0.6210 - val_accuracy: 0.9701\n",
      "Epoch 57/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.7687e-07 - accuracy: 1.0000 - val_loss: 0.6225 - val_accuracy: 0.9701\n",
      "Epoch 58/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 1.6331e-07 - accuracy: 1.0000 - val_loss: 0.6239 - val_accuracy: 0.9701\n",
      "Epoch 59/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 1.5131e-07 - accuracy: 1.0000 - val_loss: 0.6254 - val_accuracy: 0.9701\n",
      "Epoch 60/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.3928e-07 - accuracy: 1.0000 - val_loss: 0.6270 - val_accuracy: 0.9700\n",
      "Epoch 61/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.2859e-07 - accuracy: 1.0000 - val_loss: 0.6284 - val_accuracy: 0.9700\n",
      "Epoch 62/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.1883e-07 - accuracy: 1.0000 - val_loss: 0.6300 - val_accuracy: 0.9701\n",
      "Epoch 63/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 1.0952e-07 - accuracy: 1.0000 - val_loss: 0.6313 - val_accuracy: 0.9701\n",
      "Epoch 64/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.0091e-07 - accuracy: 1.0000 - val_loss: 0.6331 - val_accuracy: 0.9701\n",
      "Epoch 65/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 9.2791e-08 - accuracy: 1.0000 - val_loss: 0.6346 - val_accuracy: 0.9701\n",
      "Epoch 66/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 8.5290e-08 - accuracy: 1.0000 - val_loss: 0.6361 - val_accuracy: 0.9701\n",
      "Epoch 67/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 7.8765e-08 - accuracy: 1.0000 - val_loss: 0.6376 - val_accuracy: 0.9701\n",
      "Epoch 68/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 7.2614e-08 - accuracy: 1.0000 - val_loss: 0.6392 - val_accuracy: 0.9701\n",
      "Epoch 69/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 6.7057e-08 - accuracy: 1.0000 - val_loss: 0.6407 - val_accuracy: 0.9702\n",
      "Epoch 70/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 6.1963e-08 - accuracy: 1.0000 - val_loss: 0.6422 - val_accuracy: 0.9701\n",
      "Epoch 71/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 5.7129e-08 - accuracy: 1.0000 - val_loss: 0.6437 - val_accuracy: 0.9701\n",
      "Epoch 72/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 5.2707e-08 - accuracy: 1.0000 - val_loss: 0.6453 - val_accuracy: 0.9702\n",
      "Epoch 73/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 4.8620e-08 - accuracy: 1.0000 - val_loss: 0.6468 - val_accuracy: 0.9702\n",
      "Epoch 74/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 4.4763e-08 - accuracy: 1.0000 - val_loss: 0.6484 - val_accuracy: 0.9702\n",
      "Epoch 75/100\n",
      "165/165 [==============================] - 2s 13ms/step - loss: 4.1269e-08 - accuracy: 1.0000 - val_loss: 0.6499 - val_accuracy: 0.9702\n",
      "Epoch 76/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 3.7903e-08 - accuracy: 1.0000 - val_loss: 0.6515 - val_accuracy: 0.9703\n",
      "Epoch 77/100\n",
      "165/165 [==============================] - 2s 11ms/step - loss: 3.5033e-08 - accuracy: 1.0000 - val_loss: 0.6531 - val_accuracy: 0.9702\n",
      "Epoch 78/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 3.2195e-08 - accuracy: 1.0000 - val_loss: 0.6547 - val_accuracy: 0.9703\n",
      "Epoch 79/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 2.9762e-08 - accuracy: 1.0000 - val_loss: 0.6563 - val_accuracy: 0.9703\n",
      "Epoch 80/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 2.7381e-08 - accuracy: 1.0000 - val_loss: 0.6578 - val_accuracy: 0.9703\n",
      "Epoch 81/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 2.5255e-08 - accuracy: 1.0000 - val_loss: 0.6595 - val_accuracy: 0.9703\n",
      "Epoch 82/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 2.3334e-08 - accuracy: 1.0000 - val_loss: 0.6611 - val_accuracy: 0.9704\n",
      "Epoch 83/100\n",
      "165/165 [==============================] - 1s 9ms/step - loss: 2.1526e-08 - accuracy: 1.0000 - val_loss: 0.6628 - val_accuracy: 0.9704\n",
      "Epoch 84/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 1.9891e-08 - accuracy: 1.0000 - val_loss: 0.6643 - val_accuracy: 0.9704\n",
      "Epoch 85/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 1.8279e-08 - accuracy: 1.0000 - val_loss: 0.6660 - val_accuracy: 0.9704\n",
      "Epoch 86/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 1.6916e-08 - accuracy: 1.0000 - val_loss: 0.6675 - val_accuracy: 0.9705\n",
      "Epoch 87/100\n",
      "165/165 [==============================] - 1s 8ms/step - loss: 1.5597e-08 - accuracy: 1.0000 - val_loss: 0.6691 - val_accuracy: 0.9705\n",
      "Epoch 88/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.4393e-08 - accuracy: 1.0000 - val_loss: 0.6708 - val_accuracy: 0.9706\n",
      "Epoch 89/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.3295e-08 - accuracy: 1.0000 - val_loss: 0.6725 - val_accuracy: 0.9706\n",
      "Epoch 90/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.2267e-08 - accuracy: 1.0000 - val_loss: 0.6740 - val_accuracy: 0.9706\n",
      "Epoch 91/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 1.1282e-08 - accuracy: 1.0000 - val_loss: 0.6757 - val_accuracy: 0.9706\n",
      "Epoch 92/100\n",
      "165/165 [==============================] - 2s 9ms/step - loss: 1.0388e-08 - accuracy: 1.0000 - val_loss: 0.6773 - val_accuracy: 0.9706\n",
      "Epoch 93/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 9.5424e-09 - accuracy: 1.0000 - val_loss: 0.6790 - val_accuracy: 0.9707\n",
      "Epoch 94/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 8.7931e-09 - accuracy: 1.0000 - val_loss: 0.6806 - val_accuracy: 0.9708\n",
      "Epoch 95/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 8.1289e-09 - accuracy: 1.0000 - val_loss: 0.6823 - val_accuracy: 0.9708\n",
      "Epoch 96/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 7.4165e-09 - accuracy: 1.0000 - val_loss: 0.6838 - val_accuracy: 0.9709\n",
      "Epoch 97/100\n",
      "165/165 [==============================] - 2s 12ms/step - loss: 6.8148e-09 - accuracy: 1.0000 - val_loss: 0.6855 - val_accuracy: 0.9709\n",
      "Epoch 98/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 6.3011e-09 - accuracy: 1.0000 - val_loss: 0.6872 - val_accuracy: 0.9708\n",
      "Epoch 99/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 5.7987e-09 - accuracy: 1.0000 - val_loss: 0.6889 - val_accuracy: 0.9708\n",
      "Epoch 100/100\n",
      "165/165 [==============================] - 2s 10ms/step - loss: 5.3814e-09 - accuracy: 1.0000 - val_loss: 0.6905 - val_accuracy: 0.9708\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=256, epochs=100, \n",
    "                    validation_split=0.3, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10.2 训练集和验证集准确率可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xt8VeWd7/HPj1xIQsItQW5BwJFRLiKXiFiGilo7qFMUvMHYVnvGcsbRcTpznFOYznnRckptOxzr2HH0oOLtWBlKvdC+arFVGPVUPQS5yEUEryTcAsg19+R3/lhPkk1IyCbZIZD9fb9e+8Vez3rW2s/DgvVd61lrr23ujoiISJeOboCIiJwZFAgiIgIoEEREJFAgiIgIoEAQEZFAgSAiIoACQUREAgWCiIgACgQREQlSO7oBpyIvL8+HDBnS0c0QETmrrFmzZp+792mp3lkVCEOGDKGwsLCjmyEiclYxs8/iqachIxERARQIIiISKBBERAQ4y64hNKWqqoqioiLKy8s7uimdQkZGBvn5+aSlpXV0U0TkNDvrA6GoqIicnByGDBmCmXV0c85q7s7+/fspKipi6NChHd0cETnN4hoyMrPFZrbXzDY2M9/M7CEz225mG8xsXMy8281sW3jdHlM+3szeD8s8ZK3cm5eXl5Obm6swSAAzIzc3V2dbIkkq3msITwFTTzL/GmBYeM0GHgEws97APOBSYAIwz8x6hWUeAb4ds9zJ1n9SCoPE0d+lSPKKa8jI3d8wsyEnqXI98IxHv8f5jpn1NLP+wBTg9+5+AMDMfg9MNbNVQHd3fyeUPwPcALzSyn6cdrXu7D9aSU1t5/sJ0sNlVTzw6taOboaIxLj9S0PIze7arp+RqGsIA4EdMdNFoexk5UVNlJ/AzGYTnXVw7rnnJqi5bePuFH9RxhellRw+dIhXXvolt95+5ymt4+5v3sz9P3+c7j16NFvn4YU/YvylX2Li5CltbPGpOVJezc9X7mi5ooicNtPGDDxrAqHduPsiYBFAQUHBGXE4XnK0gi9KK+nbPYPu1Qd5+fmnWPC9+46rU11dTWpq83+9b77++xY/538/+NM2t7U1thzJ5JP7r+uQzxaRjpOo7yEUA4NipvND2cnK85soP+MdKqti96Fyemamc05OV+bMmcNHH33EmDFjuOSSS5g8eTLTpk1jxIgRANxwww2MHz+ekSNHsmjRovr1DBkyhH379vHpp58yfPhwvv3tbzNy5Ei++tWvUlZWBsAdd9zBsmXL6uvPmzePcePGcdFFF/HBBx8AUFJSwtVXX83IkSO58847GTx4MPv27TvNfysi0hkk6gxhOXCPmS0huoB8yN13mdkK4EcxF5K/Csx19wNmdtjMJgLvAt8Eft7WRvzg15vYvPNwW1dznD/tm83tXxpKrUcnJ5XVtWSlp5DfKxMz48c//jEbN25k3bp1rFq1iuuuu46NGzfW37a5ePFievfuTVlZGZdccgk33ngjubm5x33Gtm3beP7553nssce45ZZb+NWvfsXXv/71E9qSl5fHe++9x7//+7+zcOFCHn/8cX7wgx9w5ZVXMnfuXH73u9/xxBNPJLT/IpI84goEM3ue6AJxnpkVEd05lAbg7o8CvwWuBbYDpcC3wrwDZvY/gdVhVfPrLjADf0N091Im0cXkM/KC8tGKGiqra+nWNQWArLQU+vbIoEuXpu/GmTBhwnH38D/00EO8+OKLAOzYsYNt27adEAhDhw5lzJgxAIwfP55PP/20yXXPmDGjvs4LL7wAwFtvvVW//qlTp9KrV68mlxURaUm8dxnNamG+A3c3M28xsLiJ8kJgVDyfH695XxuZyNWx/2gFxQfLGNAzg55Z6XEt061bt/r3q1at4g9/+ANvv/02WVlZTJkypcl7/Lt2bbhQlJKSUj9k1Fy9lJQUqqurT6UrIiItSopnGe09XM5n+4+x40ApxQfLOFRW1eIy1TW17D5cTnbXVHpkNv8Yh5ycHI4cOdLkvEOHDtGrVy+ysrL44IMPeOedd1rdh+ZMmjSJpUuXAvDqq6/yxRdfJPwzRCQ5nPF3GSVCVY1TXlVLrTs1tc6Bo5UM65tNRlpKs8vsPlRObS0M6Jl50i9r5ebmMmnSJEaNGkVmZiZ9+/atnzd16lQeffRRhg8fzgUXXMDEiRMT2i+AefPmMWvWLJ599lkuu+wy+vXrR05OTsI/R0Q6P3M/I+7kjEtBQYE3/oGcLVu2MHz48LjXUV1Ty9Y9R8hMS2FoXrcmd/alFdVsLzlKn5yu9O+R2eZ2t6eKigpSUlJITU3l7bff5q677mLdunVtWuep/p2KyJnNzNa4e0FL9ZLiDCFWakoXzsnJYNehMo6UV9O9ieGg3YfLSe0S1TvTff7559xyyy3U1taSnp7OY4891tFNEpGzVNIFAkBudjoHjlWy61A52RmpdIk5SzhaUc3Rimr698gkpZk7ic4kw4YNY+3atR3dDBHpBJLionJjXczo3yODiuoaDhyrrC93d/YcLictpQu53eK7q0hEpLNIykAAyMlIJbtrKrsPlXOoLAqFYxXVHKuopk9O12a/ZyAi0lklbSCYGYN6Z5GRlsJn+0vZfaic3YcrSEvpQm+dHYhIEkraQABIS+nCeXnd6JWVzt4j5ZRWVnNOTtfjrimIiCSLpA4EgC5djPxemQzomUmPzDR6tfPZQXZ2NgA7d+7kpptuarLOlClTaHx7bWMPPvggpaWl9dPXXnstBw8eTFxDRSTpJH0gQDR8lJfdlcG53U7b2cGAAQPqn2TaGo0D4be//S09e/ZMRNNEJEkpENpozpw5PPzww/XT3//+9/nhD3/IVVddVf+o6pdffvmE5T799FNGjYoe5VRWVsbMmTMZPnw406dPP+5ZRnfddRcFBQWMHDmSefPmAdED83bu3MkVV1zBFVdcATQ8ThvggQceYNSoUYwaNYoHH3yw/vOae8y2iAh0tu8hvDIHdr+f2HX2uwiu+XGzs2+99Va+853vcPfd0bP9li5dyooVK7j33nvp3r07+/btY+LEiUybNq3ZR2A88sgjZGVlsWXLFjZs2MC4cePq5y1YsIDevXtTU1PDVVddxYYNG7j33nt54IEHWLlyJXl5eceta82aNTz55JO8++67uDuXXnopl19+Ob169Yr7Mdsikpx0htBGY8eOZe/evezcuZP169fTq1cv+vXrxz/90z8xevRovvKVr1BcXMyePXuaXccbb7xRv2MePXo0o0ePrp+3dOlSxo0bx9ixY9m0aRObN28+aXveeustpk+fTrdu3cjOzmbGjBm8+eabQPyP2RaR5NS5zhBOciTfnm6++WaWLVvG7t27ufXWW3nuuecoKSlhzZo1pKWlMWTIkCYfe92STz75hIULF7J69Wp69erFHXfc0ar11In3Mdsikpx0hpAAt956K0uWLGHZsmXcfPPNHDp0iHPOOYe0tDRWrlzJZ599dtLlv/zlL/OLX/wCgI0bN7JhwwYADh8+TLdu3ejRowd79uzhlVcafkOoucduT548mZdeeonS0lKOHTvGiy++yOTJkxPYWxHprDrXGUIHGTlyJEeOHGHgwIH079+f2267ja997WtcdNFFFBQUcOGFF550+bvuuotvfetbDB8+nOHDhzN+/HgALr74YsaOHcuFF17IoEGDmDRpUv0ys2fPZurUqQwYMICVK1fWl48bN4477riDCRMmAHDnnXcyduxYDQ+JSIuS7vHX0jL9nYp0LvE+/jquISMzm2pmW81su5nNaWL+YDN7zcw2mNkqM8uPmfcTM9sYXrfGlD9lZp+Y2brwGhNv50REJPFaDAQzSwEeBq4BRgCzzGxEo2oLgWfcfTQwH7g/LHsdMA4YA1wK3Gdm3WOW+0d3HxNebftVFxERaZN4zhAmANvd/WN3rwSWANc3qjMCeD28XxkzfwTwhrtXu/sxYAMwte3NPt7ZNOx1ptPfpUjyiicQBgI7YqaLQlms9cCM8H46kGNmuaF8qpllmVkecAUwKGa5BWGY6Wdm1pVWyMjIYP/+/dqRJYC7s3//fjIyzvxfihORxEvUXUb3Af9mZncAbwDFQI27v2pmlwB/BEqAt4GasMxcYDeQDiwCvks03HQcM5sNzAY499xzT/jg/Px8ioqKKCkpSVBXkltGRgb5+fktVxSRTieeQCjm+KP6/FBWz913Es4QzCwbuNHdD4Z5C4AFYd4vgA9D+a6weIWZPUkUKidw90VEgUFBQcEJpwFpaWkMHTo0jm6IiMjJxDNktBoYZmZDzSwdmAksj61gZnlmVreuucDiUJ4Sho4ws9HAaODVMN0//GnADcDGtndHRERaq8UzBHevNrN7gBVACrDY3TeZ2Xyg0N2XA1OA+83MiYaM7g6LpwFvhoe6HQa+7u7VYd5zZtYHMGAd8NeJ65aIiJyqs/6LaSIicnIJ/WKaiIh0fgoEEREBFAgiIhIoEEREBFAgiIhIoEAQERFAgSAiIoECQUREAAWCiIgECgQREQEUCCIiEigQREQEUCCIiEigQBAREUCBICIigQJBREQABYKIiAQKBBERARQIIiISxBUIZjbVzLaa2XYzm9PE/MFm9pqZbTCzVWaWHzPvJ2a2MbxujSkfambvhnX+h5mlJ6ZLIiLSGi0GgpmlAA8D1wAjgFlmNqJRtYXAM+4+GpgP3B+WvQ4YB4wBLgXuM7PuYZmfAD9z9/OBL4C/ant3RESkteI5Q5gAbHf3j929ElgCXN+ozgjg9fB+Zcz8EcAb7l7t7seADcBUMzPgSmBZqPc0cEPruyEiIm0VTyAMBHbETBeFsljrgRnh/XQgx8xyQ/lUM8syszzgCmAQkAscdPfqk6wTADObbWaFZlZYUlIST59ERKQVEnVR+T7gcjNbC1wOFAM17v4q8Fvgj8DzwNtAzams2N0XuXuBuxf06dMnQc0VEZHG4gmEYqKj+jr5oayeu+909xnuPhb4Xig7GP5c4O5j3P1qwIAPgf1ATzNLbW6dIiJyesUTCKuBYeGuoHRgJrA8toKZ5ZlZ3brmAotDeUoYOsLMRgOjgVfd3YmuNdwUlrkdeLmtnRERkdZrMRDCOP89wApgC7DU3TeZ2XwzmxaqTQG2mtmHQF9gQShPA940s83AIuDrMdcNvgv8g5ltJ7qm8ESC+iQiIq1g0cH62aGgoMALCws7uhkiImcVM1vj7gUt1dM3lUVEBFAgiIhIoEAQERFAgSAiIoECQUREAAWCiIgECgQREQEUCCIiEigQREQEUCCIiEigQBAREUCBICIigQJBREQABYKIiAQKBBERARQIIiISKBBERARQIIiISBBXIJjZVDPbambbzWxOE/MHm9lrZrbBzFaZWX7MvJ+a2SYz22JmD5mZhfJVYZ3rwuucxHVLREROVYuBYGYpwMPANcAIYJaZjWhUbSHwjLuPBuYD94dlvwRMAkYDo4BLgMtjlrvN3ceE1962dkZERFovnjOECcB2d//Y3SuBJcD1jeqMAF4P71fGzHcgA0gHugJpwJ62NlpERBIvnkAYCOyImS4KZbHWAzPC++lAjpnluvvbRAGxK7xWuPuWmOWeDMNF/6NuKElERDpGoi4q3wdcbmZriYaEioEaMzsfGA7kE4XIlWY2OSxzm7tfBEwOr280tWIzm21mhWZWWFJSkqDmiohIY/EEQjEwKGY6P5TVc/ed7j7D3ccC3wtlB4nOFt5x96PufhR4BbgszC8Ofx4BfkE0NHUCd1/k7gXuXtCnT59T6pyIiMQvnkBYDQwzs6Fmlg7MBJbHVjCzPDOrW9dcYHF4/znRmUOqmaURnT1sCdN5Ydk04C+AjW3vjoiItFaLgeDu1cA9wApgC7DU3TeZ2XwzmxaqTQG2mtmHQF9gQShfBnwEvE90nWG9u/+a6ALzCjPbAKwjOuN4LGG9EhGRU2bu3tFtiFtBQYEXFhZ2dDNERM4qZrbG3QtaqqdvKouICKBAEBGRQIEgIiKAAkFERAIFgoiIAAoEEREJFAgiIgIoEEREJFAgiIgIoEAQEZFAgSAiIoACQUREAgWCiIgACgQREQkUCCIiAigQREQkUCCIiAigQBARkUCBICIiQJyBYGZTzWyrmW03szlNzB9sZq+Z2QYzW2Vm+THzfmpmm8xsi5k9ZGYWyseb2fthnfXlIiLSMVoMBDNLAR4GrgFGALPMbESjaguBZ9x9NDAfuD8s+yVgEjAaGAVcAlwelnkE+DYwLLymtrUzIiLSevGcIUwAtrv7x+5eCSwBrm9UZwTweni/Mma+AxlAOtAVSAP2mFl/oLu7v+PuDjwD3NCmnoiISJvEEwgDgR0x00WhLNZ6YEZ4Px3IMbNcd3+bKCB2hdcKd98Sli9qYZ0iInIaJeqi8n3A5Wa2lmhIqBioMbPzgeFAPtEO/0ozm3wqKzaz2WZWaGaFJSUlCWquiIg0Fk8gFAODYqbzQ1k9d9/p7jPcfSzwvVB2kOhs4R13P+ruR4FXgMvC8vknW2fMuhe5e4G7F/Tp0yfObomIyKmKJxBWA8PMbKiZpQMzgeWxFcwsz8zq1jUXWBzef0505pBqZmlEZw9b3H0XcNjMJoa7i74JvJyA/oiISCu1GAjuXg3cA6wAtgBL3X2Tmc03s2mh2hRgq5l9CPQFFoTyZcBHwPtE1xnWu/uvw7y/AR4Htoc6rySkRyIi0ioW3eRzdigoKPDCwsKOboaIyFnFzNa4e0FL9fRNZRERARQIIiISKBBERARQIIiISKBAEBERQIEgIiKBAkFERAAFgoiIBAoEEREBFAgiIhIoEEREBFAgiIhIoEAQERFAgSAiIoECQUREAAWCiIgECgQREQEUCCIiEsQVCGY21cy2mtl2M5vTxPzBZvaamW0ws1Vmlh/KrzCzdTGvcjO7Icx7ysw+iZk3JrFdExGRU5HaUgUzSwEeBq4GioDVZrbc3TfHVFsIPOPuT5vZlcD9wDfcfSUwJqynN7AdeDVmuX9092WJ6YqIiLRFPGcIE4Dt7v6xu1cCS4DrG9UZAbwe3q9sYj7ATcAr7l7a2saKiEj7iScQBgI7YqaLQlms9cCM8H46kGNmuY3qzASeb1S2IAwz/czMusbZZhERaQeJuqh8H3C5ma0FLgeKgZq6mWbWH7gIWBGzzFzgQuASoDfw3aZWbGazzazQzApLSkoS1FwREWksnkAoBgbFTOeHsnruvtPdZ7j7WOB7oexgTJVbgBfdvSpmmV0eqQCeJBqaOoG7L3L3Ancv6NOnT1ydEhGRUxdPIKwGhpnZUDNLJxr6WR5bwczyzKxuXXOBxY3WMYtGw0XhrAEzM+AGYOOpN19ERBKlxUBw92rgHqLhni3AUnffZGbzzWxaqDYF2GpmHwJ9gQV1y5vZEKIzjP9stOrnzOx94H0gD/hhm3oiIiJtYu7e0W2IW0FBgRcWFnZ0M0REzipmtsbdC1qqp28qi4gIoEAQEZFAgSAiIoACQUREAgWCiIgACgQREQkUCCIiAigQREQkUCCIiAigQBARkUCBICIigAJBREQCBYKIiAAKBBERCRQIIiICKBBERCRQIIiICACpHd0AEZFOwx3KD8GR3XBkFxzdE/15ZM/x0xVHGpZJ6Qo5fSGnP2T2BrNQngbdzmmYN3gSZHRv1+YrEERE6tTWQvnBaMd+Ml4LZQcadvx7NsGudbBzPVQcOrF+ek7Djj3/EujavWHHX1UOR3fDwc9h57qGZWoqoXQ/ENpyT+GZEQhmNhX4VyAFeNzdf9xo/mBgMdAHOAB83d2LzOwK4GcxVS8EZrr7S2Y2FFgC5AJrgG+4e2VbOyQiSajsCzi6t+l5VWUxR+q7o9fRPVB6gPqdbU1VtPzR3VBbfeqfn5IOfUfCqBmQez7k9It2/jn9ILsvdM1uXb9i29Xz3Nat4xSYt5CEZpYCfAhcDRQBq4FZ7r45ps4vgd+4+9NmdiXwLXf/RqP19Aa2A/nuXmpmS4EX3H2JmT0KrHf3R07WloKCAi8sLDz1XopI67nDF59AdQXkDoOUkxxHukfDIUd2H390G6u2Go6VRHWOlUBGj2jnmZUXfc7OdbDnfeiSCtn9oiPrtKymP+/Qjqj+wc/i709WbrTerN7QJSUqs5Rox53TNxqm6RLHsXJW77BMP+g5GFLT42/DaWZma9y9oKV68ZwhTAC2u/vHYcVLgOuBzTF1RgD/EN6vBF5qYj03Aa+EMDDgSuAvw7ynge8DJw0EkQ5TUxWOMndHR2ytOYpMhC6pMTuuPg07rtrqqF1HdsOxvVBbE5V7LZTuazgqrolpd0aPaD3Z/aLhibr+VZWFCg6Hi2HX+mhcHCA1E/qNgt5/AhbuSakui8bIj4aj76rS+PtjKeA1x5dl5UL/i6P3X3wCn78dhVFTuuXBgDEw/o7oCLpuGCZWSnoIln6QfQ6kdo2/fUkmnkAYCOyImS4CLm1UZz0wg2hYaTqQY2a57r4/ps5M4IHwPhc46O51/zqLwuckp4qjUHmsYTot4/gxRkmsujHbup1Y2RdNjxlXlcHu96Ox4ZIPop3r2cq6REfgqRmhwKHsIFQeOb5eZq9ovLtOt1wYOSPaQadlReGwax189seGOilp0c52wNiGHW9Ov2jHXncE3rgt3fpEdTJ6Rv/2j+6JAq3nIOg+UP/2O0iiLirfB/ybmd0BvAEUA/Wxb2b9gYuAFae6YjObDcwGOPfc9h9Da3e1tfDxSnjvmWhnc3QPVB49sV5qZsN/rJx+x/9Hq/vPVneElpYJvYaeXf+Jamth31Y4VByN7ZZ9Ab3Pi472uodjg4rD0U4ivVt0Gp+SCpWl0d/b7g3Rxb+WVB5rGDeuuwAYz3J1uvWB/mPggmujnVVO/6gspYOGB2oqw5nArujIvy7IzMIdKf0huw90SWsoz8qNwqCpoZ6Ko9G/wZS06MyjpaPni29NbH8gGl/vmg25f5L4dcspiScQioFBMdP5oayeu+8kOkPAzLKBG9099n/dLcCL7l4VpvcDPc0sNZwlnLDOmHUvAhZBdA0hjvaeGarKYi5ghT8PFcGW5dHdBFm5MGQydP/zmItOYYded8RUdxFs1wY48ipUHWv+83qcC3/65zDs6igccvpB15wTQ6LiKOzdDNXl0bTXRhfX6j6vupnr+undGsIo9uylsrShneUxd1ekZUK/i6IdfM8h0U746B7Ytw22rYAPX42GNpqS0TMaIqguiykMO7ayA6d2pN4lrWGIpfd5MGRSw7h03UW/zN4N4RorJS0E71kUtKeqbmcsQnyBsBoYFu4KKiYa+vnL2ApmlgcccPdaYC7RHUexZoVyANzdzWwl0XWFJcDtwMut7USHKz0AK38E+z5s2LGWN3HrWZc0OHciXDUPhn/t1McyK440DHOUxozGle6Hbb+Htf8HVj/WUJ6W1XB2kdkL9m+LdshNXeiD6Kg3NbOJGR6FVOOx3sa6dqc+1KqONYyzW5fjd+Jde8D5V8H5X4mOCuuGDvZti4Yj9mxqCKBu50RnUHXj29l9o5DpPyYaD26JpUAXff9SJB4t3mUEYGbXAg8S3Xa62N0XmNl8oNDdl5vZTcD9RHuaN4C73b0iLDsE+L/AoBAYdes8jygMegNriW5VbebKUeSMuMuo8li0s6qzawP8x23Rzqr/mIaLdPW3ndUdifaPdsrtebRZVQbFa+Dwroaj9rqx8tJ90ZnDgDHQb/Tx9zNn9g5HyidpX20NHNsXrTd2iKtuaCv7nOiIuk51RXQmsnNddEbUrU/0d9FjUDTWHFtXRNpVvHcZxRUIZ4oOD4RP3oCnp0X3GV8wFXIGwGvzo9vPbnkW8sd3XNtERJqRyNtOpc6b/ysaU+6RD+88CrVV0dfJb346upAnInIWUyDEa9d6+HhVNP4/+R+i8fy9H0RDMBr+EJFOQIEQrz/+HNKzoeC/RNNdc2DQJR3bJhGRBNLtF/E4uAM2vgDjbofMnh3dGhGRdqFAiMc74YkaE+/q2HaIiLQjBUJLyg7Ce0/DqBujb6qKiHRSCoSW/GFe9N2DL/1tR7dERKRdKRBOZsuvYc1TMOle6D+6o1sjItKuFAjNObwTlv9t9O3jK/65o1sjItLuFAhNqa2FF/9r9PiFG584o3/4QkQkUfQ9hKa893T0mIppP4e88zu6NSIip4XOEBqrqYK3Hoh+CHvsN1quLyLSSSgQGtv4q+jpnJPv69zPwRcRaUSBEKu2Ft58APqOin5sRkQkiSgQYn3wm+hnHf/s73V2ICJJR4FQxz16vHXv82Dk9I5ujYjIaadAqPPR69HPN/7Z30OXlI5ujYjIaZccgfDxf0YXi09m9ePR7/eOnnl62iQicoZJjkB45xFY8c/R7wI35chu+HAFjJmlL6GJSNKKKxDMbKqZbTWz7WY2p4n5g83sNTPbYGarzCw/Zt65ZvaqmW0xs81mNiSUP2Vmn5jZuvAak6hOnWD0LXBkJ3z6VtPz1y8Br9H3DkQkqbUYCGaWAjwMXAOMAGaZ2YhG1RYCz7j7aGA+cH/MvGeAf3H34cAEYG/MvH909zHhta4N/Ti5C66B9Bx4f+mJ89xh7bNw7mWQN6zdmiAicqaL5wxhArDd3T9290pgCXB9ozojgNfD+5V180NwpLr77wHc/ai7lyak5aciLRNGTIPNy6Gq7Ph5n78D+7fr7EBEkl48gTAQ2BEzXRTKYq0HZoT304EcM8sF/hQ4aGYvmNlaM/uXcMZRZ0EYZvqZmXVt6sPNbLaZFZpZYUlJSVydatJFN0PFYfjwd8eXr302+q3kkTe0ft0iIp1Aoi4q3wdcbmZrgcuBYqCG6OF5k8P8S4DzgDvCMnOBC0N5b+C7Ta3Y3Re5e4G7F/Tp06f1LRz6ZcjuBxt+2VBWfhg2vQijZkB6t9avW0SkE4gnEIqB2N+OzA9l9dx9p7vPcPexwPdC2UGis4l1YbipGngJGBfm7/JIBfAk0dBU++mSAhfdBNtehdID0WMq/vhzqCqFsd9s148WETkbxBMIq4FhZjbUzNKBmcDy2ApmlmdmdeuaCyyOWbanmdUd2l8JbA7L9A9/GnADsLEtHYnL6Fugtgr+8yfwxNXwxk9h2J9DfkG7f7SIyJmuxd9DcPdqM7sHWAGkAIsJo2XMAAAEjElEQVTdfZOZzQcK3X05MAW438wceAO4OyxbY2b3Aa+FHf8a4LGw6udCUBiwDvjrxHatCf1GQ58L4d1Hoy+h3fAoXDxTzy0SEQHM3Tu6DXErKCjwwsLCtq3ko9fh83fhsr+BjB6JaZiIyBnMzNa4e4tDIcn3i2l/cmX0EhGR4yTHoytERKRFCgQREQEUCCIiEigQREQEUCCIiEigQBAREUCBICIigQJBRESAs+ybymZWAnzWysXzgH0JbM7ZIhn7nYx9huTst/ocn8Hu3uLjos+qQGgLMyuM56vbnU0y9jsZ+wzJ2W/1ObE0ZCQiIoACQUREgmQKhEUd3YAOkoz9TsY+Q3L2W31OoKS5hiAiIieXTGcIIiJyEkkRCGY21cy2mtl2M5vT0e1pD2Y2yMxWmtlmM9tkZn8Xynub2e/NbFv4s1dHtzXRzCzFzNaa2W/C9FAzezds7/8IP/3aqZhZTzNbZmYfmNkWM7uss29rM/v78G97o5k9b2YZnXFbm9liM9trZhtjyprcthZ5KPR/g5mNa8tnd/pAMLMU4GHgGmAEMMvMRnRsq9pFNfDf3H0EMBG4O/RzDvCauw8DXgvTnc3fAVtipn8C/Mzdzwe+AP6qQ1rVvv4V+J27XwhcTNT/TrutzWwgcC9Q4O6jiH7Odyadc1s/BUxtVNbctr0GGBZes4FH2vLBnT4QgAnAdnf/2N0rgSXA9R3cpoRz913u/l54f4RoBzGQqK9Ph2pPAzd0TAvbh5nlA9cBj4dpA64EloUqnbHPPYAvA08AuHulux+kk29rol94zDSzVCAL2EUn3Nbu/gZwoFFxc9v2euAZj7wD9DSz/q397GQIhIHAjpjpolDWaZnZEGAs8C7Q1913hVm7gb4d1Kz28iDw34HaMJ0LHHT36jDdGbf3UKAEeDIMlT1uZt3oxNva3YuBhcDnREFwCFhD59/WdZrbtgndvyVDICQVM8sGfgV8x90Px87z6JayTnNbmZn9BbDX3dd0dFtOs1RgHPCIu48FjtFoeKgTbuteREfDQ4EBQDdOHFZJCu25bZMhEIqBQTHT+aGs0zGzNKIweM7dXwjFe+pOIcOfezuqfe1gEjDNzD4lGgq8kmhsvWcYVoDOub2LgCJ3fzdMLyMKiM68rb8CfOLuJe5eBbxAtP07+7au09y2Tej+LRkCYTUwLNyNkE50IWp5B7cp4cLY+RPAFnd/IGbWcuD28P524OXT3bb24u5z3T3f3YcQbdfX3f02YCVwU6jWqfoM4O67gR1mdkEougrYTCfe1kRDRRPNLCv8W6/rc6fe1jGa27bLgW+Gu40mAodihpZOnbt3+hdwLfAh8BHwvY5uTzv18c+ITiM3AOvC61qiMfXXgG3AH4DeHd3Wdur/FOA34f15wP8DtgO/BLp2dPvaob9jgMKwvV8CenX2bQ38APgA2Ag8C3TtjNsaeJ7oOkkV0dngXzW3bQEjuovyI+B9oruwWv3Z+qayiIgAyTFkJCIicVAgiIgIoEAQEZFAgSAiIoACQUREAgWCiIgACgQREQkUCCIiAsD/B42RkqTMIaOtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(history.history['accuracy'])\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.legend(['training', 'validation'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验小结\n",
    "\n",
    "在本实验中，你实现了分别通过：`正态分布随机值`，`Xavier/Glorot 权重初始值`，以及 `HE 权重初始值` 对相同的数据集与神经网络架构进行训练。通过上述结果对比，可以看出三种权重初始值最终的准确率接近。然而，`HE 初始值` 明显比其他两个更快进入平稳收敛状态，也验证了 `HE 初始值` 与 `RuLu 激活函数` 是更理想的搭配组合。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}