12. 实现二层神经网络误差反向传播法.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实验：实现二层神经网络误差反向传播法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验概要\n",
    "\n",
    "#### 神经网络学习的基本步骤\n",
    "\n",
    "利用随机梯度下降法，求梯度并更新权重和偏置参数，整个过程是个循环过程。\n",
    "\n",
    "1. 从训练数据中随机选择一部分数据\n",
    "2. 构建网络，利用前向传播，求出输出值。然后利用输出值与目标值得到损失函数，利用损失函数，利用反向传播方法，求各参数的梯度。\n",
    "3. 将权重参数沿梯度方向进行微小更新\n",
    "4. 重复以上 1、2、3 步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 神经网络学习的反向传播法的实现\n",
    "\n",
    "为定义和保存以上神经网络架构，需要先定义几个实例变量：\n",
    "- 保存权重参数的字典型变量 params。\n",
    "- 保存各层的信息的顺序字典 layers，这里的顺序是插入数据的先后顺序。\n",
    "- 神经网络的最后一层 lastlayer\n",
    "\n",
    "除了以上三个实例变量，还需要定义一些方法\n",
    "- 构造函数，以初始化变量和权重等\n",
    "- 预测方法，根据神经网络各层的前向传播得到最后的输出值\n",
    "- 损失函数，根据输出值与目标值，得到交叉熵作为衡量两个分布的距离。\n",
    "- 评估指标，这里使用精度来衡量模型性能\n",
    "- 最后就是计算梯度，这里使用反向传播方法的得到，具体是利用导数的链式法则，从后往前，获取各层的梯度作为前层梯度往前传递（往输入端）\n",
    "\n",
    "当然，这里需要先定义好各层类，各类中各层的权重参数、包括前向传播结果，反向传播的梯度等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验目标\n",
    "\n",
    "在本实验中，我们将实现二层神经网络误差反向传播法。\n",
    "\n",
    "#### 二层神经网络 vs. 三层神经网络\n",
    "\n",
    "对于初学者，可能会对单层神经网络、二层神经网络、三层神经网络这些说法有点模糊：\n",
    "\n",
    "![](./img/1_12.png)\n",
    "\n",
    "- **左边：** 是一个 2 层神经网络，隐层由 4 个神经元（也可称为单元（unit））组成，输出层由 2 个神经元组成，输入层是 3 个神经元。\n",
    "- **右边：** 是一个 3 层神经网络，两个含 4 个神经元的隐层。注意：层与层之间的神经元是全连接的，但是层内的神经元不连接。\n",
    "\n",
    "#### 命名规则\n",
    "\n",
    "当我们说 N 层神经网络的时候，我们没有把输入层算入。因此，单层的神经网络就是没有隐层的（输入直接映射到输出）。因此，有的研究人员会认为 **逻辑回归** 或者 **SVM** 只是单层神经网络的一个特例。研究者们也会使用 **人工神经网络**（Artificial Neural Networks 缩写: **ANN**）或者 **多层感知器**（Multi-Layer Perceptrons 缩写: **MLP**）来指代神经网络。同样，很多研究人员并不喜欢神经网络算法和人类大脑之间的类比，他们更倾向于用单元（**unit**）而不是神经元（**neuron**）作为术语。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 定义激活函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    if x.ndim == 2:\n",
    "        x = x.T\n",
    "        x = x - np.max(x, axis=0)\n",
    "        y = np.exp(x) / np.sum(np.exp(x), axis=0)\n",
    "        return y.T \n",
    " \n",
    "    #防止出现溢出情况\n",
    "    x = x - np.max(x) \n",
    "    return np.exp(x) / np.sum(np.exp(x))\n",
    " \n",
    "class Sigmoid:\n",
    "    def __init__(self):\n",
    "        self.out=None\n",
    "        \n",
    "    def forward(self,x):\n",
    "        self.out=1 / (1 + np.exp(-x))\n",
    "        return self.out\n",
    "    \n",
    "    def backward(self,dout):\n",
    "        dx=dout*(1.0 -self.out) * self.out\n",
    "        return dx\n",
    "    \n",
    "class Relu:\n",
    "    def __init__(self):\n",
    "        self.mask = None\n",
    " \n",
    "    def forward(self, x):\n",
    "        self.mask = (x <= 0)\n",
    "        out = x.copy()\n",
    "        out[self.mask] = 0\n",
    " \n",
    "        return out\n",
    " \n",
    "    def backward(self, dout):\n",
    "        dout[self.mask] = 0\n",
    "        dx = dout\n",
    " \n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 定义 Affine 层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Affine:\n",
    "    def __init__(self, W, b):\n",
    "        self.W =W\n",
    "        self.b = b\n",
    "        \n",
    "        self.x = None\n",
    "        self.original_x_shape = None\n",
    "        # 权重和偏置参数的导数\n",
    "        self.dW = None\n",
    "        self.db = None\n",
    " \n",
    "    def forward(self, x):\n",
    "        # 对应张量\n",
    "        self.original_x_shape = x.shape\n",
    "        x = x.reshape(x.shape[0], -1)\n",
    "        self.x = x\n",
    " \n",
    "        out = np.dot(self.x, self.W) + self.b\n",
    " \n",
    "        return out\n",
    " \n",
    "    def backward(self, dout):\n",
    "        dx = np.dot(dout, self.W.T)\n",
    "        self.dW = np.dot(self.x.T, dout)\n",
    "        self.db = np.sum(dout, axis=0)\n",
    "        \n",
    "        # 还原输入数据的形状（对应张量）\n",
    "        dx = dx.reshape(*self.original_x_shape)  \n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 定义 SoftmaxWithLoss 层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SoftmaxWithLoss:\n",
    "    def __init__(self):\n",
    "        self.loss = None\n",
    "        # softmax的输出\n",
    "        self.y = None \n",
    "        # 监督数据\n",
    "        self.t = None \n",
    " \n",
    "    def forward(self, x, t):\n",
    "        self.t = t\n",
    "        self.y = softmax(x)\n",
    "        self.loss = cross_entropy_error(self.y, self.t)\n",
    "        \n",
    "        return self.loss\n",
    " \n",
    "    def backward(self, dout=1):\n",
    "        batch_size = self.t.shape[0]\n",
    "        # 监督数据是one-hot-vector的情况\n",
    "        if self.t.size == self.y.size: \n",
    "            dx = (self.y - self.t) / batch_size\n",
    "        else:\n",
    "            dx = self.y.copy()\n",
    "            dx[np.arange(batch_size), self.t] -= 1\n",
    "            dx = dx / batch_size\n",
    "        \n",
    "        return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_entropy_error(y, t):\n",
    "    if y.ndim == 1:\n",
    "        t = t.reshape(1, t.size)\n",
    "        y = y.reshape(1, y.size)\n",
    "        \n",
    "    # 如果t为one-hot格式，把它转换为数字格式\n",
    "    if t.size == y.size:\n",
    "        t = t.argmax(axis=1)\n",
    "             \n",
    "    batch_size = y.shape[0]\n",
    "    return -np.sum(np.log(y[np.arange(batch_size), t] + 1e-8)) / batch_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 定义神经网络类\n",
    "\n",
    "实例变量如下：\n",
    "\n",
    "- **params：** 保存神经网络的参数的字典型变量。\n",
    "   - `params['W1']` 是第 1 层的权重， `params['b1']` 是第 1 层的偏置\n",
    "   - `params['W2']` 是第 2 层的权重， `params['b2']` 是第 2 层的偏置\n",
    "\n",
    "- **layers：** 保存神经网络的层的有序字典型变量。\n",
    "   - 以 `layers['Affine1']`、 `layers['ReLu1']`、 `layers['Affine2']` 的形式，\n",
    "   - 通过有序字典保存各个层\n",
    "\n",
    "- **lastLayer：** 神经网络的最后一层。本实验中为 SoftmaxWithLoss 层\n",
    "\n",
    "`TwoLayerNet` 类使用了层。通过使用层，获得识别结果的处理（`predict()`）和计算梯度的处理（`gradient()`）只需通过层之间的传递就能完成。\n",
    "\n",
    "\n",
    "通过将神经网络的组成元素以层的方式实现，可以轻松地构建神经网络。这个用层进行模块化的实现具有很大优点。因为想另外构建一个神经网络（比如 5 层、 10 层、 20 层……的大的神经网络）时，只需像组装积木那样添加必要的层就可以了。之后，通过各个层内部实现的正向传播和反向传播，就可以正确计算进行识别处理或学习所需的梯度。下面是代码实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from collections import OrderedDict\n",
    " \n",
    "class TwoLayerNet:\n",
    " \n",
    "    # 进行初始化。参数从头开始依次是输入层的神经元数、隐藏层的神经元数、\n",
    "    # 输出层的神经元数、初始化权重时的高斯分布的规模\n",
    "    def __init__(self, input_size, hidden_size,output_size, weight_init_std = 0.01):\n",
    "        \n",
    "        # 初始化权重\n",
    "        self.params = {}\n",
    "        self.params['W1'] = weight_init_std * np.random.randn(input_size, hidden_size)\n",
    "        self.params['b1'] = np.zeros(hidden_size)\n",
    "        self.params['W2'] = weight_init_std * np.random.randn(hidden_size, output_size) \n",
    "        self.params['b2'] = np.zeros(output_size)\n",
    " \n",
    "        # 生成层\n",
    "        self.layers = OrderedDict()\n",
    "        self.layers['Affine1'] = Affine(self.params['W1'], self.params['b1'])\n",
    "        \n",
    "        # self.layers['Sigmoid1'] = Sigmoid()\n",
    "        self.layers['Relu1'] = Relu()\n",
    "        self.layers['Affine2'] = Affine(self.params['W2'], self.params['b2'])\n",
    " \n",
    "        self.lastLayer = SoftmaxWithLoss()\n",
    "    \n",
    "    # 进行识别（推理）。参数x是图像数据\n",
    "    def predict(self, x):\n",
    "        for layer in self.layers.values():\n",
    "            x = layer.forward(x)\n",
    "        \n",
    "        return x\n",
    "        \n",
    "    # 计算损失函数的值。参数x是图像数据、t是正确解标签\n",
    "    # x:输入数据, t:监督数据\n",
    "    def loss(self, x, t):\n",
    "        y = self.predict(x)\n",
    "        return self.lastLayer.forward(y, t)\n",
    "    \n",
    "    # 计算识别精度\n",
    "    def accuracy(self, x, t):\n",
    "        y = self.predict(x)\n",
    "        y = np.argmax(y, axis=1)\n",
    "        \n",
    "        # print(\"预测值\",y[0],y.shape)        \n",
    "        if t.ndim != 1: \n",
    "            t = np.argmax(t, axis=1)\n",
    "        accuracy = np.sum(y == t) / float(x.shape[0])\n",
    "        return accuracy\n",
    "\n",
    "    # 通过数值微分计算关于权重参数的梯度\n",
    "    # x:输入数据, t:监督数据\n",
    "    def numerical_gradient(self, x, t):\n",
    "        loss_W = lambda W: self.loss(x, t)\n",
    "        \n",
    "        grads = {}\n",
    "        grads['W1'] = numerical_gradient(loss_W, self.params['W1'])\n",
    "        grads['b1'] = numerical_gradient(loss_W, self.params['b1'])\n",
    "        grads['W2'] = numerical_gradient(loss_W, self.params['W2'])\n",
    "        grads['b2'] = numerical_gradient(loss_W, self.params['b2'])\n",
    "        \n",
    "        return grads\n",
    "    \n",
    "    # 通过误差反向传播法计算关于权重参数的梯度   \n",
    "    def gradient(self, x, t):\n",
    "        # 前向\n",
    "        self.loss(x, t)\n",
    " \n",
    "        # 后向\n",
    "        dout = 1\n",
    "        dout = self.lastLayer.backward(dout)\n",
    "        \n",
    "        layers = list(self.layers.values())\n",
    "        layers.reverse()\n",
    "        for layer in layers:\n",
    "            dout = layer.backward(dout)\n",
    " \n",
    "        # 用一个字典记录各参数（权重和偏置）的梯度\n",
    "        grads = {}\n",
    "        grads['W1'], grads['b1'] = self.layers['Affine1'].dW, self.layers['Affine1'].db\n",
    "        grads['W2'], grads['b2'] = self.layers['Affine2'].dW, self.layers['Affine2'].db\n",
    " \n",
    "        return grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中，代码中的 ——\n",
    "\n",
    "```python\n",
    "# 生成层\n",
    "self.layers = OrderedDict()\n",
    "self.layers['Affine1'] = Affine(self.params['W1'], self.params['b1'])\n",
    "```\n",
    "`OrderedDict` 是有序字典，**有序** 是指它可以记住向字典里添加元素的顺序。因此，神经网络的正向传播只需按照添加元素的顺序调用各层的 `forward()`方法就可以完成处理，而反向传播只需要按照相反的顺序调用各层即可。因为 Afne 层和 ReLU 层的内部会正确处理正向传播和反向传播，所以这里要做的事情仅仅是以正确的顺序连接各层，再按顺序（或者逆序）调用各层。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. 使用误差反向传播法训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1 导入库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "# 导入pickle执行序列化\n",
    "import pickle  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2 读入数据\n",
    "\n",
    "使用 data 中内置的 mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "base_path = os.environ.get(\"BATH_PATH\",'./data/')\n",
    "data_path = os.path.join(base_path + \"lab12/\")\n",
    "result_path = \"result/\"\n",
    "os.makedirs(result_path, exist_ok=True)\n",
    "file_path = data_path + \"mnist.pkl\"\n",
    "\n",
    "def load_mnist(normalize=True, flatten=True, one_hot_label=False):\n",
    "    \"\"\"读入MNIST数据集\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    normalize : 将图像的像素值正规化为0.0~1.0\n",
    "    one_hot_label : \n",
    "        one_hot_label为True的情况下，标签作为one-hot数组返回\n",
    "        one-hot数组是指[0,0,1,0,0,0,0,0,0,0]这样的数组\n",
    "    flatten : 是否将图像展开为一维数组\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    (训练图像, 训练标签), (测试图像, 测试标签)\n",
    "    \"\"\"\n",
    "    with open(file_path, 'rb') as f:\n",
    "        \n",
    "        # 通过利用对象 f，使用 pickle 加载存储在本地的数据\n",
    "        dataset = pickle.load(f)        \n",
    "    \n",
    "    # normalize=True，执行标准化\n",
    "    if normalize:\n",
    "        for key in ('train_img', 'test_img'):\n",
    "            dataset[key] = dataset[key].astype(np.float32)\n",
    "            \n",
    "            # 由于图像的像素值是0-255，直接除以255，将像素值标准化为[0-1]区间内\n",
    "            dataset[key] /= 255.0      \n",
    "    \n",
    "    # one_hot_label=False，不执行one-hot表示，此操作忽略\n",
    "    if one_hot_label:\n",
    "        \n",
    "        # 引用 change_one_hot_label() 函数执行one-hot表示\n",
    "        dataset['train_label'] = _change_one_hot_label(dataset['train_label']) \n",
    "        \n",
    "        # 引用 change_one_hot_label() 函数执行one-hot表示\n",
    "        dataset['test_label'] = _change_one_hot_label(dataset['test_label'])   \n",
    "    \n",
    "    # flatten=True，此操作忽略\n",
    "    if not flatten:\n",
    "         for key in ('train_img', 'test_img'):\n",
    "            dataset[key] = dataset[key].reshape(-1, 1, 28, 28)\n",
    "\n",
    "    return (dataset['train_img'], dataset['train_label']), (dataset['test_img'], \n",
    "                                                            dataset['test_label']) \n",
    "\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.3 参数设置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 适当设定循环的次数，可以增加迭代次数提升准确率，\n",
    "# 同时可以看到更平滑上升的学习曲线，然而训练时间同时也会被拉长\n",
    "iters_num = 2000  \n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 100\n",
    "learning_rate = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.4 模型训练\n",
    "\n",
    "> **训练预计需时 30 分钟，请耐心等待。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "学习速率：0.09000000000000001\n",
      "训练集准确率, 测试集准确率 | 0.14038333333333333, 0.1334\n",
      "学习速率：0.08100000000000002\n",
      "训练集准确率, 测试集准确率 | 0.9017833333333334, 0.905\n",
      "学习速率：0.07290000000000002\n",
      "训练集准确率, 测试集准确率 | 0.9155, 0.9215\n",
      "学习速率：0.06561000000000002\n",
      "训练集准确率, 测试集准确率 | 0.9283666666666667, 0.9283\n",
      "训练总耗时： 10.856628656387329 秒\n"
     ]
    }
   ],
   "source": [
    "# 记录模型训练时间\n",
    "import time \n",
    "# 记录训练开始时间\n",
    "start_sum = time.time() \n",
    "\n",
    "network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)\n",
    "train_loss_list = []\n",
    "train_acc_list = []\n",
    "test_acc_list = []\n",
    "iter_per_epoch = max(train_size / batch_size, 1)\n",
    "\n",
    "for i in range(iters_num):\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "\n",
    "    # 通过误差反向传播法求梯度\n",
    "    grad = network.gradient(x_batch, t_batch)\n",
    "\n",
    "    for key in ('W1', 'b1', 'W2', 'b2'):\n",
    "        network.params[key] -= learning_rate * grad[key]\n",
    "\n",
    "    loss = network.loss(x_batch, t_batch)\n",
    "    train_loss_list.append(loss)\n",
    "    if i % iter_per_epoch == 0:\n",
    "         \n",
    "        # 更新\n",
    "        if i%100==0:\n",
    "            learning_rate*=0.9\n",
    "\n",
    "       # 更新参数\n",
    "        print(\"学习速率：\" + str(learning_rate))\n",
    "        train_acc = network.accuracy(x_train, t_train)\n",
    "        test_acc = network.accuracy(x_test, t_test)\n",
    "        train_acc_list.append(train_acc)\n",
    "        test_acc_list.append(test_acc)\n",
    "        print(\"训练集准确率, 测试集准确率 | \" + \n",
    "              str(train_acc) + \", \" + str(test_acc))\n",
    "        \n",
    "# 记录训练结束时间\n",
    "end_sum = time.time() \n",
    "print('训练总耗时：',end_sum - start_sum, '秒')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5 结果可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xl4VOX5xvHvMzNZCDsJOyiLuAAKCCKbij9FWdwQV0StbcHWYq21VrS4L7VaqVp3q9a6oLiwiooiiCsYEAQEJKBIWCQgWyDbzLy/P2aISUjIABlOlvtzXbmYs8zMfTLhfeac95z3mHMOERERAJ/XAUREpPJQURARkUIqCiIiUkhFQURECqkoiIhIIRUFEREpFLeiYGbPm9kmM1tSxnIzs0fNLMPMvjGz4+OVRUREYhPPPYX/AgP3sXwQ0CH6Mwp4Mo5ZREQkBnErCs65OcDP+1jlXOB/LuJLoIGZNY9XHhERKV/Aw/duCawtMp0Znbeh5IpmNorI3gS1a9fufvTRRx+SgCIi1cX8+fM3O+cal7eel0UhZs65Z4BnAHr06OHS09M9TiQiUrWY2ZpY1vPy7KN1QOsi062i80RExCNeFoUpwBXRs5B6Adudc3sdOhIRkUMnboePzGw80B9IM7NM4HYgAcA59xQwHRgMZAC7gavilUVERGITt6LgnLu0nOUO+EO83l9ERPafrmgWEZFCKgoiIlJIRUFERAqpKIiISCEVBRERKaSiICIihVQURESkkIqCiIgUUlEQEZFCVWKUVBGRmDgHLgzhYOQnsXZkfu4OyN8VmedCEA5F1kvrEFn+82rYtSW6LPpcXwDa9IssXzsPdqyLPC8cJBQsIBhIIafD2eQHw/iWTYHtPxIORZaFQ0FyE1P5sd0l5AVDtFr+PEnZ63DhIC4UJBwOsTWpFV+0uIL8YJhTv3+IunkbsXAI50JYOMjqxKMYX/dK8oNhbtpyKyt9bak35G6GHBff286oKIjUNKEgBHOgIDfSaCamRBrNTd9CQQ4EcyGYF2kYD+sN9VvCtrWQ8UG0USzScHYaCg0Ph5+WwpK3iiwLRRrYPtdCg8Pgh09hwUu/PG9Pwzz4n5HXXzoJ5j37y/I964x4G1e7MaEvn8Y39wlcKFjs/b+79DNyfbVpOu9+mi99FnOhYpv68pkLyQtBr6V302nDW8WW5VsyY455n/xgmMvX38OJ2TOLLd9qDRhW+0XygmHuy7uPU9xXhcv8wNpwU/rnJwPwasK/6OP/ttjzl4YP56o57QF4LXEqrW0NQfyE8RHEz/fhI3hsRR8S/T56B1aRymbC5idsAcLmJ8/lkpMUIjHgIy+hPolJ9amd5K+ov4IyWWQIoqpD91OQask5yP7pl0Z5z791m0GjdpEGfMmbRZbnRhr2dv2h7cmQvQmm31j8uQU50Pc6OPaCSKP93JmR54SDv7zveU9B10thzRfwwt53z915zvNktx+ML+NDmk4dsdfy+Se/wMa03qT9OJ0T5t+Is0ijt6dxe+PoR/gx+Sg6ZH3A6eufJIyPEH6C+AnhY1zDv7GWZpywew6Dc6YRdD6CzkcBPgqcj5uDo/gpVIcB9hVn+r8i7HyFzw3h4+/B4eSSRH/fQnr4VkRe20XeI4SPZ0NDCOOjpy3jCN96wvjAn4D5Ajh/Ap8k9CMp4ONofqCJbxs+XwBfIIDPHwB/MuvrdCQx4KNp+Cdqk0cgIZGAP4A/IQF/QhLBlKYkBnykkEOS30cgIYGE6E9iIEBiwEdiwEdSwEei309Sgo9Ev69wfsBnmFn8/q6KMLP5zrke5a6noiBSRDgc/aYcbVTNB/Wiu+trv4LcbcUb3XotoMOAyPKPH4DdW4os3w2te0Gf0ZHlz5wKuduLN9zdRsDgByFUAHen7RVnW7ffs/HEWyjI3sqxLx9XPKoFWHLkaBa1/TW+7I0MXjCKfEuiwJLIt0TyLZHPG5zDopReJOVs5vStr5LjEshxiewOJ7A7nMg837FkhJqTGNzJUcEV7AwlkB0KkE8CBfjZ6Bqxi1okkU99dhGieKOcR2KkoS1DwGeFDWDRxjAp4I/8W2Re0eV7ppP28dy91in5GqUsC/hrbjeqioJUD+EQ+KK7zLs2Q87W4o0yLvJtGWDlB7B55S+HRoI5kFQPTvlrZPnMu2H918Ub5UbtcBe/TF4wTOCFMwisL/63ld2sF8sGjievIEy3KadTe+f3xZb/mNqPqZ0fIT8Y5qr0c0kO7ijSKCexKKU3r9f/NfmhML/b/HdcOEyOSySHBHaHE1jgjuL90AnkhcJc4D4gzyWQSyK5RP790TVhrWuKEaalbSHX/bIsWMbR3wS/kRTwFzaISSUb4mLzS18vqZQGtmiDnVTk22/JhvyXb8Y+fL5D8y1YyqeiIJVfKEj43ZvYeeq95IeNxM8eJGXpa1gwN/ITygWMWRcuIT8YptPcv3JY5pRiL5ETqM8j3WeQHwwzLOMmOm2fU7gs3xLZ4G/JDalPkBcMc/XOx2gXzCDXJRZ+Y14dbsp9BZFR3kf4P6CxbS/W8G5wqcwOdwXgOFuFn3C00U4k1yWyi2R2EOnMLNYgltMY7z3PX6yxLfu5pTXsxRtyNcRSGhUFqfR2ffI4tWfeQu/cf7OBVIb55tDXv4Rcl0BetOHNcUk8GhoKGN1tBS1tc+GyXJfIbpJY4TuCpICPVP9ukgKGCyRjgVokJsTybbnsxjiWhljfiqWqiLUo6Owj8cbOn/DNupdPw8dy+Zm9qZucQFLgOHwBHw1LNLqnFU6fosZYJM5UFMQTmyfdRL1QLku7juWaUzt4HUdEompuV7x4Jrj6E9JWTWR84DwuP+t0r+OISBHaU5BDbvKy7dQJ9aDl0FtJSdSfoEhloj0FOaTWb8vh1rkB3jjifk7v0tbrOCJSgoqCHDo71rPi+d9R1+3k9rM7eZ1GREqhoiCHzE9v3kDv7e9wbZ/GtG6U4nUcESmFioIcEnkrPqLpj9N5LflCLhpwstdxRKQM6uWT+AvmkT3xT2wIN+XoYbeSGNB3EZHKSv87Je42f/gvUnPXMOPwP9PryBZexxGRfdCegsSVc447Vx9DW0ZwxcVXeR1HRMqhPQWJq8lfr2PqjwGaDPwraXWSvI4jIuVQUZC42bVkOk2nDufkljC852FexxGRGOjwkcRHQS65U/5Ck1CYm87tqUHrRKoI7SlIXGycfj+p+ev49Mib6XRYY6/jiEiMVBSkwoU2r6bR148xw/py/gXDvY4jIvtBRUEq3A9v30a+82Nn3kvd5ASv44jIflCfglSoTTtzuWzdMM5ufDK3nNjV6zgisp+0pyAVpyCXv09bzM/BZC69eDhm6lwWqWriWhTMbKCZrTCzDDMbU8ryw8xslpl9bWbfmNngeOaR+Fo7+Q5+t+xXXHtSc9o1ruN1HBE5AHErCmbmBx4HBgEdgUvNrGOJ1cYCE5xz3YBLgCfilUfiK/+n5TRb8iw/JB7ByNOO9TqOiBygeO4p9AQynHOrnXP5wGvAuSXWcUC96OP6wPo45pF4cY6N468lxyVQ9+z7SU7we51IRA5QPItCS2BtkenM6Lyi7gBGmFkmMB24trQXMrNRZpZuZulZWVnxyCoHIWvuBA7bNo/3mvyWPl2O8TqOiBwErzuaLwX+65xrBQwGXjKzvTI5555xzvVwzvVo3FgXQlUmzjnWffwcy1wbTh6+V7eRiFQx8SwK64DWRaZbRecV9RtgAoBz7gsgGUiLYyapYO8v3ciwraNZeNLTNGuozmWRqi6eReEroIOZtTWzRCIdyVNKrPMjcBqAmR1DpCjo+FAVsXvT94ybMo8jmzfkwlN7eh1HRCpA3C5ec84FzWw08D7gB553zi01s7uAdOfcFOAG4Fkzu55Ip/OvnHMuXpmkAjlH1ktX8UTuT2wf/ikBv9dHIkWkIsT1imbn3HQiHchF591W5PG3QN94ZpD4WPfJ/zh859dMbP0XhrZJ9TqOiFQQfb2T/RbevY2U2bezhCM49dK/eB1HRCqQioLst4wJt1A/tI2N/e6hQZ1aXscRkQqkAfFkv/ycncvaH1byQ8pgBpw20Os4IlLBVBRkv9z/3grezr+e6aN6acA7kWpIRUFi9t3nk5k7fzO/ObkPR7Zo5HUcEYkDFQWJSUH2Fpp8MJqHkg/nmP+70us4IhIn6miWmKwc/1fqhneSf/o91E7SdwmR6kpFQcqVtfxzjs58i4/qDaV3n1O8jiMicaSvfLJv4RC7Jv4JR32OGf53dS6LVHMqCrJPH3+7jlW72tLq2Cs4o3kzr+OISJypKEiZcvJDjJ2eQVKja5g+7CSv44jIIaA+BSlT+ku30HzrAu4+tzOJAf2piNQE2lOQUmUumsVJa5+ioPkl9G6vAe9Eagp9/ZO9uFABwWk38BONOPay+7yOIyKHkIqC7GXxpHG0KVjF8q630DhVewkiNYmKghSzY/M62i1+mK8TutHv7N94HUdEDjH1KUgx4z7NIhA8n4su/C1+3U1NpMbR/3op9E3mNl6cu45gz2s4slM3r+OIiAdUFASAUEE+4RfP4YKUhfz5jCO9jiMiHlFREAC+fvPvdC1YxAUnHE695ASv44iIR1QUhM3rv+eY5U+wIPlEep55mddxRMRDKgrC2vHX4ydE2oUPa8A7kRpORaGGWzR3Ft12zmL+Yb/msPYdvY4jIh7TKak1WF4wxPWfQI+ksdw1/Bqv44hIJaA9hRrshVlLWb15N4PPv5LkWrW9jiMilYCKQg217vsVXPzpIP7WZgX9j2ridRwRqSRUFGog5xwbJ1xPEgWcO+Qcr+OISCWiolADzf/wdbrnfMaSI0bRpPURXscRkUpERaGG2bUrm2af3c6PvlYcf/FYr+OISCWjolDDTJv8Os3dT+QOeIBAYrLXcUSkktEpqTXI8o07uGVJc9Yc9xp/7T3Q6zgiUglpT6GGCIfCPP3GNOrXSmDk2f/ndRwRqaRUFGqIz6e/xL+2/J6Hu2+mYe1Er+OISCWlolADbN22jXbz7+bHQBv6DRjmdRwRqcTiWhTMbKCZrTCzDDMbU8Y6F5nZt2a21MxejWeemurrV8bSgiwY8hC+BO0liEjZ4tbRbGZ+4HFgAJAJfGVmU5xz3xZZpwNwM9DXObfVzHRpbQVbvOgr+m16lcVpgzm22+lexxGRSi6eewo9gQzn3GrnXD7wGnBuiXVGAo8757YCOOc2xTFPjVMQCjPp/ZlstQa0u+whr+OISBUQz6LQElhbZDozOq+oI4EjzewzM/vSzEo9T9LMRplZupmlZ2VlxSlu9fPi5z/w3M/HsmjYbGo3auF1HBGpArzuaA4AHYD+wKXAs2bWoORKzrlnnHM9nHM9GjdufIgjVk0bs7JY9sF/Oe2oxgzo3NrrOCJSRcRUFMzsbTMbYmb7U0TWAUVbo1bReUVlAlOccwXOue+B74gUCTlIi1+5hYd8j3BvX5/upiYiMYu1kX8CGA6sNLP7zeyoGJ7zFdDBzNqaWSJwCTClxDqTiOwlYGZpRA4nrY4xk5Rh7rzP6L/1LZY2O49mR/bwOo6IVCExFQXn3IfOucuA44EfgA/N7HMzu8rMEsp4ThAYDbwPLAMmOOeWmtldZrZnvOb3gS1m9i0wC7jRObfl4DapZsvND5L03o3kWAodhv/T6zgiUsXEfEqqmaUCI4DLga+BV4B+wJVEv+2X5JybDkwvMe+2Io8d8Ofoj1SAj954jMHhpaw68R7a11P/i4jsn5iKgplNBI4CXgLOds5tiC563czS4xVO9s+qrGwmLsumZcNT6HLmH7yOIyJVUKx7Co8652aVtsA5p4PWlYBzjlsnLWFJ4ATuG3Uj+Lw+sUxEqqJYW46ORU8VNbOGZnZNnDLJAZj98Uy6/fAcN53RjsZ1k7yOIyJVVKxFYaRzbtueiegVyCPjE0n2146cPFI/voWRie9zSZc0r+OISBUWa1HwW5GT3aPjGmlktUrio/H/4ji3gp0n3Ya/dkOv44hIFRZrn8J7RDqVn45OXx2dJx77NmMNJ615jDV1juPw/r/2Oo6IVHGxFoWbiBSC30enPwD+E5dEErNQ2LHmzTEcabtIuvjf6lwWkYMWU1FwzoWBJ6M/Ukm8OncNE7b3oeUJ3TjusK5exxGRaiDW6xQ6AH8HOgLJe+Y759rFKZeUY9POXB54fwXHtT+BY4ee6HUcEakmYj189AJwO/Av4FTgKrwfYbVGm/XKA9wWnk+PIf/RgHciUmFibdhrOedmAuacW+OcuwMYEr9Ysi/zlqzgzA1PcWKDnbRtplNQRaTixLqnkBcdNnulmY0mMgR2nfjFkrLkB8NsmXQzdSyXWpf8G7SXICIVKNY9heuAFOCPQHciA+NdGa9QUrap0yYyKDiT9UdfRVKLTl7HEZFqptw9heiFahc75/4CZBPpTxAPrP15N22+/gc/JzTmsKF3eh1HRKqhcouCcy5kZv0ORRgpm3OO26csZY37I6+f3xqSdPRORCperH0KX5vZFOANYNeemc65t+OSSvbywTdr+Gj5T4wd0oe0jjoTWETiI9aikAxsAf6vyDwHqCgcArvygoQm/5EJdbI5vvcHXscRkWos1iua1Y/goYkTX2dE+GM2dP4DgYDf6zgiUo3FekXzC0T2DIpxzmkEtjhbvm4LPb+9j61JzWh+1liv44hINRfr4aNpRR4nA0OB9RUfR4oKhx3zxt/LFb5Mss96CRJTvI4kItVcrIeP3io6bWbjgU/jkkgKvTV/DX12vMv6ZqfQoss5XscRkRog1j2FkjoATSoyiBS3dVc+9737HZ2aPsb/RugiNRE5NGLtU9hJ8T6FjUTusSBx8uzkmeTkFjB2WH989ep5HUdEaohYDx/VjXcQ+cWC1RsYtvx6hjZuR4dmOmwkIodOTGMfmdlQM6tfZLqBmZ0Xv1g1VzAUZtGEe2nv20DrM6/zOo6I1DCxDoh3u3Nu+54J59w2IvdXkAr25szPuSTndTa2PIPkjmd6HUdEaphYi0Jp6x1oJ7WUYcP2HNI+uwOfz0fTC8d5HUdEaqBYi0K6mY0zs/bRn3HA/HgGq4kemJJOM7aQ0/sGrEFrr+OISA0Ua1G4FsgHXgdeA3KBP8QrVE00e8UmJi7dwZxTXqPBadd7HUdEaqhYzz7aBYyJc5YaK7cgxLtv/5duaUfz25M7gF+3vxYRb8R69tEHZtagyHRDM3s/frFqllfenc1duQ/wVNOpJAZUEETEO7G2QGnRM44AcM5tRVc0V4jVm3bSPv1unD9A03N1NzUR8VasRSFsZoftmTCzNpQyaqrsH+cck15/lv6+rwmeNAbqtfA6kojUcLGeVvo34FMz+xgw4CRgVNxS1RDvzM/gos2Ps7XuETQ8Wf32IuK9mPYUnHPvAT2AFcB44AYgJ465qr0duQU88t43rE3qQL1hj4I/wetIIiIxdzT/FphJpBj8BXgJuCOG5w00sxVmlmFmZZ69ZGbDzMyZWY/YYld942Z8x6pdydS54nX8bft6HUdEBIi9T+E64ARgjXPuVKAbsG1fTzAzP/A4MAjoCFxqZh1LWa9u9PXn7kfuKm1J5jbS5j3An7rCsa3ql/8EEZFDJNaikOucywUwsyTn3HLgqHKe0xPIcM6tds7lE7no7dxS1rsb+AeRC+KqvVDYMf31JxkdmMTIVmu9jiMiUkysRSEzep3CJOADM5sMrCnnOS2Boq1eZnReITM7HmjtnHtnXy9kZqPMLN3M0rOysmKMXDlN+HwZV+x4mm31O1Kr90iv44iIFBPrFc1Dow/vMLNZQH3gvYN5YzPzAeOAX8Xw/s8AzwD06NGjyp4Km7Uzj7wP76WJbcMueAN8fq8jiYgUs98jnTrnPo5x1XVA0VHdWkXn7VEX6AzMNjOAZsAUMzvHOZe+v7mqgufffocb3HSyOw2nXusTvI4jIrKXeI6p8BXQwczamlkicAkwZc9C59x251yac66Nc64N8CVQbQvC56s2879lYea3uoJ6Q+72Oo6ISKniVhScc0FgNPA+sAyY4JxbamZ3mVmNusdkfjDMrZOWkNoolS6/Gge1U72OJCJSqrjeKMc5Nx2YXmLebWWs2z+eWbz04qxF3L3tZpIG3klygvoRRKTy0t3T4mztz7tJ/uR+evmX4WurMQRFpHLTOM1x5JzjuTcmM9zeJ6fLldCiq9eRRET2SUUhjmYs3cDZ6x4iP7EBtQfe4XUcEZFyqSjEya68IHMmPUd330oSBt0LtRp6HUlEpFwqCnHy6MyVjM/uyqpT/k2g66VexxERiYk6muNgxcadvPTpCi7s0Y72p57tdRwRkZipKFSwcNjxwhtvMifxdhK7vOZ1HBGR/aKiUMHemr+GS7MepXatRGq17ux1HBGR/aI+hQq0dVc+301/jC6+1SQN+Tsk1/M6kojIflFRqECPv/MFo8OvsqtFH3zHXuB1HBGR/abDRxVk/pqtFCx6izqJufiHPgyRkV9FRKoU7SlUgGAozNhJS5hR51zyRn4Gjcu7KZ2ISOWkolABXvw0g+yNK7n97I6ktDjG6zgiIgdMh48O0obtOfw083E+SnqFQJM+QHOvI4mIHDDtKRykRyZ9wrX2OsHD+mI6bCQiVZz2FA7C7BWbODHjYVICQfznjlPnsohUedpTOEC5BSHefvs1hvo/w/X9I6S29zqSiMhBU1E4QE/MXkXj7OXk1D2cwMl/8TqOiEiFUFE4AKuzsnlq9iq2HDuSWn+cC4kpXkcSEakQKgr7yTnHw2/Npm/Ccm4Zcgwk1PI6kohIhVFH836a+s0Gzlj3KAMTFhHwXQEkex1JRKTCaE9hP+zILWDGlFc5yz8X38l/htqpXkcSEalQKgr74ZH3lnBDwbPk1WuDr+91XscREalwOnwUoyXrtlMr/QnaBjbCOW9Bgg4biUj1oz2FGITCjr9NXExBYgPyjx0OR5zudSQRkbjQnkIMxs/7kUWZ2/n1JX8isWtLr+OIiMSN9hTKkbUzj0/ee50xzRdwznEa7E5EqjftKZTjn9MWMtY9S1PqYeGbwJfodSQRkbhRUdiHL1ZtocXSp2gd2ARn/wcCKggiUr3p8FEZ8oNhnpw4g98FphLqeD60O8XrSCIicaeiUIZn56ziqu1P4g8k4h94n9dxREQOCRWFUqz9eTf/npXBspYXEDjrn1BPHcwiUjOoT6EUd05dis+M8y4ZCQ004J2I1BzaUyhhxtKNdPzuKV5qP4sW9XXVsojULHEtCmY20MxWmFmGmY0pZfmfzexbM/vGzGaa2eHxzFOe3flBXpg8g2sTJtGt7jbdXlNEapy4FQUz8wOPA4OAjsClZtaxxGpfAz2cc8cBbwIPxCtPLB758Dv+kPMUllAL3xl3exlFRMQT8dxT6AlkOOdWO+fygdeAc4uu4Jyb5ZzbHZ38EmgVxzz7tGLjTjZ8Pp5+/qUEBtwOdZp4FUVExDPxLAotgbVFpjOj88ryG+Dd0haY2SgzSzez9KysrAqMGOGc446JC7k58ArBpsdBj19X+HuIiFQFleLsIzMbAfQASr1CzDn3DPAMQI8ePVxFv/+b8zP5Ys0Ovjn9Kc7s1Ax8/op+CxGRKiGeRWEd0LrIdKvovGLM7HTgb8Apzrm8OOYp1dZd+fxz+mK6H96QAf/XG3zqXBaRmiueReEroIOZtSVSDC4Bhhddwcy6AU8DA51zm+KYpUwPvLecR4J30aFJH3y+Pl5EEBGpNOLWp+CcCwKjgfeBZcAE59xSM7vLzM6JrvYgUAd4w8wWmtmUeOUpzYIft5I7/1V6+ZaR2ua4Q/nWIiKVUlz7FJxz04HpJebdVuSxZ7cwC4bC3PfWlzyT+CqhFt3xd7vCqygiIpVGpeho9sKLX6xhyJYXaBjYiZ01Dny6uFuksigoKCAzM5Pc3Fyvo1Q5ycnJtGrVioSEhAN6fo0sChu35/LsjAXMTPwMuv8GWnT1OpKIFJGZmUndunVp06YNppEFYuacY8uWLWRmZtK2bdsDeo0aWRTunvYtW8MpbLvyE2o3aeR1HBEpITc3VwXhAJgZqampHMz1XDXumMnH32XxzZKFjO7fnpaHt4NaDbyOJCKlUEE4MAf7e6tRRSG3IMSDk75gWvJt/D7/Oa/jiIhUOjWqKDwxexWX7vgv9dhN4PjLvY4jIpXUtm3beOKJJw7ouYMHD2bbtm0VnOjQqTFF4fvNu/js4xlcGvgIO/FqaNbZ60giUkntqygEg8F9Pnf69Ok0aFB1D0vXmI7m9xZnclfgOVztJtD/Zq/jiEiM7py6lG/X76jQ1+zYoh63n92pzOVjxoxh1apVdO3alQEDBjBkyBBuvfVWGjZsyPLly/nuu+8477zzWLt2Lbm5uVx33XWMGjUKgDZt2pCenk52djaDBg2iX79+fP7557Rs2ZLJkydTq1bxuzlOnTqVe+65h/z8fFJTU3nllVdo2rQp2dnZXHvttaSnp2Nm3H777QwbNoz33nuPW265hVAoRFpaGjNnzqzQ302NKQq/75pEaEE+/jPug+R6XscRkUrs/vvvZ8mSJSxcuBCA2bNns2DBApYsWVJ4qufzzz9Po0aNyMnJ4YQTTmDYsGGkpqYWe52VK1cyfvx4nn32WS666CLeeustRowYUWydfv368eWXX2Jm/Oc//+GBBx7goYce4u6776Z+/fosXrwYgK1bt5KVlcXIkSOZM2cObdu25eeff67wba8xRYGGbfBf+xUk6J7LIlXJvr7RH0o9e/Ysdu7/o48+ysSJEwFYu3YtK1eu3KsotG3blq5dI9dBde/enR9++GGv183MzOTiiy9mw4YN5OfnF77Hhx9+yGuvvVa4XsOGDZk6dSonn3xy4TqNGlX8KfU1pk8BgMQU3WJTRA5I7dq1Cx/Pnj2bDz/8kC+++IJFixbRrVu3Uq++TkpKKnzs9/tL7Y+49tprGT16NIsXL+bpp5/2/CrumlUURERiULduXXbu3Fnm8u3bt9OwYUNSUlJYvnw5X3755QG/1/bt22nZMnL/sRdffLFw/oABA3j88ccLp7du3UqvXr2YM2cO33//PUBcDh+pKIiIlJCamkrfvn3p3LkzN954417LBw4cSDAY5JhjjmHMmDH06tW7VrZdAAAJ/ElEQVTrgN/rjjvu4MILL6R79+6kpaUVzh87dixbt26lc+fOdOnShVmzZtG4cWOeeeYZzj//fLp06cLFF198wO9bFnOuwm9kFlc9evRw6enpXscQkThatmwZxxxzjNcxqqzSfn9mNt8516O852pPQURECqkoiIhIIRUFEREppKIgIiKFVBRERKSQioKIiBRSURARKeFghs4GePjhh9m9e3cFJjp0VBREREqoyUWh5gyIJyJV1wtD9p7X6TzoORLyd8MrF+69vOtw6HYZ7NoCE64ovuyqd/b5diWHzn7wwQd58MEHmTBhAnl5eQwdOpQ777yTXbt2cdFFF5GZmUkoFOLWW2/lp59+Yv369Zx66qmkpaUxa9asYq991113MXXqVHJycujTpw9PP/00ZkZGRga/+93vyMrKwu/388Ybb9C+fXv+8Y9/8PLLL+Pz+Rg0aBD333///v729ouKgohICSWHzp4xYwYrV65k3rx5OOc455xzmDNnDllZWbRo0YJ33okUme3bt1O/fn3GjRvHrFmzig1bscfo0aO57bbbALj88suZNm0aZ599Npdddhljxoxh6NCh5ObmEg6Heffdd5k8eTJz584lJSUlLmMdlaSiICKV376+2Sem7Ht57dRy9wzKM2PGDGbMmEG3bt0AyM7OZuXKlZx00knccMMN3HTTTZx11lmcdNJJ5b7WrFmzeOCBB9i9ezc///wznTp1on///qxbt46hQ4cCkJycDESGz77qqqtISUkB4jNUdkkqCiIi5XDOcfPNN3P11VfvtWzBggVMnz6dsWPHctpppxXuBZQmNzeXa665hvT0dFq3bs0dd9zh+VDZJamjWUSkhJJDZ5955pk8//zzZGdnA7Bu3To2bdrE+vXrSUlJYcSIEdx4440sWLCg1OfvsacApKWlkZ2dzZtvvlm4fqtWrZg0aRIAeXl57N69mwEDBvDCCy8Udlrr8JGIiAeKDp09aNAgHnzwQZYtW0bv3r0BqFOnDi+//DIZGRnceOON+Hw+EhISePLJJwEYNWoUAwcOpEWLFsU6mhs0aMDIkSPp3LkzzZo144QTTihc9tJLL3H11Vdz2223kZCQwBtvvMHAgQNZuHAhPXr0IDExkcGDB3PffffFdds1dLaIVDoaOvvgaOhsERGpECoKIiJSSEVBRCqlqnZou7I42N+bioKIVDrJycls2bJFhWE/OefYsmVL4XUOB0JnH4lIpdOqVSsyMzPJysryOkqVk5ycTKtWrQ74+SoKIlLpJCQk0LZtW69j1EhxPXxkZgPNbIWZZZjZmFKWJ5nZ69Hlc82sTTzziIjIvsWtKJiZH3gcGAR0BC41s44lVvsNsNU5dwTwL+Af8cojIiLli+eeQk8gwzm32jmXD7wGnFtinXOBF6OP3wROMzOLYyYREdmHePYptATWFpnOBE4sax3nXNDMtgOpwOaiK5nZKGBUdDLbzFYcYKa0kq9dhWlbKp/qsh2gbamsDmZbDo9lpSrR0eycewZ45mBfx8zSY7nMuyrQtlQ+1WU7QNtSWR2KbYnn4aN1QOsi062i80pdx8wCQH1gSxwziYjIPsSzKHwFdDCztmaWCFwCTCmxzhTgyujjC4CPnK5WERHxTNwOH0X7CEYD7wN+4Hnn3FIzuwtId85NAZ4DXjKzDOBnIoUjng76EFQlom2pfKrLdoC2pbKK+7ZUuaGzRUQkfjT2kYiIFFJREBGRQtWyKFSn4TVi2JZfmVmWmS2M/vzWi5zlMbPnzWyTmS0pY7mZ2aPR7fzGzI4/1BljFcO29Dez7UU+k7Lv5O4hM2ttZrPM7FszW2pm15WyTpX4XGLclqryuSSb2TwzWxTdljtLWSd+bZhzrlr9EOnUXgW0AxKBRUDHEutcAzwVfXwJ8LrXuQ9iW34FPOZ11hi25WTgeGBJGcsHA+8CBvQC5nqd+SC2pT8wzeucMWxHc+D46OO6wHel/H1Vic8lxm2pKp+LAXWijxOAuUCvEuvErQ2rjnsK1Wl4jVi2pUpwzs0hcoZZWc4F/ucivgQamFnzQ5Nu/8SwLVWCc26Dc25B9PFOYBmRUQaKqhKfS4zbUiVEf9fZ0cmE6E/JM4Li1oZVx6JQ2vAaJf84ig2vAewZXqOyiWVbAIZFd+3fNLPWpSyvCmLd1qqid3T3/10z6+R1mPJEDz90I/KttKgq97nsY1uginwuZuY3s4XAJuAD51yZn0tFt2HVsSjUNFOBNs6544AP+OXbg3hnAXC4c64L8G9gksd59snM6gBvAX9yzu3wOs/BKGdbqszn4pwLOee6EhkJoqeZdT5U710di0J1Gl6j3G1xzm1xzuVFJ/8DdD9E2SpaLJ9bleCc27Fn9985Nx1IMLM0j2OVyswSiDSirzjn3i5llSrzuZS3LVXpc9nDObcNmAUMLLEobm1YdSwK1Wl4jXK3pcTx3XOIHEutiqYAV0TPdukFbHfObfA61IEws2Z7ju+aWU8i/88q3ZeOaMbngGXOuXFlrFYlPpdYtqUKfS6NzaxB9HEtYACwvMRqcWvDqsQoqfvDVc7hNQ5IjNvyRzM7BwgS2ZZfeRZ4H8xsPJGzP9LMLBO4nUgHGs65p4DpRM50yQB2A1d5k7R8MWzLBcDvzSwI5ACXVNIvHX2By4HF0ePXALcAh0GV+1xi2Zaq8rk0B160yI3KfMAE59y0Q9WGaZgLEREpVB0PH4mIyAFSURARkUIqCiIiUkhFQURECqkoiIhIIRUFkTiLjs45zescIrFQURARkUIqCiJRZjYiOo79QjN7OjooWbaZ/Ss6rv1MM2scXbermX0ZHYhwopk1jM4/wsw+jA66tsDM2kdfvk50wMLlZvZKkStr74/eA+AbM/unR5suUkhFQQQws2OAi4G+0YHIQsBlQG0iV5F2Aj4mcvUywP+Am6IDES4uMv8V4PHooGt9gD1DQnQD/gR0JHJ/jL5mlgoMBTpFX+ee+G6lSPlUFEQiTiMymOBX0WESTiPSeIeB16PrvAz0M7P6QAPn3MfR+S8CJ5tZXaClc24igHMu1zm3O7rOPOdcpnMuDCwE2hAZ7jgXeM7MzicyjISIp1QURCIMeNE51zX6c5Rz7o5S1jvQcWHyijwOAYHoOPg9idwk5SzgvQN8bZEKo6IgEjETuMDMmgCYWSMzO5zI/5ELousMBz51zm0HtprZSdH5lwMfR+/4lWlm50VfI8nMUsp6w+jY//WjwzhfD3SJx4aJ7I9qN0qqyIFwzn1rZmOBGWbmAwqAPwC7iNzkZCyRu2BdHH3KlcBT0UZ/Nb+MHno58HR0RMsC4MJ9vG1dYLKZJRPZU/lzBW+WyH7TKKki+2Bm2c65Ol7nEDlUdPhIREQKaU9BREQKaU9BREQKqSiIiEghFQURESmkoiAiIoVUFEREpND/A51hU74+dobzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "markers = {'train': 'o', 'test': 's'}\n",
    "x = np.arange(len(train_acc_list))\n",
    "\n",
    "plt.plot(x, train_acc_list, label='train acc')\n",
    "plt.plot(x, test_acc_list, label='test acc', linestyle='--')\n",
    "plt.xlabel(\"epochs\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.ylim(0, 1.0)\n",
    "plt.legend(loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实验小结\n",
    "\n",
    "在本实验中，你实现了二层神经网络误差反向传播法。了解了神经网络中的误差反向传播法，并以层为单位实现了神经网络中的处理。同时，在 ReLU 层、Softmax-with-Loss 层、 Afne 层、Softmax 层中实现了 forward 和 backward 方法。通过将数据正向和反向地传播，可以高效地计算权重参数的梯度。通过使用层进行模块化，神经网络中可以自由地组装层，轻松构建出自己喜欢的网络。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}