9.模型训练.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "10bb295f",
   "metadata": {},
   "source": [
    "## 9.模型训练\n",
    "\n",
    "**学习目标**\n",
    "\n",
    "1. 熟悉模型训练的流程和方法\n",
    "\n",
    "2. 熟悉常用的损失函数和优化器\n",
    "\n",
    "3. 能够选择合适的超参数训练出高精度模型\n",
    "\n",
    "4. 熟练使用torchvision库\n",
    "\n",
    "5. 会使用torch.save保存pth模型\n",
    "\n",
    "****"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb514f9d",
   "metadata": {},
   "source": [
    "### 9.1 模型训练的基本步骤\n",
    "\n",
    "训练一个卷积神经网络模型通常包括以下步骤：\n",
    "\n",
    "（1）数据准备\n",
    "\n",
    "准备数据集：收集并准备训练数据，包括图像、标注等。对数据进行预处理，如缩放、归一化、数据增强等。\n",
    "\n",
    "划分数据集：将数据集划分为训练集、验证集和测试集。\n",
    "\n",
    "（2）模型构建\n",
    "\n",
    "定义模型结构：设计CNN的架构，包括卷积层、池化层、全连接层、激活函数等。\n",
    "\n",
    "正则化技术：应用如Dropout、L2正则化等技术减少过拟合。\n",
    "\n",
    "选择损失函数：根据任务类型选择合适的损失函数，如交叉熵损失（用于分类任务）或均方误差损失（用于回归任务）。\n",
    "\n",
    "选择优化器：选择一个优化算法，如SGD（随机梯度下降）、Adam、RMSprop等。\n",
    "\n",
    "设置学习率和超参数：设置学习率、批大小、训练轮数（epochs）、正则化参数等超参数。\n",
    "\n",
    "\n",
    "（3）模型训练\n",
    "\n",
    "初始化模型权重：使用适当的方法初始化模型的权重，如随机初始化或使用预训练模型的权重。\n",
    "\n",
    "前向传播：在训练过程中，输入数据通过CNN模型进行前向传播，计算预测结果。\n",
    "\n",
    "计算损失：使用损失函数计算模型预测与真实标签之间的差异。\n",
    "\n",
    "反向传播：根据损失计算梯度，并通过反向传播算法更新模型的权重。\n",
    "\n",
    "优化和调整：在训练过程中监控模型性能，使用验证集评估模型，调整超参数和模型结构。\n",
    "\n",
    "（4）模型测试\n",
    "\n",
    "模型评估：在测试集上评估模型的性能，使用各种指标如准确率、精确率、召回率等。\n",
    "\n",
    "模型调优：根据评估结果对模型进行进一步的调优，如超参数调整、模型结构改进等。\n",
    "\n",
    "保存和部署：训练完成后，保存模型的权重和结构，以便将来的部署和使用。\n",
    "\n",
    "这些步骤构成了训练CNN模型的基本流程，实际操作中可能需要根据具体任务和数据集进行调整。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc4174d2",
   "metadata": {},
   "source": [
    "### 9.2 蝴蝶与飞蛾数据集\n",
    "\n",
    "蝴蝶是一类美丽的昆虫，属于鳞翅目（Lepidoptera）的蝴蝶科（Papilionidae）。它们以其丰富多彩的翅膀和优雅的飞行姿态而闻名。以下是关于蝴蝶的一些基本介绍：\n",
    "\n",
    "形态特征：蝴蝶的翅膀通常覆盖着细小的鳞片，这些鳞片形成了各种图案和颜色。它们的身体细长，通常有三对足和一对触角。\n",
    "\n",
    "生命周期：蝴蝶的生命周期包括四个阶段：卵、幼虫（毛虫）、蛹和成虫。从卵孵化成幼虫后，毛虫会大量进食并经历几次蜕皮，之后形成蛹。在蛹阶段，它们会经历变态，最终变成成虫。\n",
    "\n",
    "食性：成虫蝴蝶主要以花蜜为食，而毛虫则根据种类不同，可能以特定植物的叶子为食。\n",
    "\n",
    "繁殖：蝴蝶通过飞行和寻找合适的植物产卵来繁殖。雌蝴蝶会选择适合其幼虫生长的植物产卵。\n",
    "\n",
    "生态作用：蝴蝶在生态系统中扮演着重要角色，包括作为传粉者帮助植物繁殖，以及作为食物链中的一环，为其他动物提供食物来源。\n",
    "\n",
    "种类多样性：全世界已知的蝴蝶种类超过17,500种，它们分布在各种不同的生态环境中，从热带雨林到寒带地区。\n",
    "\n",
    "保护问题：由于栖息地破坏、气候变化和农药使用等原因，一些蝴蝶种类面临生存威胁。保护蝴蝶的自然栖息地和生物多样性对于维护生态平衡至关重要。\n",
    "\n",
    "文化意义：在不同的文化中，蝴蝶常常象征着变革、美丽、自由和灵魂。它们也常被用于艺术和文学作品中。\n",
    "\n",
    "蝴蝶是自然界中的一个迷人现象，它们的多样性和生态作用提醒我们要珍惜和保护自然环境。\n",
    "\n",
    "由于蝴蝶种类如此繁多，我们很难知道一种蝴蝶到底是什么品种的，现在我们想让计算机帮助我们识别蝴蝶，因此我们制作了蝴蝶数据集，并设计了识别模型。\n",
    "\n",
    "<img src=\"./images/butterfly.jpg\" style=\"zoom:100%;\" />\n",
    "\n",
    "数据集：\n",
    "\n",
    "100种蝴蝶或飞蛾的数据集。所有图像已经处理为 224 X 224 X 3的jpg格式。\n",
    "训练集由 12594 张图像组成，分为 100 个子目录，每个种类的蝴蝶或飞蛾一个子目录。\n",
    "测试集由 500 张图像组成，分为 100 个子目录，每个种类的蝴蝶或飞蛾有 5 个测试图像。\n",
    "验证集由 500 张图像组成，分为 100 个子目录，每个种类的蝴蝶或飞蛾有 5 个验证图像。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde98bef",
   "metadata": {},
   "source": [
    "### 9.3  ResNet18模型训练——蝴蝶品种识别\n",
    "\n",
    "1. **导入必需的模块**\n",
    "\n",
    "torchvision是PyTorch的一个图形库，主要用来构建计算机视觉模型。torchvision包括以下的部分：\n",
    "\n",
    "- torchvision.datasets： 加载数据的函数及常用的数据集接口；\n",
    "\n",
    "- torchvision.models：  包含常用的模型结构（含预训练模型）；\n",
    "\n",
    "- torchvision.transforms：常用的图片变换，例如裁剪、旋转等；\n",
    "\n",
    "- torchvision.utils：其他的一些有用的方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "188817fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from matplotlib import pyplot as plt\n",
    "from models import resnet18\n",
    "import torch\n",
    "from torch import nn\n",
    "from torchvision import transforms, datasets\n",
    "from torch.utils.data import DataLoader, random_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "765d59a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa4b600",
   "metadata": {},
   "source": [
    "2. **数据加载和预处理**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4a0084d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_transform = transforms.Compose([\n",
    "    transforms.Resize((224,224)),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomVerticalFlip(),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(\n",
    "        mean=[0.485, 0.456, 0.406],\n",
    "        std=[0.229, 0.224, 0.225]\n",
    "    )\n",
    "])\n",
    "\n",
    "val_transform = transforms.Compose([\n",
    "    transforms.Resize((224,224)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(\n",
    "        mean=[0.485, 0.456, 0.406],\n",
    "        std=[0.229, 0.224, 0.225]\n",
    "    )\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7108ed50",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = datasets.ImageFolder('./datasets/butterflies and moths/train', train_transform)\n",
    "val_dataset = datasets.ImageFolder('./datasets/butterflies and moths/valid', val_transform)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4f1ed145",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataloader = DataLoader(train_dataset, 64, shuffle=True, num_workers=4, pin_memory=True)\n",
    "val_dataloader = DataLoader(val_dataset, 64, shuffle=False, num_workers=4, pin_memory=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f54b466",
   "metadata": {},
   "source": [
    "至此，数据集加载和预处理的定义就完成了，接下来进行的就是模型的训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10b2693",
   "metadata": {},
   "source": [
    "3.参数设置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3732711a",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 10\n",
    "learning_rate = 0.01\n",
    "momentum = 0.9\n",
    "\n",
    "num_category = len(train_dataset.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c05de8c4",
   "metadata": {},
   "source": [
    "4. **模型构建**\n",
    "\n",
    "在上一个任务中，构建了一个ResNet18模型，我们可以使用上述数据集来训练已构建好的ResNet18模型。模型放在了`./models/resnet18.py`文件中，可以直接调用该模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b6f4c743",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))\n",
      "  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (2): ReLU()\n",
      "  (3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
      "  (4): ResidualBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential()\n",
      "  )\n",
      "  (5): ResidualBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential()\n",
      "  )\n",
      "  (6): ResidualBlock(\n",
      "    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential(\n",
      "      (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2))\n",
      "      (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (7): ResidualBlock(\n",
      "    (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential()\n",
      "  )\n",
      "  (8): ResidualBlock(\n",
      "    (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential(\n",
      "      (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2))\n",
      "      (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (9): ResidualBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential()\n",
      "  )\n",
      "  (10): ResidualBlock(\n",
      "    (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential(\n",
      "      (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2))\n",
      "      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (11): ResidualBlock(\n",
      "    (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (shortcut): Sequential()\n",
      "  )\n",
      "  (12): AdaptiveAvgPool2d(output_size=(1, 1))\n",
      "  (13): Flatten(start_dim=1, end_dim=-1)\n",
      "  (14): Linear(in_features=512, out_features=100, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = resnet18.resnet18(num_classes=num_category)\n",
    "model = model.to(device)\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004a6f3f",
   "metadata": {},
   "source": [
    "5. **实例化损失函数（loss function）**\n",
    "\n",
    "在利用深度神经网络做图像分类问题时，经常使用交叉熵作为损失函数，由于交叉熵涉及到计算每个类别的概率，所以交叉熵几乎每次都和sigmoid(或softmax)函数一起出现。我们用神经网络最后一层输出的情况，来看整个模型预测、获得损失和学习的流程：\n",
    "\n",
    "- 神经网络最后一层得到每个类别的得分scores（也叫logits）；\n",
    "- 该得分经过sigmoid(或softmax)函数获得概率输出；\n",
    "- 模型预测的类别概率输出与真实类别的one hot形式进行交叉熵损失函数的计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "64e5b719",
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_func = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "218f43b7",
   "metadata": {},
   "source": [
    "6. **设置优化器（optimizer）**\n",
    "\n",
    "​\t\t神经网络优化器，主要是为了优化神经网络，使训练过程快起来，节省网络训练的时间。torch.optim内置了各种优化算法。要使用torch.optim，必须构造一个optimizer对象，这个对象能保存当前的参数状态并且基于计算梯度更新参数。要构造一个optimizer，必须给它一个包含参数（必须都是Variable对象）进行优化。我们可以指定optimizer的相关参数，比如学习率，权重衰减等。\n",
    "\n",
    "​\t\t普通的梯度下降算法，需要不断重复把训练集中所有的数据放入神经网络进行训练，这样会消耗大量的计算资源。使用SGD会把数据拆分后分批放入神经网络中计算。使用批数据很大程度上加速了神经网络的训练过程, 而且不会损失太多精度。\n",
    "\n",
    "​\t\t模型前面的特征提取网络的参数是通过 ImageNet 上的预训练模型获得的，只需使⽤较小的学习率进⾏微调。相⽐之下，输出层model.fc中的参数是随机初始化的，通常需要更⾼的学习率才能从头开始训练。在我们的项目代码里面，没有区分特征提取网络和输出层的学习率，统一设置为lr=0.001。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c54950ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum, weight_decay=0, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ee8cd4",
   "metadata": {},
   "source": [
    "7. **模型训练**\n",
    "\n",
    "我们直接调用内置的训练函数 train_cls和验证函数valid_cls。对于train_cls来说，第1个参数为训练的模型，第2个参数为加载的训练数据集，第3个参数loss_func为使用的损失函数，第4个参数为设置的优化器，第5个参数为训练时是否调用GPU。同样，valid_cls参数类似，不过因为验证时不需要改变网络参数，所以valid_cls函数就没有优化器的参数。我们训练模型的同时，可以打印训练过程中的关键指标，包括训练的周期数epoch、训练耗费的时长、训练loss和验证loss等来观察训练的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5ec77a5f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "周期数 1, 耗时 23.17 秒, 训练loss 3.2111, 验证loss 2.6127\n",
      "周期数 2, 耗时 22.43 秒, 训练loss 1.9700, 验证loss 1.6913\n",
      "周期数 3, 耗时 22.33 秒, 训练loss 1.4132, 验证loss 1.3495\n",
      "周期数 4, 耗时 22.80 秒, 训练loss 1.0840, 验证loss 1.4905\n",
      "周期数 5, 耗时 22.65 秒, 训练loss 0.8957, 验证loss 1.1835\n",
      "周期数 6, 耗时 22.50 秒, 训练loss 0.7623, 验证loss 1.0181\n",
      "周期数 7, 耗时 22.44 秒, 训练loss 0.6609, 验证loss 1.0328\n",
      "周期数 8, 耗时 22.29 秒, 训练loss 0.5811, 验证loss 0.9921\n",
      "周期数 9, 耗时 21.49 秒, 训练loss 0.5047, 验证loss 0.9186\n",
      "周期数 10, 耗时 21.57 秒, 训练loss 0.4538, 验证loss 1.1516\n",
      "周期数 11, 耗时 21.77 秒, 训练loss 0.4098, 验证loss 0.9643\n",
      "周期数 12, 耗时 21.43 秒, 训练loss 0.3695, 验证loss 0.6082\n",
      "周期数 13, 耗时 21.54 秒, 训练loss 0.3323, 验证loss 0.5620\n",
      "周期数 14, 耗时 21.60 秒, 训练loss 0.2871, 验证loss 0.5816\n",
      "周期数 15, 耗时 21.53 秒, 训练loss 0.2604, 验证loss 0.6201\n",
      "周期数 16, 耗时 21.49 秒, 训练loss 0.2272, 验证loss 0.6104\n",
      "周期数 17, 耗时 21.81 秒, 训练loss 0.2115, 验证loss 0.5245\n",
      "周期数 18, 耗时 21.52 秒, 训练loss 0.1928, 验证loss 0.5155\n",
      "周期数 19, 耗时 21.63 秒, 训练loss 0.1619, 验证loss 0.5991\n",
      "周期数 20, 耗时 21.80 秒, 训练loss 0.1447, 验证loss 0.4527\n",
      "周期数 21, 耗时 21.61 秒, 训练loss 0.1416, 验证loss 0.5416\n",
      "周期数 22, 耗时 21.53 秒, 训练loss 0.1286, 验证loss 0.7450\n",
      "周期数 23, 耗时 21.42 秒, 训练loss 0.1045, 验证loss 0.5077\n",
      "周期数 24, 耗时 21.54 秒, 训练loss 0.0901, 验证loss 0.4519\n",
      "周期数 25, 耗时 21.82 秒, 训练loss 0.0786, 验证loss 0.4795\n",
      "周期数 26, 耗时 21.65 秒, 训练loss 0.0647, 验证loss 0.4317\n",
      "周期数 27, 耗时 21.57 秒, 训练loss 0.0507, 验证loss 0.4656\n",
      "周期数 28, 耗时 21.72 秒, 训练loss 0.0527, 验证loss 0.3990\n",
      "周期数 29, 耗时 21.67 秒, 训练loss 0.0500, 验证loss 0.4710\n",
      "周期数 30, 耗时 21.74 秒, 训练loss 0.0451, 验证loss 0.4658\n"
     ]
    }
   ],
   "source": [
    "train_losses = []\n",
    "val_losses = []\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    since = time.time()\n",
    "    train_loss = 0\n",
    "    val_loss = 0\n",
    "    \n",
    "    # 模型训练\n",
    "    model.train()\n",
    "    for i, (data, labels) in enumerate(train_dataloader):\n",
    "        data, labels = data.to(device), labels.to(device)\n",
    "        \n",
    "        outputs = model(data)\n",
    "        loss = loss_func(outputs, labels.long())\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        train_loss += loss.item()\n",
    "    \n",
    "    train_loss /= len(train_dataloader)\n",
    "    train_losses.append(train_loss)\n",
    "    \n",
    "    # 模型验证\n",
    "    model.eval()\n",
    "    for i, (data, labels) in enumerate(val_dataloader):\n",
    "        data, labels = data.to(device), labels.to(device)\n",
    "\n",
    "        outputs = model(data)\n",
    "        loss = loss_func(outputs, labels.long())\n",
    "        \n",
    "        val_loss += loss.item()\n",
    "    \n",
    "    val_loss /= len(val_dataloader)\n",
    "    val_losses.append(val_loss)\n",
    "    \n",
    "    print(f'周期数 {epoch+1}, 耗时 {time.time()-since:.2f} 秒, 训练loss {train_loss:.5f}, 验证loss {val_loss:.5f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9daa4a7f",
   "metadata": {},
   "source": [
    "8. **绘制损失曲线**\n",
    "\n",
    "对于训练过程中的loss曲线的变化，我们可以将它可视化出来。loss曲线对于模型训练情况的评估非常重要，也是我们判断模型训练是否欠拟合或者过拟合的依据。我们使用matplotlib来绘制loss曲线,并将绘制的学习曲线显示出来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c2c27191",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiUAAAGzCAYAAADwumcoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABkL0lEQVR4nO3dd3hUZf7+8ffMpPeE9BAg9CpNwAACClJUFHUV2yJ2XVCx7Yq/tX9XXF0Ve11lLSiigh1FlCJFpcRCLwECJKGm98z5/XGSISEJpEwyk+R+Xddcc3LmzDmfjKO5fc5TLIZhGIiIiIi4mNXVBYiIiIiAQomIiIi4CYUSERERcQsKJSIiIuIWFEpERETELSiUiIiIiFtQKBERERG3oFAiIiIibkGhRERERNyCQomIm5ozZw4Wi4Xdu3e7upSTWrp0KRaLhaVLl7q6FBFp5hRKRERExC14uLoAEWneRowYQX5+Pl5eXq4uRUSaObWUiEglubm5dTrearXi4+OD1dry/nNiGAb5+fmuLkOk1Wh5/xURaeG++eYbzjzzTPz9/QkMDOS8885j48aNlY75/fffmTp1Kh07dsTHx4fo6Giuu+46jhw5Uum4hx9+GIvFwqZNm7jyyisJDQ1l+PDhAHTo0IHzzz+fn376icGDB+Pj40PHjh155513Kp2juj4lo0aNonfv3mzatImzzjoLPz8/4uLiePLJJ6v8Pnv27OGCCy7A39+fyMhI7rzzTr799tta91PZv38/119/PbGxsXh7e5OQkMCtt95KUVFRpd/xRNX12Sn/nb/99ltOP/10fH19ee211+jduzdnnXVWlXPY7Xbi4uL4y1/+Umnf7Nmz6dWrFz4+PkRFRXHzzTdz7NixU/4uIq2dbt+INCPvvvsu11xzDePGjePf//43eXl5vPLKKwwfPpwNGzbQoUMHABYvXsyuXbu49tpriY6OZuPGjbz++uts3LiRNWvWVPkjfemll9KlSxcef/xxDMNw7N+xYwd/+ctfuP7667nmmmt46623mDp1KgMHDqRXr14nrfXYsWOMHz+eiy++mMsuu4yPP/6Yf/zjH/Tp04cJEyYAZqvM2WefTWpqKnfccQfR0dHMnTuXH3/8sVafx4EDBxg8eDAZGRncdNNNdO/enf379/Pxxx+Tl5dXr1tKW7du5YorruDmm2/mxhtvpFu3bkyePJmHH36YtLQ0oqOjHcf+9NNPHDhwgMsvv9yx7+abb2bOnDlce+213H777SQnJ/Piiy+yYcMGVq5ciaenZ51rEmk1DBFxS2+//bYBGMnJyYZhGEZ2drYREhJi3HjjjZWOS0tLM4KDgyvtz8vLq3K+Dz74wACM5cuXO/Y99NBDBmBcccUVVY5v3759leMPHjxoeHt7G3fffbdj348//mgAxo8//ujYN3LkSAMw3nnnHce+wsJCIzo62rjkkksc+55++mkDMBYuXOjYl5+fb3Tv3r3KOaszZcoUw2q1Gr/++muV1+x2e6Xf8UQnfr4Vf+dFixZVOnbr1q0GYLzwwguV9v/tb38zAgICHJ/3ihUrDMB4//33Kx23aNGiaveLSGW6fSPSTCxevJiMjAyuuOIKDh8+7HjYbDaGDBlSqXXB19fXsV1QUMDhw4c544wzAFi/fn2Vc99yyy3VXrNnz56ceeaZjp8jIiLo1q0bu3btOmW9AQEBXH311Y6fvby8GDx4cKX3Llq0iLi4OC644ALHPh8fH2688cZTnt9ut7Nw4UImTpzI6aefXuX16m7Z1EZCQgLjxo2rtK9r167069ePefPmOfaVlpby8ccfM3HiRMfnPX/+fIKDgznnnHMq/TMaOHAgAQEBtW4BEmmtdPtGpJnYvn07AGeffXa1rwcFBTm2jx49yiOPPMKHH37IwYMHKx2XmZlZ5b0JCQnVnrNdu3ZV9oWGhtaqf0Tbtm2rBIPQ0FB+//13x8979uyhU6dOVY7r3LnzKc9/6NAhsrKy6N279ymPrYuaPovJkydz//33s3//fuLi4li6dCkHDx5k8uTJjmO2b99OZmYmkZGR1Z7jxH8WIlKZQolIM2G32wGzX0nFfg3lPDyO/+t82WWXsWrVKu6991769etHQEAAdrud8ePHO85TUcWWlYpsNlu1+40K/U5q0pD3OlNNLSalpaXV7q/ps5g8eTIzZ85k/vz5zJgxg48++ojg4GDGjx/vOMZutxMZGcn7779f7TkiIiLqWL1I66JQItJMdOrUCYDIyEjGjBlT43HHjh1jyZIlPPLIIzz44IOO/eUtLe6kffv2bNq0CcMwKoWHHTt2nPK9ERERBAUF8eeff570uNDQUAAyMjIICQlx7N+zZ0+dak1ISGDw4MHMmzeP6dOn8+mnnzJp0iS8vb0dx3Tq1Invv/+eYcOG1RhuRKRm6lMi0kyMGzeOoKAgHn/8cYqLi6u8fujQIeB4C8WJLRKzZ89u9Brraty4cezfv5/PP//csa+goIA33njjlO+1Wq1MmjSJL774grVr11Z5vfz3Lw9zy5cvd7yWm5vL//73vzrXO3nyZNasWcNbb73F4cOHK926AbOFqrS0lMcee6zKe0tKSsjIyKjzNUVaE7WUiDQTQUFBvPLKK/z1r39lwIABXH755URERLB3716++uorhg0bxosvvkhQUBAjRozgySefpLi4mLi4OL777juSk5Nd/StUcfPNN/Piiy9yxRVXcMcddxATE8P777+Pj48PcOrOqo8//jjfffcdI0eO5KabbqJHjx6kpqYyf/58fvrpJ0JCQhg7dizt2rXj+uuv595778Vms/HWW285Pru6uOyyy7jnnnu45557CAsLq9JiNXLkSG6++WZmzZpFUlISY8eOxdPTk+3btzN//nyee+65SnOaiEhlCiUizciVV15JbGwsTzzxBE899RSFhYXExcVx5plncu211zqOmzt3LrfddhsvvfQShmEwduxYvvnmG2JjY11YfVUBAQH88MMP3HbbbTz33HMEBAQwZcoUhg4dyiWXXOIIJzWJi4vj559/5oEHHuD9998nKyuLuLg4JkyYgJ+fHwCenp4sWLCAv/3tbzzwwANER0czY8YMQkNDK31mtdG2bVuGDh3KypUrueGGG6qdc+TVV19l4MCBvPbaa9x///14eHjQoUMHrr76aoYNG1an64m0NhajqXudiYicwuzZs7nzzjvZt28fcXFxri5HRJqIQomIuFR+fn6VeVX69+9PaWkp27Ztc2FlItLUdPtGRFzq4osvpl27dvTr14/MzEzee+89tmzZUuOwWhFpuRRKRMSlxo0bx5tvvsn7779PaWkpPXv25MMPP6wyskVEWj7dvhERERG3oHlKRERExC0olIiIiIhbaBZ9Sux2OwcOHCAwMLDeK3+KiIhI0zIMg+zsbGJjY7FaT90O0ixCyYEDB4iPj3d1GSIiIlIPKSkptG3b9pTHNYtQEhgYCJi/VMXl2UVERMR9ZWVlER8f7/g7firNIpSU37IJCgpSKBEREWlmatv1Qh1dRURExC0olIiIiIhbUCgRERERt9As+pSIiEjLZRgGJSUllJaWuroUqSObzYaHh4fTputQKBEREZcpKioiNTWVvLw8V5ci9eTn50dMTAxeXl4NPpdCiYiIuITdbic5ORmbzUZsbCxeXl6aILMZMQyDoqIiDh06RHJyMl26dKnVBGkno1AiIiIuUVRUhN1uJz4+Hj8/P1eXI/Xg6+uLp6cne/bsoaioCB8fnwadTx1dRUTEpRr6f9fiWs7856dvgoiIiLgFhRIRERFxCwolIiIiLtahQwdmz57t8nO4mjq6ioiI1NGoUaPo16+f00LAr7/+ir+/v1PO1Zy16paSOSuTmfnp7+w6lOPqUkREpIUpnxSuNiIiIjQCiVYeShYmHeCDX1LYlq5QIiLiDgzDIK+oxCUPwzBqVePUqVNZtmwZzz33HBaLBYvFwu7du1m6dCkWi4VvvvmGgQMH4u3tzU8//cTOnTu58MILiYqKIiAggEGDBvH9999XOueJt14sFgtvvvkmF110EX5+fnTp0oXPP/+8Tp/l3r17ufDCCwkICCAoKIjLLruM9PR0x+u//fYbZ511FoGBgQQFBTFw4EDWrl0LwJ49e5g4cSKhoaH4+/vTq1cvvv766zpdvz5a9e2buBBfklIyOJCR7+pSREQEyC8upeeD37rk2pseHYef16n/LD733HNs27aN3r178+ijjwJmS8fu3bsBuO+++/jPf/5Dx44dCQ0NJSUlhXPPPZd//etfeHt788477zBx4kS2bt1Ku3btarzOI488wpNPPslTTz3FCy+8wFVXXcWePXsICws7ZY12u90RSJYtW0ZJSQnTpk1j8uTJLF26FICrrrqK/v3788orr2Cz2UhKSsLT0xOAadOmUVRUxPLly/H392fTpk0EBASc8roN1apDSWyIOcnLfoUSERGppeDgYLy8vPDz8yM6OrrK648++ijnnHOO4+ewsDD69u3r+Pmxxx5jwYIFfP7550yfPr3G60ydOpUrrrgCgMcff5znn3+eX375hfHjx5+yxiVLlvDHH3+QnJxMfHw8AO+88w69evXi119/ZdCgQezdu5d7772X7t27A9ClSxfH+/fu3csll1xCnz59AOjYseMpr+kMrTyU+AKopURExE34etrY9Og4l13bGU4//fRKP+fk5PDwww/z1VdfkZqaSklJCfn5+ezdu/ek5znttNMc2/7+/gQFBXHw4MFa1bB582bi4+MdgQSgZ8+ehISEsHnzZgYNGsRdd93FDTfcwLvvvsuYMWO49NJL6dSpEwC33347t956K9999x1jxozhkksuqVRPY2nVfUoUSkRE3IvFYsHPy8MlD2etu3PiKJp77rmHBQsW8Pjjj7NixQqSkpLo06cPRUVFJz1P+a2Uip+N3W53So0ADz/8MBs3buS8887jhx9+oGfPnixYsACAG264gV27dvHXv/6VP/74g9NPP50XXnjBadeuSasOJXFloWR/RoGLKxERkebEy8uL0tLSWh27cuVKpk6dykUXXUSfPn2Ijo529D9pLD169CAlJYWUlBTHvk2bNpGRkUHPnj0d+7p27cqdd97Jd999x8UXX8zbb7/teC0+Pp5bbrmFTz/9lLvvvps33nijUWsGhRIADucUUlBcuy+XiIhIhw4d+Pnnn9m9ezeHDx8+aQtGly5d+PTTT0lKSuK3337jyiuvdGqLR3XGjBlDnz59uOqqq1i/fj2//PILU6ZMYeTIkZx++unk5+czffp0li5dyp49e1i5ciW//vorPXr0AGDGjBl8++23JCcns379en788UfHa42pVYeSED9Pxz3E1Ey1loiISO3cc8892Gw2evbsSURExEn7hzzzzDOEhoYydOhQJk6cyLhx4xgwYECj1mexWPjss88IDQ1lxIgRjBkzho4dOzJv3jwAbDYbR44cYcqUKXTt2pXLLruMCRMm8MgjjwBQWlrKtGnT6NGjB+PHj6dr1668/PLLjVozgMWo7cBsF8rKyiI4OJjMzEyCgoKceu7RTy9l56Fc3r9hCMM6hzv13CIiUrOCggKSk5NJSEho8JL34jon++dY17/frbqlBCAu1JxBT8OCRUREXEuhpHyukmMKJSIiIq7U6kNJbLCGBYuIiLgDhZLyuUoyFUpERERcqdWHkrjQ8pYSjb4RERFxJYUSxwRq+bVeIVJEREScr9WHkqggHywWKCqxczjn5FP+ioiISONp9aHEy8NKZKA3oM6uIiIirtTqQwloYT4RERF3oFBC5X4lIiIiTaFDhw7Mnj27xtenTp3KpEmTmqwed1CnUPLKK69w2mmnERQURFBQEImJiXzzzTcnfc/8+fPp3r07Pj4+9OnTh6+//rpBBTcGhRIRERHXq1Moadu2LU888QTr1q1j7dq1nH322Vx44YVs3Lix2uNXrVrFFVdcwfXXX8+GDRuYNGkSkyZN4s8//3RK8c6i2zciIiKuV6dQMnHiRM4991y6dOlC165d+de//kVAQABr1qyp9vjnnnuO8ePHc++999KjRw8ee+wxBgwYwIsvvuiU4p3leCjRXCUiIi5lGFCU65pHLaeFeP3114mNjcVut1faf+GFF3LdddcBsHPnTi688EKioqIICAhg0KBBfP/99w36aAoLC7n99tuJjIzEx8eH4cOH8+uvvzpeP3bsGFdddRURERH4+vrSpUsX3n77bQCKioqYPn06MTEx+Pj40L59e2bNmtWgehqDR33fWFpayvz588nNzSUxMbHaY1avXs1dd91Vad+4ceNYuHDhSc9dWFhIYWGh4+esrKz6llkrcWopERFxD8V58Hisa659/wHw8j/lYZdeeim33XYbP/74I6NHjwbg6NGjLFq0yNFFIScnh3PPPZd//etfeHt788477zBx4kS2bt1Ku3bt6lXe3//+dz755BP+97//0b59e5588knGjRvHjh07CAsL44EHHmDTpk188803hIeHs2PHDvLzzb9rzz//PJ9//jkfffQR7dq1IyUlhZSUlHrV0ZjqHEr++OMPEhMTKSgoICAggAULFtCzZ89qj01LSyMqKqrSvqioKNLS0k56jVmzZvHII4/UtbR6Kw8lR3KLKCguxcfT1mTXFhGR5iU0NJQJEyYwd+5cRyj5+OOPCQ8P56yzzgKgb9++9O3b1/Gexx57jAULFvD5558zffr0Ol8zNzeXV155hTlz5jBhwgQA3njjDRYvXsx///tf7r33Xvbu3Uv//v05/fTTAbMjbbm9e/fSpUsXhg8fjsVioX379vX99RtVnUNJt27dSEpKIjMzk48//phrrrmGZcuW1RhM6mPmzJmVWliysrKIj4932vlPFOTrgb+XjdyiUvZn5NMpIqDRriUiIifh6We2WLjq2rV01VVXceONN/Lyyy/j7e3N+++/z+WXX47VavaKyMnJ4eGHH+arr74iNTWVkpIS8vPz2bt3b71K27lzJ8XFxQwbNux4uZ6eDB48mM2bNwNw6623cskll7B+/XrGjh3LpEmTGDp0KGCO5DnnnHPo1q0b48eP5/zzz2fs2LH1qqUx1XlIsJeXF507d2bgwIHMmjWLvn378txzz1V7bHR0NOnp6ZX2paenEx0dfdJreHt7O0b4lD8ak8ViUWdXERF3YLGYt1Bc8bBYal3mxIkTMQyDr776ipSUFFasWMFVV13leP2ee+5hwYIFPP7446xYsYKkpCT69OlDUVHjzRw+YcIE9uzZw5133smBAwcYPXo099xzDwADBgwgOTmZxx57jPz8fC677DL+8pe/NFot9dXgeUrsdnul/h8VJSYmsmTJkkr7Fi9eXGMfFFdSKBERkdry8fHh4osv5v333+eDDz6gW7duDBgwwPH6ypUrmTp1KhdddBF9+vQhOjqa3bt31/t6nTp1wsvLi5UrVzr2FRcX8+uvv1a6UxEREcE111zDe++9x+zZs3n99dcdrwUFBTF58mTeeOMN5s2bxyeffMLRo0frXVNjqNPtm5kzZzJhwgTatWtHdnY2c+fOZenSpXz77bcATJkyhbi4OEeP3jvuuIORI0fy9NNPc9555/Hhhx+ydu3aSh+SuyhfLXi/RuCIiEgtXHXVVZx//vls3LiRq6++utJrXbp04dNPP2XixIlYLBYeeOCBKqN16sLf359bb72Ve++9l7CwMNq1a8eTTz5JXl4e119/PQAPPvggAwcOpFevXhQWFvLll1/So0cPAJ555hliYmLo378/VquV+fPnEx0dTUhISL1ragx1CiUHDx5kypQppKamEhwczGmnnca3337LOeecA5gdacrvpwEMHTqUuXPn8s9//pP777+fLl26sHDhQnr37u3c38IJHBOoHVNLiYiInNrZZ59NWFgYW7du5corr6z02jPPPMN1113H0KFDCQ8P5x//+EeDR5I+8cQT2O12/vrXv5Kdnc3pp5/Ot99+S2hoKGB2r5g5cya7d+/G19eXM888kw8//BCAwMBAnnzySbZv347NZmPQoEF8/fXXlf5muwOLYdRyYLYLZWVlERwcTGZmZqP1L1mwYR93zvuNxI5t+OCmMxrlGiIiclxBQQHJyckkJCTg4+Pj6nKknk72z7Guf7/dKyK5UGxwWZ+STLWUiIiIuIJCSZnyPiWpGQXY7W7feCQiItLiKJSUiQrywWqBolI7h3OrH00kIiIijUehpIynzUpUkHkvTJ1dRUREmp5CSQVamE9EpOk1g/EWchLO/OenUFKBJlATEWk6np6eAOTl5bm4EmmI8n9+5f88G6LeqwS3RI65ShRKREQanc1mIyQkhIMHDwLg5+eHpQ5TvYtrGYZBXl4eBw8eJCQkBJut4YvZKpRUEBdS1qdEoUREpEmUr4VWHkyk+QkJCTnlmna1pVBSgW7fiIg0LYvFQkxMDJGRkRQXF7u6HKkjT09Pp7SQlFMoqUChRETENWw2m1P/uEnzpI6uFZRPoHYsr5i8ohIXVyMiItK6KJRUEOTjSaC32XikYcEiIiJNS6HkBLEagSMiIuISCiUniC0bgaN+JSIiIk1LoeQE6uwqIiLiGgolJyjv7KrbNyIiIk1LoeQEjlldtSifiIhIk1IoOYHj9k2mQomIiEhTUig5QXkoScssoNSulStFRESaikLJCaICvbFZLRSXGhzOKXR1OSIiIq2GQskJPGxWooO0MJ+IiEhTUyipRvlcJersKiIi0nQUSqqhuUpERESankJJNRRKREREmp5CSTUcc5VoUT4REZEmo1BSjTgtyiciItLkFEqqods3IiIiTU+hpBrlo28y84vJKSxxcTUiIiKtg0JJNQJ9PAny8QAgVa0lIiIiTUKhpAax6lciIiLSpBRKaqDOriIiIk1LoaQG6uwqIiLStFp3KDEMyNwHxVXnIzkeSjRXiYiISFNo3aHklaHwbC/Yv7bKS3Ghun0jIiLSlFp3KAmKM58Pb6vyUpwW5RMREWlSrTuURHQznw9VDSXlt2/SsgootRtNWZWIiEir1LpDSXhX8/nw1iovRQb64GG1UGo3OJitfiUiIiKNrXWHkpO0lNisFqKDzVs4GoEjIiLS+Fp3KClvKcnaB4U5VV4uv4WzT/1KREREGl3rDiV+YeAXbm4f2V7l5TgNCxYREWkyrTuUwCk6u+r2jYiISFNRKAnvYj5XMyxYs7qKiIg0nTqFklmzZjFo0CACAwOJjIxk0qRJbN1adeRKRXPmzMFisVR6+Pj4NKhopwovaympZgSO1r8RERFpOnUKJcuWLWPatGmsWbOGxYsXU1xczNixY8nNzT3p+4KCgkhNTXU89uzZ06CinSqirLNrNbdvFEpERESajkddDl60aFGln+fMmUNkZCTr1q1jxIgRNb7PYrEQHR1dvwobW3lLydGdUFoMNk/HSzFloSS7oISsgmKCfDyrO4OIiIg4QYP6lGRmZgIQFhZ20uNycnJo37498fHxXHjhhWzcuPGkxxcWFpKVlVXp0WiC4sDTD+wlcGx3pZcCvD0I9jWDSKpG4IiIiDSqeocSu93OjBkzGDZsGL17967xuG7duvHWW2/x2Wef8d5772G32xk6dCj79u2r8T2zZs0iODjY8YiPj69vmadmtR7v7Hqo5n4l6uwqIiLSuOodSqZNm8aff/7Jhx9+eNLjEhMTmTJlCv369WPkyJF8+umnRERE8Nprr9X4npkzZ5KZmel4pKSk1LfM2jlJZ1fHBGoKJSIiIo2qTn1Kyk2fPp0vv/yS5cuX07Zt2zq919PTk/79+7Njx44aj/H29sbb27s+pdWPYw2c6iZQ01wlIiIiTaFOLSWGYTB9+nQWLFjADz/8QEJCQp0vWFpayh9//EFMTEyd39toHCNwam4pUSgRERFpXHVqKZk2bRpz587ls88+IzAwkLS0NACCg4Px9TX/eE+ZMoW4uDhmzZoFwKOPPsoZZ5xB586dycjI4KmnnmLPnj3ccMMNTv5VGsBx+2Y7GAZYLI6XFEpERESaRp1CySuvvALAqFGjKu1/++23mTp1KgB79+7Faj3eAHPs2DFuvPFG0tLSCA0NZeDAgaxatYqePXs2rHJnCusIFhsUZUN2KgTFOl6KC9X6NyIiIk2hTqHEMIxTHrN06dJKPz/77LM8++yzdSqqyXl4QVgCHNlh3sKpGErKWkrSsgooKbXjYdPM/CIiIo1Bf2HLOW7hVJ7ZNSLAG0+bhVK7QXp2oQsKExERaR0USsrV0NnVarUQHawROCIiIo1NoaScY1hwzWvgKJSIiIg0HoWScjXcvoEKE6gdUygRERFpLAol5cqnms9Jh/yMSi+ppURERKTxKZSU8wmCwLJRNyfM7Kq5SkRERBqfQklF5a0lJ6yBczyUaK4SERGRxqJQUlFEWb+SE0bg6PaNiIhI41MoqaiGhfliyxblyy4sITO/uKmrEhERaRUUSipyhJLKLSV+Xh6E+nkCai0RERFpLAolFZXfvjm2G4or9x9RZ1cREZHGpVBSUUAUeAeDYYejOyu9pH4lIiIijUuhpCKL5fh08ydMouaYQE2hREREpFEolJyovF/JocqhJE7DgkVERBqVQsmJaujsqj4lIiIijUuh5EQR1a+BUz4sWKFERESkcSiUnMjRUrID7HbH7rhQs6UkPauA4lJ7de8UERGRBlAoOVFIe7B5QUk+ZO517A7398bLZsVuQFqm+pWIiIg4m0LJiWwe0KazuV1hZler1UKMbuGIiIg0GoWS6pQvzHfCGjixwWWdXTMVSkRERJxNoaQ64eWdXU9YmC9Uw4JFREQai0JJdRyrBdcwgdoxtZSIiIg4m0JJdcKrn9U1Tn1KREREGo1CSXXadAYskH8Ucg87dmsCNRERkcajUFIdLz8IiTe3K3R2rRhKDMNwRWUiIiItlkJJTcKrzuxavv5NblEpWfklrqhKRESkxVIoqUk1/Up8PG208fcCYF9GniuqEhERabEUSmoSUb5acE0L82lYsIiIiDMplNSkmts3oIX5REREGotCSU3K5yrJTIGiXMfuuBA/QKFERETE2RRKauIXBn5tzO0Ka+CUt5TsUygRERFxKoWSkznJCBy1lIiIiDiXQsnJRFQdgaMJ1ERERBqHQsnJhFcdgVMeSg5mF1JUYndFVSIiIi2SQsnJVHP7JjzACy8PK4YB6VkaFiwiIuIsCiUnU3775shOKDVncLVYLI5+JVotWERExHkUSk4mqC14+oG9GI7tduzWXCUiIiLOp1ByMlZr2YrBwOEK/UqC1dlVRETE2RRKTqV8ErUKnV3jQstCSaZCiYiIiLMolJyKo7NrxQnU1KdERETE2RRKTiW8i/lc4faNJlATERFxPoWSU3HcvtkGhgFUXinYKNsnIiIiDVOnUDJr1iwGDRpEYGAgkZGRTJo0ia1bt57yffPnz6d79+74+PjQp08fvv7663oX3OTCOoHFBkXZkJ0GQEywOfomv7iUjLxiV1YnIiLSYtQplCxbtoxp06axZs0aFi9eTHFxMWPHjiU3N7fG96xatYorrriC66+/ng0bNjBp0iQmTZrEn3/+2eDim4SHF4QlmNtlt3B8PG2EB3gDsF+3cERERJzCYjTg/sOhQ4eIjIxk2bJljBgxotpjJk+eTG5uLl9++aVj3xlnnEG/fv149dVXa3WdrKwsgoODyczMJCgoqL7l1t8HV8DWr2HCUzDkJgAufPEnftuXyWt/Hci4XtFNX5OIiIibq+vf7wb1KcnMzAQgLCysxmNWr17NmDFjKu0bN24cq1evrvE9hYWFZGVlVXq4VLgW5hMREWls9Q4ldrudGTNmMGzYMHr37l3jcWlpaURFRVXaFxUVRVpaWo3vmTVrFsHBwY5HfHx8fct0jvLOroerLsynUCIiIuIc9Q4l06ZN488//+TDDz90Zj0AzJw5k8zMTMcjJSXF6deoE8dqwcdbSuIqjMARERGRhvOoz5umT5/Ol19+yfLly2nbtu1Jj42OjiY9Pb3SvvT0dKKja+6H4e3tjbe3d31Kaxzlc5XkpEFBJvgEH59ATS0lIiIiTlGnlhLDMJg+fToLFizghx9+ICEh4ZTvSUxMZMmSJZX2LV68mMTExLpV6ko+wRAYY26XzeyaEO4PwLa0bIpK7K6qTEREpMWoUyiZNm0a7733HnPnziUwMJC0tDTS0tLIzz/eWjBlyhRmzpzp+PmOO+5g0aJFPP3002zZsoWHH36YtWvXMn36dOf9Fk3BcQvH7FfSJTKAMH8v8otLSUrJcF1dIiIiLUSdQskrr7xCZmYmo0aNIiYmxvGYN2+e45i9e/eSmprq+Hno0KHMnTuX119/nb59+/Lxxx+zcOHCk3aOdUuOEThmKLFaLQzt1AaAlTsOu6oqERGRFqNOfUpqM6XJ0qVLq+y79NJLufTSS+tyKfcTUXVhvmGdw/ny91RW7jjMned0dVFhIiIiLYPWvqmtE27fAAzvHA5AUkoGOYUlrqhKRESkxVAoqa3yUHIsGUoKAYgP8yM+zJcSu8EvyUdcWJyIiEjzp1BSW4HR4B0Ehh2O7nLsHtbJbC1ZuUOhREREpCEUSmrLYqn2Fs6wzuWhRJ1dRUREGkKhpC4cnV2Pz+xaPgJnS1o2h3MKXVGViIhIi6BQUhflM7tWaClpE+BN9+hAAFbt1C0cERGR+lIoqYvwqi0lcHwUzirdwhEREak3hZK6qDhXif341PLl/Up+UigRERGpN4WSughpDzYvKMmHzOMrFw9OCMPDamHfsXz2HslzYYEiIiLNl0JJXdg8IKyTuV1hZld/bw/6twsBYOVOtZaIiIjUh0JJXUVUXgOn3NBOuoUjIiLSEAoldVXe2fVQ5VAyvIsZSlbvPILdfuo1gkRERKQyhZK6cqwWXHkETt+2Ifh52TiaW8TmtCwXFCYiItK8KZTUVUT1ocTLw8rghDAAVmnKeRERkTpTKKmrNmUTqOUdgdzK4aN8vhJ1dhUREak7hZK68vKD4Hbmdg2dXX/edZSiEvuJ7xQREZGTUCipjxpu4XSPDqSNvxf5xaUkpWQ0fV0iIiLNmEJJfThG4FQOJVarhcSyBfq0arCIiEjdKJTURw1zlcDxKecVSkREROpGoaQ+ahgWDMc7uyalZJBbWNKUVYmIiDRrCiX1UX77JiMFiiqvdRMf5kd8mC8ldoNfko+6oDgREZHmSaGkPvzbgF8bwIAj26u8PExTzouIiNSZQkl9ld/COVT1Fo76lYiIiNSdQkl9naRfydCyEThb0rI5nFPYlFWJiIg0Wwol9RVR1q+kmhE4bQK86R4dCMCqnZpyXkREpDYUSuqrvKUk9Tcwqq4KXD4KZ5Vu4YiIiNSKQkl9xQ8BrwA4thuSl1V5ubxfiTq7ioiI1I5CSX35BEHfK8ztn1+v8vLghDA8rBb2Hctn75G8Kq+LiIhIZQolDTH4JvN569dmi0kF/t4e9G8XAmjVYBERkdpQKGmIiK7QaTRgwC9vVHl5qOYrERERqTWFkoYacrP5vOFdKMqt9NLwLmYoWb3zCHZ71c6wIiIicpxCSUN1PgdCE6AgE36fV+mlvm1D8POycTS3iC1p2S4qUEREpHlQKGkoq/V4a8nPr1UaHuzlYWVwQhig2V1FRERORaHEGfpdaQ4PPrSlyvDg8vlK1NlVRETk5BRKnMEnuMbhweWdXX9JPkpRib2pKxMREWk2FEqcpYbhwd2jA2nj70VeUSlJKRkuKU1ERKQ5UChxlhqGB1utFhLLFuhTvxIREZGaKZQ4Uw3Dg8unnFcoERERqZlCiTPVMDy4vLNrUkoGuYUlrqpORETErSmUOFMNw4Pjw/yID/OlxG7wS/JRFxYoIiLivhRKnK2G4cHDNOW8iIjISSmUOFul4cGvOXarX4mIiMjJ1TmULF++nIkTJxIbG4vFYmHhwoUnPX7p0qVYLJYqj7S0tPrW7P4cw4O/cQwPHlo2AmdLWjaHcwpdVJiIiIj7qnMoyc3NpW/fvrz00kt1et/WrVtJTU11PCIjI+t66eajmuHBbQK86RETBMCqnUdcWJyIiIh78qjrGyZMmMCECRPqfKHIyEhCQkLq/L5ma8gtsHOJOTz4rPvBy59hndqwOTWLVTsOc0HfWFdXKCIi4laarE9Jv379iImJ4ZxzzmHlypUnPbawsJCsrKxKj2an8xgI61hpePAwrYMjIiJSo0YPJTExMbz66qt88sknfPLJJ8THxzNq1CjWr19f43tmzZpFcHCw4xEfH9/YZTqf1Xq8b0nZ8ODBCWF4WC2kHM1n75G8+p/72B54bSTMn1ppVWIREZHmrNFDSbdu3bj55psZOHAgQ4cO5a233mLo0KE8++yzNb5n5syZZGZmOh4pKSmNXWbjOGF4sL+3B/3bhQANaC05shPengCpSbBxAez60WnlioiIuJJLhgQPHjyYHTt21Pi6t7c3QUFBlR7NUjXDg8tXDa7X0OCDm81AkrUfrGXdgVY844xKRUREXM4loSQpKYmYmBhXXLrpnTA8eHgXM5Ss2nkEu70Ot15Sf4M550FOOkT1hhu+N4PJ7hWw9+dGKFxERKRp1TmU5OTkkJSURFJSEgDJyckkJSWxd+9ewLz1MmXKFMfxs2fP5rPPPmPHjh38+eefzJgxgx9++IFp06Y55zdwdycMD+7bNgQ/LxtHc4vYkpZdu3PsWwv/mwh5RyC2P1zzhfnc93Lz9Z/UWiIiIs1fnUPJ2rVr6d+/P/379wfgrrvuon///jz44IMApKamOgIKQFFREXfffTd9+vRh5MiR/Pbbb3z//feMHj3aSb9CMzDkFvN5/bt4leYxJCEMqOUtnN0r4Z0LzVE88WfAlM/Az3w/w+4EixW2LYK0PxqpeBERkaZhMQz3H76RlZVFcHAwmZmZzbN/id0OLw6Eo7vgvGd4s+As/u+rzYzqFsGcawfX/L6dP8AHV0JJPiSMgMs/AO+AysfMvxY2fgq9LoZL327c30NERKQO6vr3W2vfNIWKw4N/eZ2hHc0p539JPkpRib3692z9BuZONgNJ53Pgyo+qBhKAM+8ynzcugMM1dx4WERFxdwolTaXC8ODu+Rto4+9FXlEpSSkZVY/duADmXQ2lRdD9fLj8ffD0rf680X2gyzjAgJWzG/EXEBERaVwKJU2lwvBg66+vcWbZKJwFG/ZVPu63efDxdWAvgd5/gUvngIf3yc995t1l7/0QMved/FgRERE3pVDSlCoMD76ulwWAT9bv50j5qsHr5sCCm8GwQ7+r4eLXweZ56vO2GwLth4O9GFa92Di1i4iINDKFkqZUYXhwnwPz6ds2mKISO++t2QtrXoUv7gAMGHQDXPACWG21P3d535J1cyBXa+uIiEjzo1DS1MqGB1vWv8tNiVEAWFc9C4v+Yb6eOB3O/Y/ZObYuOp1tzl1Skg9rXnZmxSIiIk1CoaSpla8eXJjJ+NJlPOC/kNvs75uvjfg7jP0/sFjqfl6L5Xjfkl/eMOc1ERERaUYUSppaheHBtsUPcH3pRwD813sKxln31y+QlOt2HoR3g8Is+PVNZ1QrIiLSZBRKXKF8eHBxHgCP26fyWOZ4fqrPIn0VWa3H+5asfhmK8hpYqIiISNNRKHEFn2BInAY2Lzh/NsWDzJaTN1ckN/zcvS+BkHaQdxg2vNvw84mIiDQRhRJXOet+mLkfTr+Wa4cmYLXAsm2H2JZey0X6amLzhGEzzO2Vz0NJUYNLFRERaQoKJa7k4QVAuzZ+jO0ZDcBbPzmhtaTfVRAQBVn74I+PGn4+ERGRJqBQ4iZuODMBgE837Odw+WRq9eXpYw4tBljxDNhLG1idiIhI41MocRMD24fSNz6kbDK1PQ0/4enXgk8IHN0Jmz5r+PlEREQamUKJm7BYLNww3GwteXf1HgqKG9i64R3omKiNFc+AYTSwQhERkcalUOJGJvSOJi7ElyO5RXyWtL/hJxxyM3j6Q/ofsH1xw88nIiLSiBRK3IiHzcrUoR0Ac3iw0dDWDb8wGHSdub3iP2otERERt6ZQ4mYmD47H38vG9oM5LN/uhIX1Eqeb86Gk/Ax7VjX8fCIiIo1EocTNBPl4MnlQOwD+64zhwYHR0P9qc3vF0w0/n4iISCNRKHFD1w7rgNUCy7cdYmtaAydTAxh6O1hssHMJ7F/f8POJiIg0AoUSNxQf5sf43k6cTC0sAfr8xdz+6ZmGn09ERKQRKJS4qeuHdwRgQdJ+DmU3cDI1gOFlC/Vt/gIObmn4+URERJxMocRNDWwfSj9nTqYW2R26n29ur5zd8POJiIg4mUKJGyufev69NU6YTA3gzLLWkt8/gmNOCDoiIiJOpFDixsb3Oj6Z2sINTphMLW4gdDwLjFJY9XzDzyciIuJECiVuzMNm5dphHQB48ycnTKYGcObd5vP6dyE7veHnExERcRKFEjd32aB4Arw92HEwh2XbDjX8hB2GQ9vBUFoIX9wOpSUNP6eIiIgTKJS4OXMytXjASZOpWSww/gnw8IFti+DruzX9vIiIuAWFkmZg6lBzMrUV2w87ZzK1tgPhkjcBC6ybY66LIyIi4mIKJc1AfJgfE3rHAPDfn3Y556Q9JsK5T5nbP/wfJM11znlbss+mw+w+cGirqysREWmRFEqaievLhgcv3HDAOZOpAQy+EYbdYW5/fhvsWOKc87ZEm7+ADe9Cxl6YdzUUOqHFSkREKlEoaSYGtAtlQLsQikrtvOuMydTKjX4Y+lwG9hL4aAocSHLeuVuKgiz4+u/mtsUKh7fBZ9PUF0dExMkUSpqR8qnnnTaZGoDVChe+BAkjoCgH5l6midVO9MNjkH0AwjrCXxeC1RM2fQarX3R1ZSIiLYpCSTMyrlcUcSG+HM0tYoEzJlMr5+EFk9+DqN6Qkw7v/wXyjjrv/M3ZvrXwyxvm9vnPQseRMH6W+fPihyB5hetqExFpYRRKmpGKk6n996dk7HYn3j7wCYar5kNQnHl74oMroDjfeedvjkqL4Ys7AANOuxw6jjL3D7rB/NkohY+vhUwnBkQRkVZMoaSZmVxxMrXtTphMraKgWLj6E/AOhpQ18OmNYHfSbaLmaPVLkP4n+IbBuH8d32+xmK0mUb0h9xDMvwZKilxXp4hIC6FQ0swE+nhyedlkaq8t2+mcqecriuwBV8wFm5c54mTRzNbZofPYblj6hLk99v/AP7zy615+MPldM8Dt+xW+vb/JSxQRaWkUSpqhqcM64GmzsGbXUd5c4YRZXk/UYThc9Kq5/ctrsOoF51/DnRkGfHkXlORDhzOh35XVHxfWES5+3dz+9Q347cOmq1FEpAVSKGmG2ob68eD5PQF4YtEWft51xPkX6X0JjC27ZbH4AfjjY+dfw139+QnsXAI2bzh/tnm7pibdxsOIsuHCX8yAtD+aokIRkRZJoaSZuvqM9lzUP45Su8G0uRtIzypw/kWGTocz/mZuL7gFkpc7/xruJu8oLLrP3B5xD4R3PvV7Rt0HnceYLSvzrob8Y41bo4hIC6VQ0kxZLBYev6gP3aMDOZxTyLT311Ncanf+hcb+C3pOAnsxfHgVpG90/jXcyfcPmZ1Xw7vBsBm1e4/VBhe/ASHtzL4oC24BeyP8sxARaeEUSpoxXy8br1w9kEBvD9buOcasr7c4/yJWK1z0GrQbCoVZ8N5f6j4E1l4K2emQ+hts+9bse5G+0f060O5ZBevfMbcnPmfO31JbfmFw2bvmLZ9ti2DF041To4hIC2Yx6jh8Y/ny5Tz11FOsW7eO1NRUFixYwKRJk076nqVLl3LXXXexceNG4uPj+ec//8nUqVNrfc2srCyCg4PJzMwkKCioLuW2Ct9tTOOmd9cB8PwV/bmgb6zzL5J3FN4aD4e3QmRPuPYb8A6CvCOQnWpOupadCtlpZc8Vfs5JN+f0OFFgDHQaDZ3Pho5nmX/YXaWkEF4dbs7RMuAauOD5+p1nw3vmFPRY4KqPocsYp5YpItKc1PXvt0ddL5Cbm0vfvn257rrruPjii095fHJyMueddx633HIL77//PkuWLOGGG24gJiaGcePG1fXyUo2xvaL526hOvLx0J/d98js9ogPpEhXo3Iv4hZlzmLw5Bg5uMlfLLc4z18ypFQsEREJAFHgFwIH1ZmhJes98YIG4gdB5tBlU4gaCrc5fz/r7abYZSPwj4ZxH6n+e/lebQ4TXzYFProebl0FoBycVKSLSstW5paTSmy2WU7aU/OMf/+Crr77izz//dOy7/PLLycjIYNGiRbW6jlpKTq3UbjDlrZ9ZueMIHSP8+WzaMAJ9PJ1/obQ/4O1zzVs5AFjAPwICo8yWj8BoCIg2n8t/Dow2/9hXDBnF+ebtkp0/mKsTH9pc+To+weYMqp1Gm0EluK3zf5dyh7fDK0OhtAgu+S/0+UvDzldSaLYqHVgPMX3hum/B09c5tYqINCN1/fvd6KFkxIgRDBgwgNmzZzv2vf3228yYMYPMzMxq31NYWEhhYaHj56ysLOLj4xVKTuFITiHnv/ATqZkFTOgdzctXDcBysuGs9ZWVClkHygJIJNicEH4y95sBZecS2PkjFGRUfj2i+/FbPe2HOe+PvGHA/ybC7hXmCJqrPj75EODaykiB10eat7f6XQ0Xvuic84qINCN1DSWN3tE1LS2NqKioSvuioqLIysoiP7/6tVVmzZpFcHCw4xEfH9/YZbYIbQK8efmqAXjaLHzzZ1rjTKwGEBQDbQdCcJxzAgmY5xrwV7h0Dvx9F1z/PYyaCW0Hg8UKh7bAmpfgvUvgP93M2VYLqg+1dZL0vhlIPHzhvKedFxxC4uEvb5m1J70H6//nnPOKiLRgbjn6ZubMmWRmZjoeKSkpri6p2ejfLrTSxGprGmNitcZmtUH8IHP+jxsWmyHl0v/BgCnmgoGFmbB0ltmvZem/6x9Ocg/Dd/80t8+a6fy+Hx1HwdkPmNtf3wv71zn3/CIiLUyjh5Lo6GjS09Mr7UtPTycoKAhf3+qb4L29vQkKCqr0kNqrOLHa9MaaWK0p+YZCr0lwwQsw40/4y9vm7ZyCTFj6uBlOlj1Z93Dy7f3mRGdRfY5PEudsw++E7ueb/VXmTYHcZhgSRUSaSKOHksTERJYsWVJp3+LFi0lMTGzsS7daTTaxmitYrdD7Yrh1tXl7JLybGUZ+/BfMPg2WPQUFWac+z84f4Pd5gMWck8RZt6FOZLHApJehTWfI2gdf3tE41xERaQHqHEpycnJISkoiKSkJMIf8JiUlsXfvXsC89TJlyhTH8bfccgu7du3i73//O1u2bOHll1/mo48+4s4773TObyDV8vWy8erVAwn0MSdWe/zrzad+U3NitZrr8/ytYjjJgB//r6zl5CThpDjfXHAPYPBNZv+YxuQTbPaVAdj8pTnrq4iIVFHnULJ27Vr69+9P//79Abjrrrvo378/Dz74IACpqamOgAKQkJDAV199xeLFi+nbty9PP/00b775puYoaQIdwv155rJ+ALy9cjefJdVxJtbmwGo7Hk4u+S+Ed60cTpZXE06WPQnHkiEwFs7+Z9PUGd3H7GOCAWvfbppriog0Mw0aEtxUNE9Jwzy5aAsvL92Jr6eNz6YPo6uzJ1ZzJ/ZS2LgAlv3bnAwNzD4pidPNVpHMFHhthDnp2+T3oMfEpqtt8xfmgn1+beCuzeDh3XTXFhFxAbcbEiyud/fYbgzr3Ib84lJueXcd2QXFri6p8Vht5uRnf1sDF78JbbqYnVl/eAyeOw3m/dUMJN3Oa9pAAtB1gtk6k3cENn3etNcWEWkGFEpaAZvVwvOX9ycm2Iddh3O5d/7vNIMGsoax2uC0S2Haz5XDydGd5jT35z7Z9DXZPGDgVHP71zeb/voiIm5OoaSVqDix2qKNabyxYperS2oalcLJG+bCfxe92rjT1p/MgClgsUHKGnOlZBERcVAoaUX6twvlwYm9AHjimy38sCX9FO9oQaw2OO0ymLKw6W/bVBQUA93PM7d//a/r6hARcUMKJa3M1UPacfGAOOwG3PTOOhZuaIEjctzdoBvM59/nQWG2a2sREXEjCiWtjMVi4YmLT+OCvrGU2A1mzEvivz810ho5Ur2EEWYfl6KcsgncREQEFEpaJS8PK7Mn92Pq0A4APPblJp74ZkvL7/zqLiwWOP06c/vXt8yVikVERKGktbJaLTw0sSf3jusGwKvLdvKPT36npKVMR+/u+l1hrkx8cCPsXePqakRE3IJCSStmsViYdlZnnri4D1YLfLR2H7e8t56C4lJXl9by+YZCn0vM7bXq8CoiAgolAlw+uB2vXD0QLw8r329O56///ZnMvBY8wZq7OP1683nTZ5BzyLW1iIi4AYUSAWBcr2jevW4wgT4e/Lr7GJe9tpr0rAJXl9WyxQ2A2AFQWgQb3nV1NSIiLqdQIg5DOrbho5sTiQj0Zmt6Nhe/vIqdh3JcXVbLNqistWTd2+a6PSIirZhCiVTSIyaIT28dSkK4P/sz8rn01dX8lpLh6rJarl4Xg08wZOyFHUtcXY2IiEsplEgV8WF+zL8lkT5xwRzNLeKKN9awYrv6PDQKLz/od7W5rfVwRKSVUyiRaoUHePPBTWcwrHMb8opKuW7Or3z+2wFXl9Uylc9Zsv07OLbHtbWIiLiQQonUKMDbg7emDuK802IoLjW448MNzFmp2V+dLrwzJIwEDFg3x9XViIi4jEKJnJS3h43nL+/PlMT2GAY8/MUm/vPtVs3+6mzl6+GsfwdKCl1bi4iIiyiUyCnZrBYeuaAXd53TFYAXf9zB3z/+nfwijRZxmm7nQmAM5B2GzV+4uhoREZdQKJFasVgs3D66C/+6qDdWC8xft4/zXljB7/syXF1ay2DzgAHXmNu/aoZXEWmdFEqkTq4a0p63rx1MZKA3uw7lcvHLq3ju++1aM8cZBl4DFhvsXQXpG11djbRWOQfhvUtg5fOurkRaIYUSqbORXSP4dsYIzusTQ4nd4Nnvt3HJq6vZpYnWGiYoFrqfa26vfcu1tUjrZLfDwlthx/ew+EHYt87VFUkro1Ai9RLq78WLV/Zn9uR+BPp48FtKBuc+v4J3Vu9WJ9iGKF8P57d5UJjt2lqk9fn5VTOQAGDAlzOgtMSVFUkro1Ai9WaxWJjUP45vZ4xgWOc2FBTbefCzjUx56xfSMrVuTr0kjIQ2naEoG37/yNXVSGuS+ht8/5C5Pep+8AmBtN/h1zdcWpa0Lgol0mCxIb68e90QHprYE28PKyu2H2bc7OV8ocnW6s5qPT6Z2tq3QK1O0hSKcuGTG8zFIbudCyP/DmMeNl/74f8gS/8uS9NQKBGnsFotXDssga9uH06fuGAy84u57YMN3P7BBjLzil1dXvPS70rw8IX0PyHlF1dXI63Bt/fD4W3msPQLXgSLxRwN1nYQFOXAovtcXaG0Egol4lSdIwP59G9DuX10F2xWC5//doBxs5fz0/bDri6t+fANhd6XmNtaD0ca26bPy2YStsBFr4J/G3O/1QrnP2uOCNv0GWz7zpVVSiuhUCJO52mzctc5Xfn4lkQSwv1Jyyrg6v/+zMOfb9SEa7U1qOwWzqaFkKtAJ40kcx98fpu5PewO6Diq8uvRfeCMW83tr++BorwmLU9aH4USaTT924Xy1e3D+esZ7QGYs2o3572wgt9SMlxbWHMQNxBi+5v3+De86+pqpCWyl8KnN0NBhvldO+v/VX/cqJkQFAcZe2DFf5q0RGl9FEqkUfl5efDYpN7877oKE669sorHvtzEsdwiV5fn3sqHB69925w/QsSZfnoG9vwEnv5wyX/Bw6v647wDYMKT5vbK5+HglqarUVodhRJpEiO7RvDdnSM4/7QYSu0G//0pmRFP/ciry3ZSUKxbOtXqfQn4BJv/h7pziaurkZYk5Vf4cZa5fd5/oE2nkx/f/TzoOgHsxfDVXRoVJo1GoUSaTIifFy9eOYB3rhtMj5ggsgtKeOKbLZz1n6XMX5tCqV3/oavEyw/6XWVuq8OrOEtBFnxyPRilZvDte8Wp32OxwLlPgqcf7FkJv33Q+HVKq6RQIk1uRNcIvrptOM9c1pe4EF9SMwu49+PfOe/5Ffy49aBmhK2ofM6Sbd9Cxl7X1iItw1d3m61vIe3KRtdYave+kHYw8h/m9nf/hLyjjVejtFoWoxn8BcjKyiI4OJjMzEyCgoJcXY44UUFxKe+s3s2LP+wgq8CcznpopzbMnNCDPm2DXVydm/jfREheDmfeDaMfdHU1tVdSaM6zsnuF+X/nviHmLKEVn31Dj297eLuw2Fbit3mw4CZzmO+130C7IXV7f2kxvHomHNoM/f8KF77YOHVKi1HXv98KJeIWMvKKeHnpTuas3E1R2YrDF/SN5d5x3YgP83NxdS626TP4aAr4R8DtG8A70NUVVc9eak5LvmsZ7FoKe9dASX7t3+/hW31w8Y8w/wBGdG2MqluPo7vg1RHmEgZn/T9z1tb62LsG3hpnbl+7CNonOq9GaXEUSqRZ23csj2e+28aCpP0YBnjaLPz1jA7cdnZnQv1rGB3Q0pUWw7O9IScNPHyg09nQ/XzoNgH8wlxXl2HAkZ2QvNQMIsnLzeGlFflHQseR5pDSggzIz6jmORM4xX+GrB4w5BbzD6lPM2lBO7LTvMURP8jVlZjfobfGwf510G4oTP0SrLb6n+/z22D9OxDRA25ZATZP59UqLYpCibQIGw9k8sQ3W1hRNhNsoLcHt57VieuGJeDj2YD/mDZXO76HL+8y+wKUs9ig/VAzoHQ/D0LiG7+O7LSyALLMfM7aV/l1r0DoMNwMIh1HQUT3U/dZsNuhMKvm0LJnFWz/1jzWP8Jck6XvleaMo+4o9zD8+DisexsMO4x7HBKnubamJY/CiqfNQHfLyoZ/V/KOwounQ94RGPMIDJ/hlDKl5VEokRZlxfZDzPp6C5tSswCIDvLhrnO6cvGAODxsbvpHqbEYhrkezuYvYcuX5nZFMf2gx/lmSKlNGDiZ0hLI2g/HkuHYbkjfZAaRQyfMUWHzgvgh5urGHUdC7ACwedT/ujXZ/r25/sqR7ebPsQPg3Keg7enOv1Z9lRTBL6/DsiehMLPya64MJskrzH5JGHDp/6DXJOecN2kuLLzVvO027WcIbe+c80qLolAiLY7dbvDZb/v5z7fb2J9h9lHoGOHPXed05dzeMVitDfjj25wdTYYtX5kBZe8aKt0CCetUFlAmmrPDVteqUJBpBo7yx9Hk49uZKWAvqeaiFog5zWwFSRgJ7RLNoctNoaQIfnkNlv7b7BcB5pDp0Q9BYFTT1FAdw4Ct38B3/8/stwHm9OzjnzBbk5aXTTzmimCSdxReGQbZB5zfMdUwYM755gRsXcfDFR82LAhLi6RQIi1WQXEp767ew8tLd3CsbOXhnjFB3DOuK2d1i8TSmv+DmHMQtn5thpRdS83p6csFREP3c82OoxVDSP4phnTavM3/+w3tAGEdzQCSMMK1/VgAstNhySOQ9L75s1cgjPoHDL655llJG0v6Rlg002xFArMPzegHzZWerTbzD/ePj7smmBgGzLvaDK1tOsPNy8HL37nXOLTVDD32Ypj8vhmERSpQKJEWL7ugmLd+2s0bK3aRU2j+3/zA9qHcM7YbiZ3auLg6N1CQBTsWm7d5ti8+3qpQHb9wCEswg4fjUfZzYIz79tsA2LcWvr4XDqw3f27TxWyd6DKm8a+dexh+/Je5uq5hN29jJU4zh22fODrKVcFk7dvw5QywesIN30Nsv8a5Tnl/laA4mPaLOS29SBmFEmk1juUW8erynfxv1W4Kis1hxMM7h3PPuG70iw9xbXHuoqTQHBWz7VvAOCF4tHff4cW1ZbfDb3Ph+4ch95C5r9u5MO5fZuuOs5XfQlr2pNk5F6DnhXDOo+bnWpOmDiYHt8Dro8wh2WP/D4be1njXKsqDl88wO2EnTjc/e5EyCiXS6hzMKuDFH3fwwS97KS41v87n9Izi7rFd6R6t70urUJBpBoWfXzX7wti8zD/Ew+9yzv+5G4Z5e+y7f1boN3Ka2TLTYVjtz9HYwcRuN29rff+QOTKm09lw1SeN3+K17TuYe6k5IuzmZWafGhGaKJS89NJLPPXUU6SlpdG3b19eeOEFBg8eXO2xc+bM4dprr620z9vbm4KCglpfT6FEaiPlaB7PLdnOp+v3YTfMPncX9I3lzjFd6RDu5Hvp4p4ObYVv/gG7fjR/Dowx5zcJjDaHw3oHmc/lD6+AU//BTvsTvr3/eL+RgCiz30jfK+o+10djBpO0P8wp5FN+Nn+O7AV/XdB0nYA/mmJO9Nd2EFz3nXvf+pMm0+ihZN68eUyZMoVXX32VIUOGMHv2bObPn8/WrVuJjIyscvycOXO444472Lp16/GLWixERdX+XxSFEqmLHQdzeHbxNr76IxUAm9XCpQPbcvvoLsSG+Lq4Oml05a0ai2ZWntelOhZr1aBS8ZF3FP74qKzfiHdZv5G7Gnbby9nBpCDLPN8vr5l1evrDqPvgjFubdlKzrAPw4iAoyoHzZ8Pp157yLdLyNXooGTJkCIMGDeLFF82hZXa7nfj4eG677Tbuu+++KsfPmTOHGTNmkJGRUZfLVKJQIvXx5/5Mnv5uKz9uNfsaeNmsXHVGO64fnkDb0FY+dX1rUFwAv75hdogtyCyboC3TfORnmCNGaqvnJDjnkZP3G6kLZwQTw4A/PzFbcXLSj9c57nEIjnNOnXW15hVzPhmfYLj4Deg0unHmrZFmo1FDSVFREX5+fnz88cdMmjTJsf+aa64hIyODzz77rMp75syZww033EBcXBx2u50BAwbw+OOP06tXrxqvU1hYSGFhYaVfKj4+XqFE6mXt7qM89e1Wfk4+PgR2YPtQLugby7l9YogI1EJwrY5hQElBWUipEFYKMioHmOIC6HmBOXNuY9RQ32ByaCt8fY/ZiRnMeWnOfQo6j3Z+nXVRWgJvnGWugQTmcPS+l0P/qyG8i2trE5do1FBy4MAB4uLiWLVqFYmJxxdh+vvf/86yZcv4+eefq7xn9erVbN++ndNOO43MzEz+85//sHz5cjZu3Ejbtm2rvc7DDz/MI488UmW/QonUl2EY/LTjMC//uJM1yUco/9ZbLTCsczgT+8Yyrlc0wb5aw0OaUF2DSVEuLH8KVr1otvR4+MCZ98Cw291nleWcg/DTs/D7PLOzbbn4IWY46XVR8x/1JbXmdqHkRMXFxfTo0YMrrriCxx57rNpj1FIijSkts4Avfz/AF7+n8ltKhmO/l83KqG4RXNAvltHdo/D1aoVr7EjTq00wMQxzYrxF95mz7YI5i+qEfzvvlpKzlRTBtkWw4T1z3hzDHLaPp595m6n/1WYLVEMnPbTbzb5Dab+bnX2zUs3zunrBSgHqHkrqdLMvPDwcm81Genp6pf3p6elER0fX6hyenp7079+fHTt21HiMt7c33t5ukvqlxYkO9uGGMztyw5kd2XMkly9+O8Dnvx1gW3oO321K57tN6fh52TinZxQX9I3lzC4ReHloJIE0EosFzrrf3F7+pNlHBI4Hk6PJ8M3fYft35s/B7cww0v3cpq+1Ljy8zFtfPS8wg8LvH5oB5cgOc26Z3+aa8+X0v8ocyRRcfct5JSWF5vpLqWUBJO0Pcw2o8jljyiW9Z64snTACelxgrgcVENE4v2fF2jL3mSGxISsw15W9FDYthA3vmzM5e/qaLWiOZz/w9DHXKKr47OlX9nqFfeHdXD75Xb06ug4ePJgXXngBMDu6tmvXjunTp1fb0fVEpaWl9OrVi3PPPZdnnnmmVtdUR1dpClvSsvg8yQwo+47lO/aH+HkyoXc0E/vGMiShDbbWutaONK4TW0zGPGL+oVvxNJQWmjOzDrvdvF3TVOsNOZthQMovsOFd2LjAHKkDgMWcU6X/VdDtPPMPZP4xczh2WoUAcmhL9Wsy2bwgsqc5P4p/uDlvysGNx1+3WKH9MDOg9JgIQTEN/13yjpq/S8oac+2p/evNf05hncxA2e9KMxg0ltIS2PipeTvv8DbnnPO6b6HdGc45V5kmGRJ8zTXX8NprrzF48GBmz57NRx99xJYtW4iKimLKlCnExcUxa9YsAB599FHOOOMMOnfuTEZGBk899RQLFy5k3bp19OzZs1F+KZGGMAyDDSkZfJ50gC9/T+VwzvFbiRGB3ozuHsnoHlEM7xyuWzziXCcGk3IdR8G5/2lZnUWLcs15TTa8by7qV84nxBymnbm3+vf5hJiLQkafZoaQ6D4Q3rXq8OcjO83zb/oMUpMqvxY/xAwoPS+AkHanrtUwzEnzUn42A8jeNXB4a9XjLNbjt6n82sDgm2DQDWZQcpbSEnOY+vL/wNGd5j6fYHM+nvCuUJxvduKu+Fycb87uW1xQw3PZ9pXzILq382qliSZPe/HFFx2Tp/Xr14/nn3+eIUOGADBq1Cg6dOjAnDlzALjzzjv59NNPSUtLIzQ0lIEDB/J///d/9O/fv9F+KRFnKbUbrNl1hM+TDvDNn6lkFRz/vzRvDyvDOoczukcko7tHER3s48JKpcUwDFg6C5b925z8bdy/oNfFLXsF3qO7IGmu+cjaf3x/SLuy8FEhgAS3rftncWwPbP7CDCj7fqn8Wmz/soByIbTpZO4rKTJbaPaugb2rzRaR3INVz9umC7QbAvFnmAtWBkabM+qufhEyykKVh4+5mnXitOPnr4+SIvMW2IqnzQU1AXxDzan9B99oBhM3pGnmRRpJYUkpP+86ypLN6Xy/+SD7M/Irvd4rNojRPaIY0yOS3rHBWHWbRxri0DZzvhFnr+zrzuylZhAw7GYA8Q1x/jWyDpQFlM9h76rjLRsAUb3NP+7715mtDBXZvMwAEz/EvMURP6TmFpDSEtj8Oax6Hg5sKNtpMVdRHno7xFc/A3q1SgrNoLPi2eOtR37h5jIKg653+5FMCiUiTcAwDLamZ7Nk80G+35xOUkoGFf9Nigz0drSgDNNtHhH3lHMQtnxpBpTk5WCUHn/NN7SsBaSsJSS2v9nXpS4MA/ashJXPw/Zvj++PP8MMFd3OrXk6/uICWP8OrJx9vPUoIMoMNadf22zCqkKJiAsczinkhy0HWbI5nRXbD5NXdPw/bt4eVoZ3DufsHpGc1S1SU92LuKO8o+YIp9IisxWkTRfnrt9zcAusfgF+/8i8BpidYodON0cflXeKLcqDdXNg5XOQk2buC4yB4XfCgCmN23m2ESiUiLhYQXEpPyebt3mWVHObp3NkACO6RHBm13DOSGijVhSR1iQ7DX55HX5905w1GMzbMYNvMltiVr0AuebSGAS1hTPvhH5X172Vxk0olIi4EcMw2JKWbQaULQf5LSUDe4V/47xsVgYlhJohpUsEPWICsbTkDo0iYirMMedtWfPS8U6x5ULawZl3Q98rzflemjGFEhE3lpFXxKqdR1i+7RDLtx3iQGblznQRgd6c2TmcEV0jGN4lnPAATSIo0qKVlsDmz+Dn18zOtYNvhtMua9oVnhuRQolIM2EYBjsP5bJ82yFWbD/Eml1HyS8urXRMr9ggzuwSwYiu4QxsH4q3h271iEjzoVAi0kwVlpSybvcxlm0/xIpth9mUWnnqbF9PG6d3COWMjm1I7NSG0+KC8bBp+nsRcV8KJSItxMHsAlbuOMzybYdZsf0Qh3OKKr3u72VjUEIYiWUhpVdssKbAFxG3olAi0gLZ7QbbDmazeucRVu88ws/JR8nML650TKCPB0MSwjijYxvO6NiGnjFBmsBNRFxKoUSkFSi1G2xOzWLNLjOk/JJ8lOzCyguVBft6MiQhjMROZktK18hAhRQRaVIKJSKtUEmpnU2pWWZLyq4j/Jp8lNyiyp1mo4K8Gd0jinN6RjG0Uxt1mhWRRqdQIiIUl9r5Y38mq3ceYc2uI6zdfazSyB5/Lxsju0VwTs8ozu4WRbBfyxh+KCLuRaFERKooLCll9c4jLN6Uzveb00nPKnS8ZrNaGNwhjHN6mq0o8WF+LqxURFoShRIROSm73eCP/Zks3pTO4k3pbE3PrvR69+hAxvaM4pye0fSOC9IMsyJSbwolIlIne4/k8d2mNBZvSufX3UcrTYMfE+zDmB5RnN0jkn5tQwj1b95TXotI01IoEZF6O5ZbxA9bDvL95nSWbTtUabVjgOggH7rHBNIjJsh8RAeSEO6vSdxEpFoKJSLiFAXFZj+U7zals3LHYfYezav2OG8PK12jAukRE0j36LKwEhNIiJ9aVURaO4USEWkU2QXFbE3LZnNqFpvLnremZVdpTSkXE+zjCCh94oIZ0C6UyKDmufy6iNSPQomINBm73WDv0TwzqFQIK/uO5Vd7fHyYLwPbhTKwfSgD2ofSLSpQt35EWjCFEhFxuawKrSqbDmSRlJLB1vRsTvyvjb+XjX7tQhjYzgwp/duFEuyrOVNEWgqFEhFxS9kFxSSlZLBuzzHW7TlG0t6MKlPjWyzQJTKAge1DGdg+jIHtQ+nQxk/DkkWaKYUSEWkWSu0G2w9mO0LK+j3H2H2kamfaUD9P4sP8iAryISbYh6ggH6LLt4PNbX9vDxf8BiJyKgolItJsHc4pZP2eY6zba4aU3/ZlUlRiP+X7An08iA7yIbospEQHH9+OCfYlPsyXQB/dFhJpagolItJiFJXY2ZaeTWpmAWmZ+aRlFZCWWUhaVj5pmQWkZRZUWXiwJsG+nsSH+RIf6kd8mB9tQ8u3fYkL8cPXSwsUijhbXf9+q81TRNyWl4eV3nHB9I4LrvGY7IJi0rMKyoJLgWO7/PlARj7H8orJzC8mc38xf+7PqvY84QHexIf50jbUj/hQX+LD/IgP9aNHTCBtArwb61cUkQoUSkSkWQv08STQx5POkYE1HpNTWMK+Y3mkHM13PKccy2PfsXz2Hc0ju7CEwzmFHM4pZMPejCrvjw/zpW/bEPrFh9A3PoTescFqWRFpBLp9IyKtmmEYZOYXs+9YPilHzaCSciyPlKN57D6SR/Lh3CrvsVktdI0KpF98sCOodIkMxGbVKCGRitSnRETEibIKivljXyZJKRn8lpJBUkoGB7MLqxzn52Wjd1xZSGkbQt/4YOJCfDWcWVo1hRIRkUaWlllghpR9ZlD5fV8mOSfMuQIQ5ONBXKgfcSE+xIb4Oh7lP0cG+qh1RVo0hRIRkSZmtxvsOpxDUkomSSnH+C0lk82pWZTYT/6fV5vVQnSQD3EhvsRWCC5xIb7EhPjg7+WBp82Kh82Cp82KV9m2h9WiFhhpFhRKRETcQEFxKXuO5HEgM58DGeWPAvaXbadlFpwytJyMZ1lQ8bBa8PKw4mG14ulhwdNqxdNmxcfTSpsAb8IDvAgP8HZsR1TYDvXzwqqWGmlEGhIsIuIGfDxtdIsOpFt09aOCSu0Gh7ILHSHF8SgbxpyaWUB+USnFpfZqw0txqUFxae3maKmJ1QJh/mVhJdCbNv5mgAkP9KZdmB9dIgNo38YfLw8tmihNQy0lIiJuzjCMshBip6TUoKjUTondTnGJQbHdXnl/2XG5hSUcyS3iSE4hh3OKOJRT6Ng+nFNIRl5xra7tYbWQEO5Pl6gAOkcG0iUygC5RASSE++PtoWHRcnJqKRERaWEsFgteHhantlgUl9o5mlvEoWxzfpYjZWHlcE4hh7ILST6Sx470bHKLStl+MIftB3OANMf7bVYL7duYrSldIgPLQksAnSIC8PFUWJH6UUuJiIhUyzAMDmQWsD09mx0Hc9iensP2g9lsT8+pssJzOasF4kJ9iQ7yITLQh4hAbyKDvIkK9CEyyJvIQB8iA70J8fNUZ91WQC0lIiLiFBaLhbiy0UCjukU69huGwcHsQran57AtPZvtB3PYcTCbbek5ZOYXmzPmHs0/6bm9bFZHYIkMPB5WIoO8CQ/wJtTfizA/L0L9vQjy8VCAaSUUSkREpE4sFgtRQT5EBfkwvEu4Y79hGBzOKSL5cC4Hsws4mFXIwexCDmYXcCi7sOznAo7lFVNUamd/Rj77M04eXsDs1xLi50WYvyehfl6E+XtVCi2V9vuZnXZ1C6l5UigRERGnsFgsRAR6ExF48gUMC0tKzZBSFlQOZRc4ttOzCziaW8TR3CKO5RaRW1RKid1w9HeprSAfD0ct5beRzG3vSvtDfD01LNqNKJSIiEiT8vaw0TbUj7ahfqc8tqC4lIy8YjOk5BVVfs4t4mheMcdyizhS/nNuEUWldrIKSsgqKGHnoaprF1XkYbUQHmDeNoqo8GwGF59KQUatL41PoURERNyWj6eN6GAb0cE+tTreMAyy8ks4lGPePjpUNpqovGXm+LZ5G6nEbpCWVUBaVsEpz13e+hJ5Qlgxg4wPbQK8sFigpNTAbhiU2A3s9srPpYZBaWnZ8wmvgTkpnof1+My9HjYrnmXPjn1Wq3lc2eR55bP+enlY8fGw4WlrvjP+KpSIiEiLYbFYCPbzJNjPk86R1U9cV66oxM6R3PJbSBVCS4VAU/5cVFL71hdXs1rM1igfT2u1z96eVnw8bXh7VH6+JrED7dqcuvWqMSmUiIhIq+TlYSUm2JeYYN+THmcYBlkFJY6+LzW1vBzNLQIs2KzgYbVitYLNYsFmLX9YsVkxny3HjzGfzZaNkrIZfMufi0sNSu1lk+KVP1ezr+Ksv3YD8otLyS8uBWo3SR7AeafFNM9Q8tJLL/HUU0+RlpZG3759eeGFFxg8eHCNx8+fP58HHniA3bt306VLF/79739z7rnn1rtoERGRpmKxWAj29STY99StL65iGOaMvgXFdgqLSykssVNQzXNBsZ3CkuqfY08RzppCnUPJvHnzuOuuu3j11VcZMmQIs2fPZty4cWzdupXIyMgqx69atYorrriCWbNmcf755zN37lwmTZrE+vXr6d27t1N+CRERkdbMYrGYt2Y8bODr6epy6q3OM7oOGTKEQYMG8eKLLwJgt9uJj4/ntttu47777qty/OTJk8nNzeXLL7907DvjjDPo168fr776aq2uqRldRUREmp+6/v2u00IKRUVFrFu3jjFjxhw/gdXKmDFjWL16dbXvWb16daXjAcaNG1fj8QCFhYVkZWVVeoiIiEjLVqdQcvjwYUpLS4mKiqq0PyoqirS0tGrfk5aWVqfjAWbNmkVwcLDjER8fX5cyRUREpBly3pKTTjRz5kwyMzMdj5SUFFeXJCIiIo2sTh1dw8PDsdlspKenV9qfnp5OdHR0te+Jjo6u0/EA3t7eeHuffJpiERERaVnq1FLi5eXFwIEDWbJkiWOf3W5nyZIlJCYmVvuexMTESscDLF68uMbjRUREpHWq85Dgu+66i2uuuYbTTz+dwYMHM3v2bHJzc7n22msBmDJlCnFxccyaNQuAO+64g5EjR/L0009z3nnn8eGHH7J27Vpef/115/4mIiIi0qzVOZRMnjyZQ4cO8eCDD5KWlka/fv1YtGiRozPr3r17sVqPN8AMHTqUuXPn8s9//pP777+fLl26sHDhQs1RIiIiIpXUeZ4SV9A8JSIiIs1Po85TIiIiItJYFEpERETELSiUiIiIiFtQKBERERG3UOfRN65Q3hdXa+CIiIg0H+V/t2s7pqZZhJLs7GwArYEjIiLSDGVnZxMcHHzK45rFkGC73c6BAwcIDAzEYrE47bxZWVnEx8eTkpKiocZ1oM+tfvS51Z0+s/rR51Y/+tzq52Sfm2EYZGdnExsbW2kOs5o0i5YSq9VK27ZtG+38QUFB+gLWgz63+tHnVnf6zOpHn1v96HOrn5o+t9q0kJRTR1cRERFxCwolIiIi4hZadSjx9vbmoYcewtvb29WlNCv63OpHn1vd6TOrH31u9aPPrX6c+bk1i46uIiIi0vK16pYSERERcR8KJSIiIuIWFEpERETELSiUiIiIiFtQKBERERG30KpDyUsvvUSHDh3w8fFhyJAh/PLLL64uya09/PDDWCyWSo/u3bu7uiy3s3z5ciZOnEhsbCwWi4WFCxdWet0wDB588EFiYmLw9fVlzJgxbN++3TXFuolTfWZTp06t8t0bP368a4p1E7NmzWLQoEEEBgYSGRnJpEmT2Lp1a6VjCgoKmDZtGm3atCEgIIBLLrmE9PR0F1XsHmrzuY0aNarK9+2WW25xUcXu4ZVXXuG0005zzNqamJjIN99843jdWd+1VhtK5s2bx1133cVDDz3E+vXr6du3L+PGjePgwYOuLs2t9erVi9TUVMfjp59+cnVJbic3N5e+ffvy0ksvVfv6k08+yfPPP8+rr77Kzz//jL+/P+PGjaOgoKCJK3Ufp/rMAMaPH1/pu/fBBx80YYXuZ9myZUybNo01a9awePFiiouLGTt2LLm5uY5j7rzzTr744gvmz5/PsmXLOHDgABdffLELq3a92nxuADfeeGOl79uTTz7poordQ9u2bXniiSdYt24da9eu5eyzz+bCCy9k48aNgBO/a0YrNXjwYGPatGmOn0tLS43Y2Fhj1qxZLqzKvT300ENG3759XV1GswIYCxYscPxst9uN6Oho46mnnnLsy8jIMLy9vY0PPvjABRW6nxM/M8MwjGuuuca48MILXVJPc3Hw4EEDMJYtW2YYhvm98vT0NObPn+84ZvPmzQZgrF692lVlup0TPzfDMIyRI0cad9xxh+uKaiZCQ0ONN99806nftVbZUlJUVMS6desYM2aMY5/VamXMmDGsXr3ahZW5v+3btxMbG0vHjh256qqr2Lt3r6tLalaSk5NJS0ur9N0LDg5myJAh+u6dwtKlS4mMjKRbt27ceuutHDlyxNUluZXMzEwAwsLCAFi3bh3FxcWVvmvdu3enXbt2+q5VcOLnVu79998nPDyc3r17M3PmTPLy8lxRnlsqLS3lww8/JDc3l8TERKd+15rFKsHOdvjwYUpLS4mKiqq0Pyoqii1btrioKvc3ZMgQ5syZQ7du3UhNTeWRRx7hzDPP5M8//yQwMNDV5TULaWlpANV+98pfk6rGjx/PxRdfTEJCAjt37uT+++9nwoQJrF69GpvN5uryXM5utzNjxgyGDRtG7969AfO75uXlRUhISKVj9V07rrrPDeDKK6+kffv2xMbG8vvvv/OPf/yDrVu38umnn7qwWtf7448/SExMpKCggICAABYsWEDPnj1JSkpy2netVYYSqZ8JEyY4tk877TSGDBlC+/bt+eijj7j++utdWJm0dJdffrlju0+fPpx22ml06tSJpUuXMnr0aBdW5h6mTZvGn3/+qT5edVTT53bTTTc5tvv06UNMTAyjR49m586ddOrUqanLdBvdunUjKSmJzMxMPv74Y6655hqWLVvm1Gu0yts34eHh2Gy2Kj2D09PTiY6OdlFVzU9ISAhdu3Zlx44dri6l2Sj/fum71zAdO3YkPDxc3z1g+vTpfPnll/z444+0bdvWsT86OpqioiIyMjIqHa/vmqmmz606Q4YMAWj13zcvLy86d+7MwIEDmTVrFn379uW5555z6netVYYSLy8vBg4cyJIlSxz77HY7S5YsITEx0YWVNS85OTns3LmTmJgYV5fSbCQkJBAdHV3pu5eVlcXPP/+s714d7Nu3jyNHjrTq755hGEyfPp0FCxbwww8/kJCQUOn1gQMH4unpWem7tnXrVvbu3duqv2un+tyqk5SUBNCqv2/VsdvtFBYWOve75ty+uM3Hhx9+aHh7extz5swxNm3aZNx0001GSEiIkZaW5urS3Nbdd99tLF261EhOTjZWrlxpjBkzxggPDzcOHjzo6tLcSnZ2trFhwwZjw4YNBmA888wzxoYNG4w9e/YYhmEYTzzxhBESEmJ89tlnxu+//25ceOGFRkJCgpGfn+/iyl3nZJ9Zdna2cc899xirV682kpOTje+//94YMGCA0aVLF6OgoMDVpbvMrbfeagQHBxtLly41UlNTHY+8vDzHMbfccovRrl0744cffjDWrl1rJCYmGomJiS6s2vVO9bnt2LHDePTRR421a9caycnJxmeffWZ07NjRGDFihIsrd6377rvPWLZsmZGcnGz8/vvvxn333WdYLBbju+++MwzDed+1VhtKDMMwXnjhBaNdu3aGl5eXMXjwYGPNmjWuLsmtTZ482YiJiTG8vLyMuLg4Y/LkycaOHTtcXZbb+fHHHw2gyuOaa64xDMMcFvzAAw8YUVFRhre3tzF69Ghj69atri3axU72meXl5Rljx441IiIiDE9PT6N9+/bGjTfe2Or/B6K6zwsw3n77bccx+fn5xt/+9jcjNDTU8PPzMy666CIjNTXVdUW7gVN9bnv37jVGjBhhhIWFGd7e3kbnzp2Ne++918jMzHRt4S523XXXGe3btze8vLyMiIgIY/To0Y5AYhjO+65ZDMMw6tlyIyIiIuI0rbJPiYiIiLgfhRIRERFxCwolIiIi4hYUSkRERMQtKJSIiIiIW1AoEREREbegUCIiIiJuQaFERERE3IJCiYiIiLgFhRIRERFxCwolIiIi4hb+Px5Q9AF2VEzTAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('learning curve')\n",
    "train_loss, = plt.plot(train_losses, label='train loss')\n",
    "val_loss, = plt.plot(val_losses, label='val loss')\n",
    "plt.legend(handles=[train_loss, val_loss])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d51e3e",
   "metadata": {},
   "source": [
    "9. **模型测试**\n",
    "\n",
    "最后，我们还需要测试下训练好的模型在整个验证集上的精度。我们可以直接调用内置的measure_accuracy函数来实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d166d5be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 88.80%\n"
     ]
    }
   ],
   "source": [
    "model.eval()\n",
    "\n",
    "with torch.no_grad():\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    \n",
    "    for images, labels in val_dataloader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        \n",
    "        outputs = model(images)\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "    \n",
    "    accuracy = correct / total\n",
    "    print(f\"Accuracy: {100 * accuracy:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d24ed92",
   "metadata": {},
   "source": [
    "10. 保存模型\n",
    "\n",
    "​\t\t在前面的步骤中，我们讨论了如何加载和预处理数据，构建和训练模型，以及测试模型。当我们对所学的模型⾜够满意时，希望保存训练模型以备将来部署在各种设备中进行推理。此外，当运⾏⼀个耗时较⻓的训练过程时，最佳的做法是定期保存中间结果（检查点），以确保在服务器电源被不小⼼断掉时不会损失⼏天的计算结果。因此，我们需要知道如何加载和存储权重向量和整个模型。\n",
    "\n",
    "​\t\tPyTorch提供了内置函数来保存和加载整个⽹络。需要注意的是，下面的指令保存的是模型的参数而不是保存模型本身。当需要再次加载训练好的模型时，我们先要构建以前训练时的网络架构，然后加载模型参数到构建的网络中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fb95e3a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), './weights/resnet18.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "204e8a72",
   "metadata": {},
   "source": [
    "### 9.4   小结\n",
    "\n",
    "1. save和load函数可⽤于张量对象的⽂件读写；我们可以通过参数字典保存和加载⽹络的全部参数；保存结构必须在代码中完成，而不是在参数中完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "899558c7",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "### 拓展阅读：SGD改进算法\n",
    "\n",
    "1、动量法(Momentum)优化算法\n",
    "\n",
    "动量法(Momentum)是SGD的一种改进，它通过累积之前的梯度，来加速SGD的收敛。\n",
    "\n",
    "动量法的更新公式如下：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "v_{t+1} &= \\beta v_t - \\alpha \\frac{\\partial L}{\\partial \\theta} \\\\\n",
    "\\theta_{t+1} &= \\theta_t + v_{t+1}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中，$v$ 是累积梯度，$\\beta$ 是动量因子，$\\theta$ 是模型的参数，$\\alpha$ 是学习率，$L$ 是损失函数。\n",
    "\n",
    "动量法的优点是可以加速SGD的收敛，减少震荡，提高精度。\n",
    "\n",
    "2、Adam优化算法\n",
    "\n",
    "Adam优化算法是一种结合了动量法和RMSProp的优化算法。\n",
    "\n",
    "Adam的更新公式如下：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "m_{t+1} &= \\beta_1 m_t + (1 - \\beta_1) \\frac{\\partial L}{\\partial \\theta} \\\\\n",
    "v_{t+1} &= \\beta_2 v_t + (1 - \\beta_2) (\\frac{\\partial L}{\\partial \\theta})^2 \\\\\n",
    "\\hat{m}_{t+1} &= \\frac{m_{t+1}}{1 - \\beta_1^t} \\\\\n",
    "\\hat{v}_{t+1} &= \\frac{v_{t+1}}{1 - \\beta_2^t} \\\\\n",
    "\\theta_{t+1} &= \\theta_t - \\alpha \\frac{\\hat{m}_{t+1}}{\\sqrt{\\hat{v}_{t+1}} + \\epsilon}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中，$m$ 是动量，$v$ 是累积梯度，$\\beta_1$ 和 $\\beta_2$ 是衰减率，$\\hat{m}$ 和 $\\hat{v}$ 是平滑过的动量和累积梯度，$\\alpha$ 是学习率，$\\epsilon$ 是为了避免分母为0而添加的常数，$L$ 是损失函数。\n",
    "\n",
    "Adam的优点是可以自动调整学习率，使得模型在训练过程中不断提升精度。  \n",
    "\n",
    "综上，逻辑回归的优化方法可以分为以下几种：\n",
    "\n",
    "- BCE Loss + SGD\n",
    "\n",
    "- BCE Loss + Momentum\n",
    "\n",
    "- BCE Loss + Adam\n",
    "\n",
    "实际应用中，一般使用BCE Loss + Adam。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "d2l",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}