nn_dt_training.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shallow_nn as nn\n",
    "import optimized_tree as tr\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "from sklearn import preprocessing\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data importation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parser for the unirep files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_unirep(file_path):\n",
    "    \"\"\"\n",
    "    Parse a Unirep file.\n",
    "    \"\"\"\n",
    "    data_matrix = []\n",
    "    with open(file_path) as fp:\n",
    "        data = fp.readlines()\n",
    "    for i, line in enumerate(data):\n",
    "        if line[0] == '>':\n",
    "            data_matrix.append(np.array(\n",
    "                data[i + 1].strip().split(sep=' '),\n",
    "                dtype=np.float64))\n",
    "    return np.array(data_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will generate a dataset composed of 6000 samples: 3000 cytoplasmic proteins, and 3000 periplasmic proteins. The dataset for each class is first shuffled, then the 3000 first elements of each set are extracted onto the final dataset. This process is equivalent as random selection of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5998"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Imports unirep files.\n",
    "raw_cyt = read_unirep(\"cytoxplasmUniRef50(1).unirep\")\n",
    "raw_peri = read_unirep(\"periplasmUniRef50(1).unirep\")\n",
    "# Shuffle imported data.\n",
    "np.random.shuffle(raw_cyt)\n",
    "np.random.shuffle(raw_peri)\n",
    "# Creates mixed dataset (from shuffled = random selection).\n",
    "all_raw = np.concatenate((raw_cyt[0:3000], raw_peri[0:3000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we scale the data, giving for each feature a mean = 0, and a standard deviation = 1. This makes the learning process easier for certain type of algorithms, for example for neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0000000000000016\n"
     ]
    }
   ],
   "source": [
    "scaled_transfrom = preprocessing.scale(all_raw, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then generate the labels for the dataset. The labels are kept in an array of the size of the dataset, and are given the value 1 for cytoplasmic proteins, or 0 for periplasmic ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gives value 1 to cytoplasmic proteins, 0 to others.\n",
    "labels = np.zeros(len(all_raw), dtype=int)\n",
    "labels[0:3000] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural network training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The neural network we designed needs a one-hot encoding for the labels. One-hot encoding means that the value is stored in a sparse array, where all the values are 0 except the one corresponding the the class. For example, if there are 5 classes, an item with class 3 will be encoded as follows: [0, 0, 0, 1, 0, 0]. We designed a function that transfroms an array of integers to that format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One hot encoding for neural network.\n",
    "def one_hot(labels):\n",
    "    encoded_labels = []\n",
    "    number_classes = len(set(labels))\n",
    "    for item in labels:\n",
    "        encoded = np.zeros(number_classes, dtype=int)\n",
    "        encoded[item] = 1\n",
    "        encoded_labels.append(encoded)\n",
    "    return np.array(encoded_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "f_labels = one_hot(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, the way the network is designed requires the data corresponding to a certain sample to be stored with it's label in an array, giving a dataset on the from [[sample1_values, sample1_label], [sample2_values, sample2_label], ...]. The following function adapts the data to this requirement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepares input for neural network.\n",
    "def nn_input_format(dataset, labels):\n",
    "    encoded_data = []\n",
    "    for data, lab in zip(dataset, labels):\n",
    "        value = (data.reshape((-1,1)), lab.reshape((-1,1)))\n",
    "        encoded_data.append(value)\n",
    "    return encoded_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = nn_input_format(scaled_transfrom, f_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we proceed to training phase. Here, we will try to tune the hyperparameters to get the best accuracy. The parameters we will adjust are: the batch size, the $\\eta$ value which is the learning rate, the $\\lambda$ value which is the regularization parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross validation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To assess the accuracy of the network, we need to use k-fold cross validation. In k-fold cross validaton, the dataset is divided in k folds, and one fold is used as testing set while the others are used to train the classifier. One this step is done, the process is repeted using another fold as testing set and reimplementing the previous one in training process, and so on until all folds have been use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10-fold cross validation.\n",
    "def nn_10fold(arch, epoch, eta, batch_size, lbda):\n",
    "    # Shuffle dataset.\n",
    "    np.random.shuffle(x)\n",
    "    n = len(x) / 10  # size of fold.\n",
    "    tot_result = 0\n",
    "    # trains and test on each fold.\n",
    "    for i in range(10):\n",
    "        network = nn.ShallowNetwork(arch)  # Initialize network.\n",
    "        # Test fold.\n",
    "        x_test_1 = x[int(i * n):int((i+1) * n)]\n",
    "        # Train fold.\n",
    "        x_train_1 = x[0:int(i * n)] + x[int((i+1) * n):]\n",
    "        # Training step.\n",
    "        network.stochastic_gradient_descent(x_train_1, epoch, batch_size, eta, lbda)\n",
    "        # Current model score init.\n",
    "        result = 0\n",
    "        # Test phase using test set.\n",
    "        for elt in x_test_1:\n",
    "            result += network.predict(elt[0]) == np.where(elt[1] == 1)[0][0]\n",
    "            result /= len(x_test_1)\n",
    "            tot_result += result  # Adds fold result to total result.\n",
    "    # Averages the results.\n",
    "    final_res = tot_result / 10\n",
    "    return (final_res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Eta and batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will assess the effect of the learning rate $\\eta$ on the accuracy. $\\eta$ is one of the most important parameters in the learning process of neural network. It defines how fast the network will learn from the gradient descent; too low, it will learn very slowly and need a longer training time to be able to make reliable predictions, too high it might miss the optimum by jumping from one side to the other due to too long steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try different values of $\\eta$ for different batch size. The batch-size defines how many data points will be used to perform one step of gradien descent. Low batch size will take longer time to train, and larger batch size train the network less well. We use size that are power of 2 for purely computational purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will set the last parameter $\\lambda$ to 0.5 for now because of computation time, and because we know be pre-testing it that at least it does not break the learning. We will come back on it afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "lr = [0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.5, 0.7, 0.9]\n",
    "for eta in lr:\n",
    "    # Parameter = architecture: 64 inputs, 32hn, 2 outputs(0 or 1).\n",
    "    results.append(nn_10fold([64, 22, 2], 100, eta, 8, 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9918138396809553,\n",
       " 0.9938180151640352,\n",
       " 0.9934830095906728,\n",
       " 0.993483846639324,\n",
       " 0.9946521827550742,\n",
       " 0.9941533017714071,\n",
       " 0.9934841262782484,\n",
       " 0.9933169021979833,\n",
       " 0.9943194091633256,\n",
       " 0.9946530198075985]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_2 = []\n",
    "lr = [0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.5, 0.7, 0.9]\n",
    "for eta in lr:\n",
    "    # Parameter = architecture: 64 inputs, 32hn, 2 outputs(0 or 1).\n",
    "    results_2.append(nn_10fold([64, 22, 2], 100, eta, 16, 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9898096637294949,\n",
       " 0.9923143952341837,\n",
       " 0.9929832915537082,\n",
       " 0.9936513503585171,\n",
       " 0.9929844082420581,\n",
       " 0.9936516295306038,\n",
       " 0.9943196883354133,\n",
       " 0.9946521822913359,\n",
       " 0.9938185744387846,\n",
       " 0.9939855193469664]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_3 = []\n",
    "lr = [0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.5, 0.7, 0.9]\n",
    "for eta in lr:\n",
    "    # Parameter = architecture: 64 inputs, 32hn, 2 outputs(0 or 1).\n",
    "    results_3.append(nn_10fold([64, 22, 2], 100, eta, 32, 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9881413308723073,\n",
       " 0.9913138415413331,\n",
       " 0.9924821776570797,\n",
       " 0.9928163466455301,\n",
       " 0.992817463333879,\n",
       " 0.9936502336701674,\n",
       " 0.9931507948068423,\n",
       " 0.9923163494372504,\n",
       " 0.9929841290699706,\n",
       " 0.9946527406355118]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f52ef2bb2e8>]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hUZfrw8e+d3ilJqAkJTSBIKFIUkKYoqMDaABULKri67vtzXUFddNdFXazrrrqrKGBbO64KKwqCShOldwRDIIUSQkjvk3neP84EJiEkE9Iz98drrpk5c86ZZ47huc/TxRiDUkop9+PR0AlQSinVMDQAKKWUm9IAoJRSbkoDgFJKuSkNAEop5aa8GjoB1REWFmaio6MbOhlKKdWkbNmy5aQxJrz89iYVAKKjo9m8eXNDJ0MppZoUEUmoaLtWASmllJvSAKCUUm5KA4BSSrkpDQBKKeWmNAAopZSb0gCglFJuSgOAUkq5qSY1DkCdv13JmeQXlzC4c+uGTooCKMqDU/GQFgdZR6DTxdBhAIg0dMpUI2CMIS23iEMnc9l5NJFvkj/izQl/oaW/f61+jwYAN1BQXMKd72ziZE4hT0zoze1Doxs6Se6hpBjSE+DUQSujP/2Ih6zks/cP7wl9p0LsFAjpUP/pVfUuu6CYwyfziD+Zw6GTuWceqblkF9rwDNyPX4dPEI8i1idey9U9htTq92sAcAOfbkkmNbuQ2IgW/GXJHo5m5vPwlT3x8NC7zRqz2yH7aNnMvfR1+mEwJWf29WsJod0gerj1HNrVegS2gV+Xw/YPYeUTsGoudBkFfW+CnteAT0DD/DZVKwptJSSm5ZXJ4OMdz6nZhaf3E4EOLfzpEh7IpP5tOSpfsCnjM6KCu/LiyOfpEdq91tMmTWlFsIEDBxqdCqJ6ikvsjHr+B9qE+LL4t0P5y5Ld/OenRCb168BzN8Ti6+XZ0Els/IyBvDRIK38nf9CqxrHln9nXy79s5h7a7cwjwIXqt7SDsOMj65GZCD7B0HuSFQw6DQUPbbZrjErshqMZ+RVk8jkcSc/H7pTNhgX50Dks0PEIOv06KjQAP29PjuYcZfaa2exI3cENF9zAw4Mexs/Lr0bpE5EtxpiB5bdrCaCZ+2LbEY5k5DN3Um88PYQnJ11Ix5YBPPvNL5zIKuT1Wy+ihb93QyezcSjMPpPJn4ovm9kXZJ7Zz8MLWkVbmXrX0VZG39qR2Qe3r1kmHdoVxsyBUY9C4o9WqWDPF7DtP9CykxUIYqdY+6l6ZYzhZE6RI4PPsTL41FwOp+VyOC2PIpv99L6BPp50CQ+iX2Qrru0fQRdHJh8dFljpv7dViat4fP3j2I2d50c8z7jO4+r0N2kJoBkrsRvG/n01vt6eLPt/wxGnBsbPtyUze/FOuoQF8fadg2jfonYblxotW6FVNVP+Tj7tIOQcL7tvi8iymXvpnX3LKPCsx3unolz45SvY/gHE/wAYiLwY+t0Eva8Fvxb1lxY3UFW9fClvTyEq1MrYu5y+ow+kc3gg4UG+Zf69VaWopIgXN7/IB798QExoDM+PeJ5OIZ1q7TedqwSgAaAZW7rjKL//cBv/unkAV8e2P+vz9XEnuee9LQT7efH29MH0aBfcAKmsA/YSyEwqm7mXZvaZSWDO3KkRGO6UwTs9t+4C3o0wKGYegV2fWCWDk/vByw96XAX9boYuo+s3MDVhpfXy8U6Ze2m1zcmcsvXyHVv6l8nko8MC6RIWRMdW/njWQjtaQlYCs1bPYt+pfUzrNY0/XPQHfDx9anxeZxoA3Izdbrjq5bUUl9hZ8YeR5/xD3Xs0i+lvbySvqIT5t17E0K5h9ZzS82QM5KSUq5c/aPW4ORUPJUVn9vUJLpu5n87ku4J/y4b7DTVhDBzdarUV7PoU8tMhqC30udEKBm17N3QKG1xpvXz8yVwOn0e9fJfwQDq1turl68pX8V8xd8NcvDy8eGrYU4zuNLpOvkcDgJv5dm8KM97dzIs39uX6iyIq3fdIRj53LNpIQloeL0zuy8S+jagLYn6GU718ucy+KOfMfp6+0LpzuUy+m5XJB7Vp3v3rbUVnehH9uhzsNmgXawWCC2+AoLPWAWk2zlUvf+hkLglpeRSVnCntBfl6OWXyVgZfekcf4le/7WD5tnye2fgM//31v/QL78dzI56jfdDZpfTaogHAjRhj+M2/f+RUbiHf/XEU3p5VN0pm5hUz473NbDx0ij9d1ZMZl3apVh1mjRTnl2t0daq2yTt5Zj/xsBpCnTP30sy+RQR4aI8mctNg92KrveDYdqvButtYq73ggnHg5dvQKTwvWQXFZ+7iHRn84bSz6+V9PD2ICg0ok9Gfb718XYlLj2PWmlkczDjIXX3u4r5+9+HtUbcBSAOAG1n360mmLfyZp6+9kFuGRLl8XEFxCX/8dAdf7TzGHUOjefyamFqp4ywjJ9WqsnDO7MsPigpuXzZzL72rbxXdZDOwBnFiH+z4EHZ+AtnHrHEIF15vlQw6XtRoS0XbEtP5+dCpatXLdw4PoktYIB1a1k69fF0wxvB53OfM+3keAd4BzBs+j6Edh9bLd2sAcCNT5m/gcFoua2aPrnY/f7vd8PSyfSxcd4jxF7bjpSn9aq8OtCALFl4Bqfusniuh3c/uM9+6C/g2k8boxsJeAvHfW+0F+/5njVsI7W6VCmKnWKWnBmYrsbN8TwoL1sWzLTEDgLAgX6cM/kxvm8g6rpevC7nFuczdMJdlh5YxpN0QnhnxDGH+9dfepuMA3MSmw6f4+dApHr8m5rwGeXl4CI9fE0P7Fn48vWwfqQt+ZsHtA2kZUMNeCSU2WHwnpP0K0/4LXcc02jvQZsfDE7pdbj0KsmDvF1YwWDUXVj0JnUdYpYJeE8AnsF6TllVQzCebknhr/WGOZOQTHRrA3Em9mdS3Iy0Cmsf4lL1pe5m1ehbJOcnc3+9+7u5zN56NpLpSSwDNzO2LNrLrSCbrHh5NgE/N4vtXO4/xh4+3E9nan7enDyaydQ2mJPj6Yfj5dZjwT7jojhqlS9WSU4dg58dWNVH6YfAOhJhJVskganidjjpOOpXH2z8e5uNNSeQU2hjSuTV3X9qFMT3bNNoqnOoyxvDBLx/w4uYXaeXXimcvfZaB7c66Ca8XWgXkBnYlZzLh1XXMurIHvxvdrVbO+XN8GjPe3Yyvtydv3TGICzuex6CjjW/CsofgkvvhyqdrJV2qFhkDiT/Bjg+sUceFWdYguNgp1sjjsNr5WwLYmpjOwrWH+Hr3MTxEuCa2PXcN70KfiOY1mC2zMJM/r/8z3yV9x4iIETw17Cla+bVqsPRoAHADv31vC+sPnmT9I2NqtVvbgZRs7li0kcz8Yl6bdhEjLqhGt8K4VfD+jdB9LEz9QHvqNHbF+dao4x0fwsHvrEFzEYOtWUovvA78q5+J2UrsrNibwoK18WxNzCDEz4ubh0Rx+9CoZjkCffuJ7Ty85mFO5J3ggYse4NaYW/GQhp3DSQNAM3cgJZsrXlrD78d0449X9Kj186dkFXDHW5uIT83h43suoV+kCwOoTvwCC8daXTfv/EYbd5uarGNnRh2n7rPGWvQYb5UKul0GnpXfZGQXFPPxpiTe/vEwyen5RIUGcOewztxwUQSBvs2v+dFu7Ly9521e3voy7QLb8fyI5+kT3qehkwVoAGj2HvhoGyv2prDu4TG0DqzdYeSl0nIK+c2/11NQbOfL3w2jQ8tK7t5y02DBGGvhkxmrrCCgmiZj4NgOq1Sw61NrZtTAcOgz2WovaFc2k0tOz+Pt9Yf5yFG/P7hza+4a3pnLe7VtNvX75aXlpzFn3RzWH13P2KixPDH0CUJ8Qho6WafVKACIyDjgn4AnsMAY80y5z6OARUA4cAqYZoxJdnz2LHC1Y9cnjTEflzv2FWC6MSaoqnRoAKjY4ZO5jHnxB+4a3pk5V8fU6XcdSMnm+n//SETrABb/9pKK7+RshfDub+DIFpi+DCIapuFL1QFbEcSttNoL9n8D9mJo2wf6TmVX6BW8viWHb3Zbk+pZ9fudiY1ootNtuGjT8U08vOZhMgszmT1oNpN7TG4UA86cnXc3UBHxBP4FjAWSgU0issQYs9dptxeAd40x74jIGGAecKuIXA0MAPoBvsBqEfnaGJPlOPdAoHn/ddSD11cfxMvTgxmXdqnz77qgbTCv3NyfO9/exP99tJ35t15U9q7OGFj6gDWV8Q2LNPNvbrx8oOdV1iPvFPZdn5H107u0XDGHXuZxJks/Lo+5jiHjptEhrOEaPetDib2E+TvnM3/nfDoFd+K1y1+jR+var36tS660TAwG4owx8caYIuAjYFK5fWKAVY7X3zt9HgOsNsbYjDG5wA5gHJwOLM8Ds2v2E9zb0Yx8PtuazJSBkbQJqdmiEa4a1aMNf5nQm5X7Unjum1/KfrjuJevucNSfrFGnqlnKKbSxaGsmI1d3pd+xR7jV7xX2dr6DS4OPcW3cY3RY0Ne6EUjaaN0UNDMn8k4w49sZvLbjNa7pcg0fX/Nx7Wf+xfnW7K/Hd0H8aut9LXOlJaYjkOT0PhkovzDlDuB6rGqia4FgEQl1bP+LiPwdCABGA6Ulh/uBJcaYY5UVl0RkJjAToFMnrUcu74018RgD94ys+7t/Z7cPjSbuRA7z18TTNTyIyYMiYe8SWPVXawKykRrXm6MjGfm8vf4QH21MIrvQxqDoVsy5KoaxMW3x9LgN7C/AoTWOKSg+hi1vWdN69L0J+k5pFm1B646s409r/0RBSQFPDXuKSd3K3w+XY7dDQYY1Y2veKcg/Vclz+pn3tnIZ/u82QfgFtfpbXAkAFeXO5UP6Q8CrInIHsAY4AtiMMStEZBDwI5AKbABsItIBuBEYVdWXG2PeAN4Aqw3AhfS6jdTsQj7cmMi1/TsS0ar+1439y4QYDqfl8qfPd9GLg/RZPtPqMjjpXzrKt5nZnpTBgrXxfO2o37+6j1W/37d8bzAPT2uVtK6jrRXW9i6xgsH3T1mP6EutYBAzscn1Ciu2F/PK5pd4a997dA+K5IU+99Ol0MDmRY4M/BwZfEFG2TUonImH1bXWv7W1ZGiLCGgfa20LaH1mu39raNGx1n9TlY3AInIJ8IQx5krH+0cBjDHzzrF/EPCLMeasCUZE5APgP1hBZSFQ4PioExBvjKl0xIk2Apc17+t9vLkmnpUPjqRLeJVt6HUiM7+Yma8u4ZXcP9I6OBCve763pl9WTV6J3fDt3uMsWHuIzQnpBPt5cfPgTtw2NJqOlfUAq0hGIuz42KoePBUP3gHW1BN9b7Kmoqjv8SEu35Vbd+RHCtKZHVDCTl9vJmdlM+tUBn7l807vAEeG3apsxl3hsyOD921RL+s812QuoE1AdxHpjHVnPxW4udzJw4BTxhg78ChWj6DSev6Wxpg0EYkFYoEVxhgb0M7p+JyqMn9VVkZeEf/ZkMDVsR0aLPMHaOFZxLsBL2HLzeeekif4u0crmteYTveTU2jj081JLFp/iKRT+US29ucvE2K4cWAkQefbf79lJxg5C0Y8ZLUL7PgQdv/XqiYK6Xhm1PH5VHEUF1SegZ/PXblfy9MZ9qqgYB73z8GID8+HDWNc7wEVZ+ze9dMGV5uq/L9pjLGJyP3AcqxuoIuMMXtEZC6w2RizBKsqZ56IGKwqoN85DvcG1jrq+LOwuofayn9HY5VTaMPf27NR9l1++8fD5BaV8LvRDbg4uN0O/52J78k9HL7sTdZ8E8DvPtjKW9MHubQGgWpcjmTk886Ph/lwYyLZBTYGRrVizlW9GBvTrvb+DYhApyHWY9wzsH+ZNTHd+n/Cur9b01T3vQna9zt9911lfXlx3rm/r/xdeYs+Lt+VF5YU8uLmF/nwlw/pHdqb50c+T2RwZO1ch0ZCB4KdQ16RjUvmfUfHlv7MndSbgdGt6+V7XZFTaGPYM98xuHNr3rytAbtZrnzC6vUz7hm4+F4+3ZzErMU7uWVIJ576zYWNri+0qtiOpAwWrDvEsl3HALjKUb/v0mjv2pKdYg0y2/EhpOw++/Nyd+UVZty1eFfuvE7vrTG38ocBf8C7ipHPjZlOB11N25MyyMwvxlZi54bXN3DdgI48Mr4nbYIbvpj3n58SyMwv5v5amvDtvGx738r8B94JQ34LwI0DIzmYmsvrqw/SrU0Q04d1brj0qUpZ9fspLFwXz6bD6QT7enHX8M7cfj71+7UhuC0Mvd96HN8NWUfKZvB+LeulrhzOrNPr7enNK2NeYVTkqHr53oagAeAcShelWPnHkby3IYE318azYk8KD1zenduHRjdYFUdBcQkL1sZzafews3tg1JeEH2Hp/0GXUTD+uTI9fmZf2YP41Bye/N9eosMCGd1DG4Qbk9zT9fuHSTyVR0Qrf/58TQyTB9Wgfr+2tbvQetSzvOI8ntn4DJ/HfU7/Nv15bsRztAtsV/WBTVgj+T/e+GxJSKdbmyDat/Bn9rie3HBRBH9dupenvtrHJ5uTeGJib4Z2rb8VfUp9tDGRkzlFDXf3fyoePrrFWoD9xnfOmhDMw0P4x9R+3Pj6Bn7/wTY+u3coPdo1re5+zdHRjHze2XCYD3626vcvimrFo+N7ckXvWqzfb8Li0uN4aPVDxGfGM6PPDO7rdx9eHs0/e2z+v/A8GGPYmpjOlTFnon+X8CDenj6Ib/emMPd/e7n5zZ+5JrY9c67uVfdT2pYUg3hSZIf5a+IZFN2KIV1C6/Y7K2Dy0jEfTLaGj9/8MfhXXAIJ8PFiwe0DmfTqeu58exNf3j+MsCBdy7ch7EzOYMHaQ3y16xjGGMY76vcHdGre0zS4yhjDf3/9L89sfIYA7wBeH/s6QzvUzzq9jYEGgArEn8wlI6+YAVFlMzgR4Yre7RhxQTivrz7Iaz8cZNW+E/z+sm7cNbzzeS3BWKXcNFh0JYR04IseL3Ess4Bnro+t/e+pgrEVMefTq9jml8uro5+ja+vKRx63b+HPgtsHMnn+Bu55bwvv3z2kya3j2lSV2A0r96WwcO0hNh4+RbCvF9OHRnP70OiarerWzOQU5TB3w1y+Pvw1Q9oP4ZlL63ed3sZA++pVYGtCOgAXRVV8l+Tn7ckDl1/AygdHcmn3MJ77Zj/j/7GW1QdSazchtkL4eJq1XN+h1QStnEWfDiGM6F7Pf6TG8M2Xt7OUHE74+HLrzpfYkrKlysNiI1ry98n92JKQziOf7aQp9ThrinILbbzz42HGvPgD97y3hSMZ+Tx+TQw/PjqGx66J0czfyd60vUz+32SWJyzn9/1/z/zL57td5g8aACq0NTGdED8vuoRVPsAqsnUAb9w2kLenD8Jgrcc7893NJJ2qpF+yq5xn1bz2dX7pcR9X2VbxQscf6r175cn1f+fpzB308W7FF79ZSqhfKDNXzGTF4RVVHntVn/Y8dMUFfLH9KP/6Pq4eUut+jmXm88zXv3DJvFX8ZckeQgN9+PctA1g9axR3De9McC2uDtfUGWN4f9/7TFs2jaKSIt668i1mxs5sNIu01zetAqrA1oQMBkS1wsPFxrFRPdpwSddQFq47xCur4rj876u5b1Q37hnZ5fyrPdb/wzGr5qPYe1/P71eE8SfvXYza9SLE9LOG0dcDs385f931GgUBATw1/i0iQyJ5b/x7/P673/PQ6oeYnTebaTHTKj3H70Z342BqLi+sOEDnsCCujm1fL2lv7nYlZ7JwXTz/23kMuzGMv7A9dw7vfM6Sq7tzXqd3ZMRInhr2FC393Hs2eg0A5WTmF3PgRHa1MylfL0/uG9WN3/TryNPL9vHSygMs3prEn6/pzeW92lR6115aNXK6hmTfEjxWPoH9whswl85mxZ7j/JqaS86N/0S2zoT/zoTpX0OHfuf7M12TsoclX9/LD60Ceaj/7+nSyhp13NKvJW9e8SaPrH2EZzc9y/Hc4zw48MFzrnsqIsy7rg+Jp/L446fbiWjl33BdWJu4Erth1b4UFqw7xMZDpwjy9eL2odHcofX7ldp+Yjuz18wmNT+VWQNncWvMrTpQER0JfJbVB1K5fdFG3r97CMO6nX+d4I9xJ/nLkj38eiLndDd5Vy51bznEpz5z2W8imVr0GIVYyztGhwaw8sGReOWfhDfHgN0GM76DkA7nncZK5Zzg+IIxXNcCuof3YdFV/zmrmFxiL+HZTc/y4S8fMj56PE8Nfwofz3MvR3kyp5Df/Gs9yen5hPh50SrQh5YBPrQK8KZVgA8tHc+tArwd2x3bAq1t/t6ebvuPNq/IxuItySxad4jDaXl0bOnP9GHRTBkUqVU8lbAbO2/tfotXtr1Cu8B2vDDyBS4Mq/8xBg1N1wR20UvfHuCV735l5xNX1nhgTHGJnc+2JHM0w2leb0cG5pyNleZpQYWpTN1xO3bx5JN+b5PnE3Z6v9E923BhR8c0ayl7YOEVENrVKgn4BNYonWcnPB/zzjX8tiSZbQFBfDbpCyJDKp4DxRjDW3ve4qUtLzGo3SD+Mfofla6FmnQqj8+2JpOeW0R6XjHpeUVkOD3nFJ57qigfL4+zgkWFASTwTABp4e/dpPu5H88sON1/PzO/mH6RLZlxaReu7N0WL51vqVLO6/ReEXUFTwx9gmAf9xyTogHARbcu/JmTOUV8/X+X1un3nKUoF94aD2kH4c7lVY+EPLAcPpwKPa6Cye/V3jB5Y+Czu/gkYTlPhrVmzpA5TO05tcrDvor/isfWP0Z0SDSvXf7aeY+gLLLZych3BAVHkMjIc36uaFsxJfaK/45FIMTP26lUUTZwtAysqATig79PwzYK7j6SycJ1h1i64yh2Yxh3YTvuGt5F6/ddtPHYRh5e+zBZhVk8PPhhbrzgRrctPYLOBeSSErthe2IGk/rXUbXKudjt8Pk91tJvN33k2jD4C66EK/8G3zwC382Fy5+onbSsfpbkX77ghU5RDGl3EZN7THbpsKu7XE2YfxgPfP8Atyy7hdcuf40LWlV/al8fLw/aBPtVa84lYwzZhTYycosdAeJMqaJ8sEjNKeRASg4ZeUXkFpWc85y+Xh7nKFWUCyBO21r4e7vccaAidrth1S8nWLA2np8PnSLQx5PbLolm+jCt33dVib2E13e+zvwd84luEc3rl7/e5NbprU8aAJz8eiKb7EJb/Y+S/O5J2LcUrpxnZeyuGvJbOHnAmpQttBv0r7w3TpV2Lcb+wzwe79YHDw8bTw598pwNuxUmp/0Q3h73NvetvI/bv76df47+J4PbD65ZmlwgIoT4eRPi502nUNczykJbCZl5xU5VUeWqpZxKIPuPZ5ORV0xGfuWljRb+5auozm7XOP060PrMbgyfbUlm0frDHDqZS8eW/jx2dS8mD4okROv3XZaSm8Ijax9hc8pmJnadyJwhcwjw1sBZGQ0ATrYmWBPA1Wsxe/sH1jzoF02Hi++t3rEi1mRsp+KtMQOtoiF6+PmlI2kjfHEfH0bFsrkkg78O+Svtg6rfXbNH6x7856r/cO/Ke7ln5T38bfjfGN95/PmlqY75ennSJsSTNiGulzbsdkdpo0ywKCI9t/isAJKSVcD+49mk5xWRV0lpw0PAbqBvZEtevbk/43q30/r9alqTvIbH1j1GQUkBTw9/moldJzZ0kpoEDQBOtiSkExroQ6f6Km4n/AhL/h90HglXPX9+6+h6eluTsi0ca40avnuV1ThcHekJ8NHNHG7Znn9453Npu0u5ttu11U+LQ/ug9rwz/h3+7/v/Y/aa2aTkpnB779ubRR2sh4fQwt+bFv7eRFVjOqZCW8mZaqlywSK30MZlvdowoFOrZnGN6lOxvZhXtr7CW3ve4oJWF/D8yOfp0qLyaUrUGRoAnGxLTGdAVD39IyydVbNVFEw+e1bNavFvaU3O9uZl8MEUuPtbaw51VxRkwYdTKSkp4rFO/fDJO8YTQ5+o8TVo4duC+WPnM2fdHF7c8iLH844za+Astx1x6evlSdsQT9pWo7ShKnck5wizV89m58mdTOkxhYcGPoSfl17f6tBypsOp3CLiT+bWT/VPfgZ8MBUwcPMnrmfWlWndBaa+b80b9Mlt1gyiVSmxweI7IXU/7w69jR0Z+3l0yKO0CaidOfx9PX15bsRz3BpzK+/ve59Za2ZRYCuolXMr9/ZtwrfcuORG4jPjeWHkCzx28WOa+Z8HDQAOpRPA1XkDcIkNPr3DKgFM+U/1q2sqEzUUJr4Mh9bAsoeqHnm24jGI+5a4yx7llYRlXNbpMq7ufHXtpQfwEA9mD5rNrIGzWJmwknu+vYfMwsxa/Q7lPgpLCnn6p6d58IcHiQqJ4pMJn3BldDU6TqgytArIYWtiOl4eQmxEi7r7EmPg69kQ/z1M+tf5N9hWpt/NcPJXq2E57AK45HcV77dpAfz8GsVD7mVOxmaCvIN4/OLH66z667bet9EmsA1/Wvsnblx6IzdccAMTukw4r4Zm5Z4OZx5m1ppZ/HLqF26LuY0HBjzQpNfpbQw0ADhsSUind4eQup2zfuMbsHkhDPu/mnfZrMyYxyHtV1g+x6oa6lGuF87B72DZbOh+JQvbR7F3x1e8OPJFQv3rdpGZcdHjaOPfhpe3vcwr217h1W2vMqjdICZ2ncjYqLHaZU+d09KDS3nypyfx8fTh1TGvMjJyZEMnqVlwqQpIRMaJyH4RiRORRyr4PEpEVonIThH5QUQinD57VkR2Ox5TnLYvFJEdjmMWi0jlcy/XoeISOzuTMxlQl/X/v35rDdrqeQ1c9kTdfQ9Yo4KvnQ/t+8Liu6wBZqVS98Mnd0B4T3657GHm73yD8dHjuSL6irpNk8OAtgN4e9zbfH3d19zb716O5hzlsfWPMeqTUcxZN4efj/2M3djrJS2q8csrzuPx9Y/zp3V/olfrXiyesFgz/1pU5VQQIuIJHADGAsnAJuAmY8xep30+Bf5njHlHRMYA040xt4rI1cADwHjAF1gNjDHGZIlIiDEmy3H834ETxphnKktLXU0FsSs5kwmvruOVm/ozoW8djAJO2WvN3dM6GqZ/A771FOuyjlkTx4kHzFgFHl7W++J8iu/6hqnrZnOq4BSfT/y8wabFNcaw7cQ2lhxcwvLDy8kpzqFdYDsmdJnAxK4TiW4R3SDpUg3v1/RfeZpxrIgAACAASURBVGj1QxzKPMSM2Bnc2/det1inty7UZCqIwUCcMSbecaKPgEnAXqd9YoA/OF5/D3zhtH21McYG2ERkBzAO+MQp8xfAH2iwSYm2Jla+AliN5KRaXTN9AuGmj+sv8wcIaQ83fWjNMfThTeDlCzkpcMdXvHb4Kw6kH+CVMa806JzoIsKAtgMY0HYAjwx+hO+TvufLg1+ycPdC3tz1JrHhsUzqOokro6+khW8dts+oRsMYw2e/fsYzG58hyDuIN654g4vbX9zQyWqWXAkAHYEkp/fJwJBy++wArgf+CVwLBItIqGP7Xxx3+AHAaJwCh4i8BVzl2PbHir5cRGYCMwE6derkQnKrb0tCOu1C/OjQspYXdy8ugI9uhtxUmL4MWnSs3fO7okM/uO5Na5AYBm54i12+vizcvZBJXScxKnJU/afpHPy8/BjfeTzjO4/nRN4Jvor/iiUHl/DkT0/yzMZnGBU5ikldJzG041C8PbTxr7lJykpidfJqViauZEvKFi5pfwl/u/RvbrlUY31xJQBU1C2k/N36Q8CrInIHsAY4AtiMMStEZBDwI5AKbABOz/drjJnuqGJ6BZgCvHXWFxnzBvAGWFVALqS32rYmptf+3b8xsOT3kLzRGqnbcUDtnr86el0D1y+AkiIKel7FnP9NJtw/nIcHP9xwaapCm4A2TL9wOnf0voN9p/ax5OASlsUv49uEb2nt15qru1zNxK4T6dm6Z0MnVZ2nYnsx209sZ03yGlYnr+ZQ5iEAurTowkMDH+LWmFurNReVqj5XAkAy4DwZfARw1HkHY8xR4DoAR2Pu9caYTMdnTwNPOz77APi13LElIvIxMIsKAkBdS8kqIDk9n+nDOtfuide8ALs+gTGPQe/f1O65z0efGwB4ddMLHMo8xPzL5zeJudFFhJjQGGJCY/jjRX9k3ZF1LDm4hA9/+ZD39r7HBa0uYGLXiadnI1WNW0ZBBmuPrGVN8hrWH11PdlE2Xh5eDGo7iCk9pjCi44hzrj2hap8rAWAT0F1EOmPd2U8FbnbeQUTCgFPGGDvwKLDIsd0TaGmMSRORWCAWWOGo9+9qjIlzvJ4A/FJbP6o6zgwAq8V68D2fw/dPQewUuPSh2jtvDW1N2cq7e9/lxgtuZGjHoQ2dnGrz9vRmdKfRjO40moyCDL4+/DVL4pbwwuYXeGnLSwzrOIyJXScyKnIUvp6+DZ1chVWfH5cRx+rk1axJXsOO1B3YjZ1Qv1Au63QZIyNGckmHSwj0ruVFjZRLqgwAxhibiNwPLAc8gUXGmD0iMhfYbIxZAowC5omIwaoCKh195A2sdQwuygKmOc7nAbwjIiFYVUw7gGpOhVk7tiam4+PlQe8OtdTAeGQLfP5biBwCE185vwne6kBecR6PrX+MDkEd+OPACptbmpSWfi25qedN3NTzJuIz4llycAlL45eyJnkNwT7BjIsex8SuE+kb3lcnWKtnhSWFbDy2kTXJa1iTvIajuVaFQa/WvZgZO5ORESOJCY3R6p1GwO1XBLvu3+vxEGHxvbVwR5yZbHWz9PKFu7+DoPAan/Jk/kle2/4aBzMPUv7/laF0MXlT9j3mdCtN6basoiwSshJYdOUiBrUbVON0NUYl9hJ+Pv4zSw4uYVXCKgpKCogKiWJClwlM6DqBDkH1vNCPGzmRd+J0Xf7Px34m35aPv5c/F7e/mBERI7i046W0DWzb0Ml0W7oiWAUKbSXsPpLF9GHRtXCyHGuJxqI8uO3LGmf+xfZiPvrlI/69/d8UlBTQN7zv6Zk0xfGf483pbWWepez7EN8Qbou5rdlm/gCeHp4M7TCUoR2GkjMkh28TvmXJwSW8uv1VXt3+KoPbDdZRx7XEbuzsTdvL6uTVrE5azb5T+wBoH9ieSV0nMTJyJIPaDdKquEbOrUsAWxLSuf61H3l92kWMu/D81rAFrCUdP54GB76Gmz+F7pfXKF0bj21k3sZ5xGXEMazjMB4Z9IgOiKqB5OxklsYvZenBpSRlJ+Hv5c/YqLFM7DqRQe0GaVWEi3KLc9lwdAOrk1ezNnktaQVpeIgHfcP7MiJiBCMjRtKtZTetcmuEtARQgW2OAWADomrYALzqCdj/lbU6Vw0y/+O5x3lh8wssP7ycjkEdeXn0y4yKHKX/oGooIjiCe/vey29jf8v21O18Gfclyw8vZ8nBJadHHU/oOoHOLWq5J1gzkJSVxJoja1idtJpNKZuw2W0EewczvONwRkSOYHiH4Q06kFDVjFuXAO79zxZ2H81k7ewx53+Sbf+BL38HA++Cq188r0bfopIi3tnzDm/uehO7sXNXn7uY3nu6zm9ehwpsBadHHW84ugG7sRMbHsvELhMZ13mc2446ttltbDuxjbXJa1mdvJr4zHgAOrfozMiIkYyIGEG/Nv10IF4Tc64SgNsGAGMMF89bxSVdQvnH1P7nd5LD6+Dd31jTOt/y6Xmt6rUmeQ3PbnyWxOxELut0GbMGzaJjUAOMGHZjqXmpfBX/FV8e/JK4jDi8PbzdatRxRkEG646uY03SGtYdXVemb/7IyJHaN78Z0Cqgco5k5JOSVXj+I4DTDlr1/q07w41vVzvzT8pK4rlNz/FD8g9Eh0Qz//L5TbJvfnMQHhDOHRfewe29b2ffqX0sPbiUr+K/arajjkv75pd209yeuh27sdPar7X2zXczbhsAtjgGgPU/nxXA8tOtCd4Qay1ef9frQPNt+SzctZC3dr+Fp4cnD170INN6TdOFLRoB51HHDw58kHXJzWfUcWFJIZuOb2J10uoK++aP6DiC3mG9tUHczbhtANiWmEGAjyc921VzOoSSYmtJx/TDVnfP1l1cOswYw8rElTy/6XmO5R7jqs5X8eBFD2rf6EbK26PsqONvDn/DkoNnRh0P7TCUid0mMjpydKPt6ngi78Tpuvyfjv1Evi0fP08/Lu5wMTNiZ2jffOW+AWBLQjp9I1ri5VmNO57TSzr+AJP+DdHDXDosPiOeeRvn8dOxn7ig1QX8bfjfGNjurOo41Ui19GvJ1J5TmdpzaplRx7NWz2pUo46d++avSV7D3jRr4t32ge2Z2HUiIyOsvvnauUCVcstG4LwiG32eWMG9I7vy0JU9XD/wp9fhm4dh2AMw9q8uHfL6jteZv2M+/l7+3N//fib3mKyLWjQDJfYSNh7fyJKDS1iZsLLBRh3nFufy09GfTmf62jdfVUR7ATn5KT6NqW/8xKI7BjKmp4tF4F+/hQ8mQ4+rYPJ71rKLVdiXto/J/5vM2KixzBkyp87X3FUNI7c4lxWHV7A0fimbjm8CqNNRx0nZSacbcDcd30SxvZhg72CGdRzGiIgRDO84nFZ+dbi8qWpyNAA4+fcPcTz3zX62PT6WVoE+VR9weknHznDnN9bqXi548IcH2XB0A8tvWE6IT0gNU62agiM5R1h6cClLDi4pM+p4QtcJDG43+LwaWW12W5l587Vvvqou7QbqZGtCOl3DA13L/Mss6fiRy5l/fEY8KxNWcnefuzXzdyMdgzry276/5Z7Ye2o06jizMJN1R9axOnk1646c6Zs/sO1AbrzgRkZEjKBTSN2skKfch9sFAGMMWxMzuKxnm6p3Li6Aj285ryUdF+xagJ+XH7fG3FqD1KqmSkTo36Y//dv055HBj/BD0g9nrXXsPOrYGMPBjIOn6/LL980fETGCS9pfQpBPPa4prZo9twsAh9PyOJVb5NoAsP89AEk/V3tJx6TsJJYdWsYtvW7RuliFn5cf4zqPY1zncWVGHT/181M8u+lZhrQfwqHMQxzJOQJYffNn9JnByIiR2jdf1Sm3CwClA8AGVBUAck7Ajg/hkvurvaTjot2L8BAPbu99+/kmUzVTzqOOfzn1C0sOLmF18mq6t+zOXX3uYkTHEdo3X9UbtwsAWxPTCfbzolt4FUXplN3Wc/crqnX+47nH+SLuC67rdh1tAlyoZlJuSUToFdqLXqG9eHjwww2dHOWm3K5suTUhnf6dWuHhUUW/6JQ91nPbC6t1/nf2vIMxhjv73HmeKVRKqfrhVgEgu6CY/SnZXOTK/D8peyC4PQS63nc/LT+NxQcWc3WXq3VGT6VUo+dWAWB7UgbGuLgATMpuaNu7Wud/b+97FJYUcnefu88zhUopVX/cKgBsTchABPpFVhEASoohdX+1AkBmYSYf7f+IK6Kv0JWllFJNglsFgC2J6fRoG0ywXxUjJtPioKSoWvX/H/zyAbnFuczoM6OGqVRKqfrhUgAQkXEisl9E4kTkkQo+jxKRVSKyU0R+EJEIp8+eFZHdjscUp+3vO865W0QWiUidjmO32w3bEtOr7v4JTg3ArpUA8orzeH/f+4yKGEWP1tWYXE4ppRpQlQFARDyBfwHjgRjgJhGJKbfbC8C7xphYYC4wz3Hs1cAAoB8wBJglIqXzIrwP9AT6AP5AnVacx6XmkF1gc7EBeDd4eENod5fO/cn+T8gszGRGrN79K6WaDldKAIOBOGNMvDGmCPgImFRunxhgleP1906fxwCrjTE2Y0wusAMYB2CMWWYcgI1ABHXI5QFgYJUAwnuAV9VzBRXYCnh7z9tc3P5iYsNja5pMpZSqN64EgI5AktP7ZMc2ZzuA6x2vrwWCRSTUsX28iASISBgwGiizurSj6udW4JuKvlxEZorIZhHZnJqa6kJyK7Y1IZ3WgT5Eh7owNW/KHperfz6P+5y0gjRmxs4877QppVRDcCUAVDRiqvwc0g8BI0VkGzASOALYjDErgGXAj8CHwAbAVu7YfwNrjDFrK/pyY8wbxpiBxpiB4eHhLiS3YlsS0xnQqWXVC2PknYKsIy4FgOKSYhbtXkT/Nv0Z2FZX+FJKNS2uBIBkyt61RwBHnXcwxhw1xlxnjOkPzHFsy3Q8P22M6WeMGYsVTH4tPU5E/gKEAw/W6FdUIT23iPjUXNeqf05Yy+i5EgD+F/8/juceZ0afGbriklKqyXElAGwCuotIZxHxAaYCS5x3EJEwkdNTFj4KLHJs93RUBSEisUAssMLx/m7gSuAmY4y9Nn7MuWxLctT/uzoCGKrsAlpiL2HBrgX0at2L4R2H1zSJSilV76oMAMYYG3A/sBzYB3xijNkjInNFZKJjt1HAfhE5ALQFnnZs9wbWishe4A1gmuN8AK879t0gIttF5M+19aPK25qQgaeH0DfChRHAx3dBQCgEVT4j4/LDy0nMTmRm7Ey9+1dKNUkuzQZqjFmGVZfvvO3PTq8XA4srOK4AqydQReest5lItySk07tDCP4+nlXvXNoAXEmmbjd23tz1Jl1bdGVMpzG1mFKllKo/bjEd9Ks39+dkTlHVO9pL4MQ+GDi90t2+T/qeuIw45l06TxfrUEo1WW4RAEKDfAkN8q16x1OHwJZfaQOwMYY3d75JRFAE46LH1WIqlVKqfuntq7PSRWAqCQAbjm5gT9oe7upzF14ebhE/lVLNlAYAZyl7QDwgvNc5d5m/cz5tA9oysevEc+6jlFJNgQYAZyl7rPl/vP0q/Hjz8c1sPbGV6RdOx8ez6mkilFKqMdMA4KyKRWDe3PUmrf1ac1336+oxUUopVTc0AJQqyIKMhHMGgN0nd/Pj0R+5LeY2/L386zlxSilV+zQAlDqxz3o+xwjgN3e+SbBPMFN6TKnwc6WUamo0AJSqpAfQgfQDfJf0HdN6TSPIJ6ieE6aUUnVDA0CplD3g2wJanL0swYJdCwjwCuCWXrc0QMKUUqpuaAAodY4pIBKyElh+eDlTek6hhW+LBkqcUkrVPg0AAMaccxGYhbsW4u3hzW0xtzVAwpRSqu5oAADISISi7LMCwNGcoyw9uJTrul9HmH9YAyVOKaXqhgYAOOcaAG/tfgsEpveufHI4pZRqijQAwJkeQG3OTAFRWFLI53GfM6HLBNoHtW+ghCmlVN3RAABWAGjVGXzPdPFMzk6msKSQIe2HNGDClFKq7mgAgAobgBOzEgHoFNypIVKklFJ1TgNAUR6kHTyr/j8x2xEAQjQAKKWaJw0AqfsAU2EJoIVvC+37r5RqtjQAlPYAand2CUCrf5RSzZkGgJQ94B0ILaPLbE7MSiQyOLJh0qSUUvXApQAgIuNEZL+IxInIIxV8HiUiq0Rkp4j8ICIRTp89KyK7HY8pTtvvd5zPiEjDjbJK2QNtY8DjzKUoKiniWO4xokKiGixZSilV16oMACLiCfwLGA/EADeJSEy53V4A3jXGxAJzgXmOY68GBgD9gCHALBEJcRyzHrgcSKiF33F+jKlwEZjknGQMRksASqlmzZUSwGAgzhgTb4wpAj4CJpXbJwZY5Xj9vdPnMcBqY4zNGJML7ADGARhjthljDtcw/TWTfQzy08/uAeToAqolAKVUc+ZKAOgIJDm9T3Zsc7YDuN7x+logWERCHdvHi0iAo5pnNFCt22oRmSkim0Vkc2pqanUOrdrpKSB0DIBSyv24EgCkgm2m3PuHgJEisg0YCRwBbMaYFcAy4EfgQ2ADYKtOAo0xbxhjBhpjBoaHh1fn0KqdngKibI1WYnYiwT7B2gVUKdWsuRIAkil71x4BHHXewRhz1BhznTGmPzDHsS3T8fy0MaafMWYsVjD5tVZSXhtS9kCLSPBvWWZzYlYiUcFRiFQU+5RSqnlwJQBsArqLSGcR8QGmAkucdxCRMBEpPdejwCLHdk9HVRAiEgvEAitqK/E1do41ABKzE4kM0QZgpVTzVmUAMMbYgPuB5cA+4BNjzB4RmSsiEx27jQL2i8gBoC3wtGO7N7BWRPYCbwDTHOdDRP6fiCRjlSh2isiCWvxdVbMVwskDZwWA4pJijuUe0/p/pVSz5+XKTsaYZVh1+c7b/uz0ejGwuILjCrB6AlV0zpeBl6uT2Fp18gDYbRV2AbUbu/YAUko1e+47Evgci8AkZVsdnnQMgFKquXPfAHB8F3j6QuuuZTYnZFnj0nQWUKVUc+e+ASBlD7TpCZ5la8ESsxIJ9g6mlW+rBkqYUkrVD/cOAOWqf8CqAooMidQuoEqpZs89A0DOCcg9UWEX0ISsBKKCtQFYKdX8uWcAOMcUEMUlxRzNPapjAJRSbsHNA0DZKqAjOUewG7uOAVBKuQX3DQBB7SCw7DIEpesA6xgApZQ7cNMAcPYaAKBjAJRS7sX9AkCJDVJ/OWcDcKB3IK39WjdAwpRSqn65XwBIi4OSogq7gJYuBK9dQJVS7sD9AkDpGgAVVQFlJekIYKWU23DDALAHPLwg7IIym4vtxRzJOaI9gJRSbsM9A0BYD/DyKbP5WM4xSkyJlgCUUm7DPQPAORaBAV0HWCnlPtwrAOSnQ1byOXsAgc4CqpRyH+4VAFL2Ws/nmAQuwCuAUL/Qek6UUko1DDcLABXPAQTWNNCdQrQLqFLKfbhZANgF/q0huN1ZH5WOAVBKKXfhZgHA0QBc7i7fZrdxJPuI1v8rpdyK+wQAewmc2Fdh/f+xnGPYjE1LAEopt+JSABCRcSKyX0TiROSRCj6PEpFVIrJTRH4QkQinz54Vkd2OxxSn7Z1F5GcR+VVEPhYRn/LnrVXph6E4r/IuoFoCUEq5kSoDgIh4Av8CxgMxwE0iElNutxeAd40xscBcYJ7j2KuBAUA/YAgwS0RCHMc8C7xkjOkOpAN31fznVKKSKSB0DIBSyh25UgIYDMQZY+KNMUXAR8CkcvvEAKscr793+jwGWG2MsRljcoEdwDixutqMARY79nsH+M35/wwXpOwB8YA2vc76KDErEX8vf8L8wyo4UCmlmidXAkBHIMnpfbJjm7MdwPWO19cCwSIS6tg+XkQCRCQMGA1EAqFAhjHGVsk5ARCRmSKyWUQ2p6amuvKbKpayB0K7gbf/WR/pLKBKKXfkSgCoKFc05d4/BIwUkW3ASOAIYDPGrACWAT8CHwIbAJuL57Q2GvOGMWagMWZgeHi4C8k9h3MsAgNnxgAopZQ7cSUAJGPdtZeKAI4672CMOWqMuc4Y0x+Y49iW6Xh+2hjTzxgzFivj/xU4CbQUEa9znbNWFWZbjcAVBACb3UZyTrLW/yul3I4rAWAT0N3Ra8cHmAoscd5BRMJEpPRcjwKLHNs9HVVBiEgsEAusMMYYrLaCGxzH3A58WdMfc04n9lnPFXQBPZ57HJvdpiUApZTbqTIAOOrp7weWA/uAT4wxe0RkrohMdOw2CtgvIgeAtsDTju3ewFoR2Qu8AUxzqvd/GHhQROKw2gQW1tJvOpsLPYB0HWCllLvxqnoXMMYsw6rLd972Z6fXiznTo8d5nwKsnkAVnTMeq4dR3UvZA74h0OLsTD4xywoAUSFR9ZIUpZRqLNxjJPCpQxVOAQFWCcDP049w/xo0MCulVBPkUgmgyZv2GRRmVfhRYlYikSGR2gVUKeV23KMEIAJ+LSr8KDE7kahgrf5RSrkf9wgA51BiLyE5O5nIEG0AVkq5H7cOAMfzjlNsL9YxAEopt+TWAUB7ACml3JlbB4CkbGuKIx0DoJRyR24dABKyEvDz9KNNQJuGTopSStU7tw4AidmJRARH4CFufRmUUm7KrXO+pKwkbQBWSrkttw0AdmMnKTtJG4CVUm7LbQNASm4KRfYiHQOglHJbbhsAErITAF0HWCnlvtw2AOgYAKWUu3PbAJCUnYSvp692AVVKuS23DQAJWQlEBkdqF1CllNty29wvKTtJRwArpdyaWwaA0i6g2gCslHJnbhkATuSdoLCkUBeCV0q5NbcMAKU9gDQAKKXcmXsGgGxHANAqIKWUG3PPAJCViI+HD+0C2zV0UpRSqsG4FABEZJyI7BeROBF5pILPo0RklYjsFJEfRCTC6bPnRGSPiOwTkZfFsfq6iExx7L9HRJ6rvZ9UNZ0FVCmlXAgAIuIJ/AsYD8QAN4lITLndXgDeNcbEAnOBeY5jhwLDgFjgQmAQMFJEQoHngcuMMb2BtiJyWe38pKolZCVo9Y9Syu25cgs8GIgzxsQbY4qAj4BJ5faJAVY5Xn/v9LkB/AAfwBfwBlKALsABY0yqY7+VwPXn+yOqw27sJGcnawOwUsrtuRIAOgJJTu+THduc7eBMBn4tECwiocaYDVgB4ZjjsdwYsw+IA3qKSLSIeAG/ASoclSUiM0Vks4hsTk1NrWiXaknNS6WgpEBLAEopt+dKAJAKtply7x/CqtrZBowEjgA2EekG9AIisILGGBEZYYxJB+4FPgbWAocBW0Vfbox5wxgz0BgzMDw83IXkVu50DyAtASil3JyXC/skU/buPAI46ryDMeYocB2AiAQB1xtjMkVkJvCTMSbH8dnXwMXAGmPMUmCpY/tMoKSGv8UlOgZAKaUsrpQANgHdRaSziPgAU4ElzjuISJjI6S41jwKLHK8TsUoGXiLijVU62Oc4po3juRVwH7Cgpj/GFYnZiXh7eNMuQLuAKqXcW5UBwBhjA+4HlmNl3p8YY/aIyFwRmejYbRSwX0QOAG2Bpx3bFwMHgV1Y7QQ7HHf+AP8Ukb3AeuAZY8yBWvpNlUrMsrqAenp41sfXKaVUo+VKFRDGmGXAsnLb/uz0ejFWZl/+uBLgnnOc86ZqpbSWJGYnagOwUkrhZiOBjTE6DbRSSjm4VQBIzU8l35avy0AqpRRuFgBO9wDSKiCllHKzAOAYAxAZolVASinlXgEgKxEvDy/aB7Zv6KQopVSDc68AkJ1IRFAEXh4udX5SSqlmzb0CQFaijgBWSikHtwkAxhgdA6CUUk7cJgCkFaSRb8vXMQBKKeXgNgEgISsBQMcAKKWUg9sEAB0DoJRSZblNAEjKTsJLvGgfpF1AlVIK3CgAJGQl0DG4o3YBVUopB7cJADoJnFJKleUWAcAYQ0JWgtb/K6WUE7cIAGkFaeTZ8nQQmFJKOXGLAJCUnQRoDyCllHLmFgFAxwAopdTZ3CIAJGYl4ime2gVUKaWcuEUASMpOokNQB7w9vBs6KUop1Wi4Raf4Hq170CGoQ0MnQymlGhWXSgAiMk5E9otInIg8UsHnUSKySkR2isgPIhLh9NlzIrJHRPaJyMsiIo7tN4nILscx34hIWO39rLLu7nM3f7joD3V1eqWUapKqDAAi4gn8CxgPxAA3iUhMud1eAN41xsQCc4F5jmOHAsOAWOBCYBAwUkS8gH8Cox3H7ATur5VfpJRSyiWulAAGA3HGmHhjTBHwETCp3D4xwCrH6++dPjeAH+AD+ALeQAogjkego0QQAhytwe9QSilVTa4EgI5AktP7ZMc2ZzuA6x2vrwWCRSTUGLMBKyAcczyWG2P2GWOKgXuBXVgZfwywsKIvF5GZIrJZRDanpqa6+LOUUkpVxZUAIBVsM+XeP4RVtbMNGAkcAWwi0g3oBURgBY0xIjJCRLyxAkB/oANWFdCjFX25MeYNY8xAY8zA8PBwV36TUkopF7jSCygZcJ5FLYJy1TXGmKPAdQAiEgRcb4zJFJGZwE/GmBzHZ18DFwP5juMOOrZ/ApzVuKyUUqruuFIC2AR0F5HOIuIDTAWWOO8gImEiUnquR4FFjteJOBp9HXf9I4F9WCWEGBEpvaUf69iulFKqnlRZAjDG2ETkfmA54AksMsbsEZG5wGZjzBJgFDBPRAywBvid4/DFwBisun4DfGOMWQogIn8F1ohIMZAA3FGbP0wppVTlxJjy1fmN18CBA83mzZsbOhlKKdWkiMgWY8zAs7Y3pQAgIqlYpQVXhQEn6yg5TZFej7L0epxNr0lZzeV6RBljzupF06QCQHWJyOaKop670utRll6Ps+k1Kau5Xw+3mAxOKaXU2TQAKKWUm2ruAeCNhk5AI6PXoyy9HmfTa1JWs74ezboNQCml1Lk19xKAUkqpc9AAoJRSbqrJBwAXFqvxFZGPHZ//LCLR9Z/K+uXCNXlQRPY6FuNZJSJRDZHO+lLV9XDa7wYRMSLSbLv9gWvXQ0QmO/5G9ojIB/Wdxvrmwr+ZTiLyvYhsc/y7uaoh0lnrjDFN9oE1FGeAVwAAApZJREFUNcVBoAvWmgM7gJhy+9wHvO54PRX4uKHT3QiuyWggwPH63uZ8TVy5Ho79grGmMfkJGNjQ6W7gv4/uwDagleN9m4ZOdyO4Jm8A9zpexwCHGzrdtfFo6iUAVxarmQS843i9GLisdFnKZqrKa2KM+d4Yk+d4+xPWDK/NlSt/IwBPAs8BBfWZuAbgyvWYAfzLGJMOYIw5Uc9prG+uXBODtXAVQAuayQJWTT0AuLJYzel9jDE2IBMIrZfUNQxXromzu4Cv6zRFDavK6yEi/YFIY8z/6jNhDcSVv48LgAtEZL2I/CQi4+otdQ3DlWvyBDBNRJKBZcDv6ydpdcuV9QAaM1cWq3Fln+bE5d8rItOAgVjTdDdXlV4PxzTmL+E+s9G68vfhhVUNNAqrdLhWRC40xmTUcdoaiivX5CbgbWPMiyJyCfCe45rY6z55daeplwCqXKzGeR/HYvQtgFP1krqG4co1QUQuB+YAE40xhfWUtoZQ1fUIBi4EfhCRw1gLFi1pxg3Brv6b+dIYU2yMOQTsxwoIzZUr1+Qu4BMAYy1164c1UVyT1tQDQJWL1Tje3+54fQPwnXG05DRTrizg0x+Yj5X5N/f63UqvhzEm0xgTZoyJNsZEY7WJTDTGNNd5x135N/MFVkcBRCQMq0oovl5TWb9cuSaJwGUAItILKwA0+UXKm3QAcNTply5Wsw/4xDgWqxGRiY7dFgKhIhIHPEgzX3rSxWvyPBAEfCoi20Wk/B97s+Hi9XAbLl6P5UCaiOwFvgdmGWPSGibFdc/Fa/JHYIaI7AA+BO5oDjeSOhWEUkq5qSZdAlBKKXX+NAAopZSb0gCglFJu6v+3VwcCAAAAAIL8rQe5JBIAwJQAAKYEADAlAICpAOmVEY1A+ZKDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(lr, results)\n",
    "plt.plot(lr, results_2)\n",
    "plt.plot(lr, results_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#nn_10fold([64, 32, 2], 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can try different values of $\\lambda$ and see how it changes the accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_4 = []\n",
    "lb = [0.01, 0.03, 0.05, 0.7, 0.1, 0.15, 0.2, 0.25, 0.4, 0.6]\n",
    "for lbd in lb:\n",
    "    results_4.append(nn_10fold([64, 22, 2], 100,  0.2, 8, lbd))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9934849637945108,\n",
       " 0.9924827360012545,\n",
       " 0.9933174605421572,\n",
       " 0.9938188536108727,\n",
       " 0.994485795727331,\n",
       " 0.9931505156339757,\n",
       " 0.9934846846224229,\n",
       " 0.9938180160953829,\n",
       " 0.9944866332443703,\n",
       " 0.9934844054511105]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the learning is not consistently changed by the regularization parameter, keeping it to 0.5 seems to be a good idea."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decion tree training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's take a look at decision tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, as for this algorithm we will need to use separated data and labels not as in neural network, we need another function to shuffle de data and lables by still keeping them paired."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shuffles data and lables the same way.\n",
    "def unison_shuffled_copies(a, b):\n",
    "    assert len(a) == len(b)\n",
    "    p = np.random.permutation(len(a))\n",
    "    return a[p], b[p]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple tree, to see how it presents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "`- Split(feature=41, value=-0.1807, score=0.8765), Classes: {1: 3000, 0: 2998}\n",
      "   |- Split(feature=60, value=-0.7723, score=0.9391), Classes: {1: 2876, 0: 276}\n",
      "   |  |- Split(feature=32, value=-0.8111, score=0.9729), Classes: {0: 196, 1: 20}\n",
      "   |  |  |- Split(feature=50, value=-0.9239, score=1.0), Classes: {1: 18, 0: 1}\n",
      "   |  |  |  |- 0, Classes: {0: 1}\n",
      "   |  |  |  `- 1, Classes: {1: 18}\n",
      "   |  |  `- Split(feature=55, value=-0.414, score=1.0), Classes: {0: 195, 1: 2}\n",
      "   |  |     |- 1, Classes: {1: 2}\n",
      "   |  |     `- 0, Classes: {0: 195}\n",
      "   |  `- Split(feature=33, value=-0.1661, score=0.959), Classes: {1: 2856, 0: 80}\n",
      "   |     |- Split(feature=62, value=-0.6359, score=0.8632), Classes: {1: 22, 0: 32}\n",
      "   |     |  |- Split(feature=2, value=1.5285, score=0.9394), Classes: {1: 20, 0: 2}\n",
      "   |     |  |  |- 1, Classes: {1: 19}\n",
      "   |     |  |  `- 0, Classes: {0: 2, 1: 1}\n",
      "   |     |  `- Split(feature=61, value=0.8879, score=0.9583), Classes: {1: 2, 0: 30}\n",
      "   |     |     |- 0, Classes: {0: 29}\n",
      "   |     |     `- 1, Classes: {1: 2, 0: 1}\n",
      "   |     `- Split(feature=60, value=-0.2299, score=0.9713), Classes: {1: 2834, 0: 48}\n",
      "   |        |- Split(feature=46, value=-0.7649, score=0.8627), Classes: {0: 23, 1: 60}\n",
      "   |        |  |- 1, Classes: {0: 1, 1: 54}\n",
      "   |        |  `- 0, Classes: {0: 22, 1: 6}\n",
      "   |        `- Split(feature=5, value=-1.1497, score=0.9834), Classes: {1: 2774, 0: 25}\n",
      "   |           |- 1, Classes: {1: 6, 0: 4}\n",
      "   |           `- 1, Classes: {1: 2768, 0: 21}\n",
      "   `- Split(feature=42, value=-0.582, score=0.9419), Classes: {0: 2722, 1: 124}\n",
      "      |- Split(feature=62, value=0.8013, score=0.8143), Classes: {1: 67, 0: 45}\n",
      "      |  |- Split(feature=22, value=1.5268, score=0.8571), Classes: {1: 66, 0: 11}\n",
      "      |  |  |- Split(feature=2, value=0.8232, score=0.9203), Classes: {1: 66, 0: 6}\n",
      "      |  |  |  |- 1, Classes: {1: 66, 0: 3}\n",
      "      |  |  |  `- 0, Classes: {0: 3}\n",
      "      |  |  `- 0, Classes: {0: 5}\n",
      "      |  `- Split(feature=63, value=1.0355, score=1.0), Classes: {0: 34, 1: 1}\n",
      "      |     |- 0, Classes: {0: 34}\n",
      "      |     `- 1, Classes: {1: 1}\n",
      "      `- Split(feature=46, value=-0.9913, score=0.9675), Classes: {0: 2677, 1: 57}\n",
      "         |- Split(feature=5, value=-0.2767, score=0.8118), Classes: {0: 9, 1: 18}\n",
      "         |  |- Split(feature=30, value=-0.5621, score=1.0), Classes: {0: 8, 1: 2}\n",
      "         |  |  |- 0, Classes: {0: 8}\n",
      "         |  |  `- 1, Classes: {1: 2}\n",
      "         |  `- Split(feature=50, value=0.2397, score=1.0), Classes: {1: 16, 0: 1}\n",
      "         |     |- 0, Classes: {0: 1}\n",
      "         |     `- 1, Classes: {1: 16}\n",
      "         `- Split(feature=47, value=0.9029, score=0.9755), Classes: {0: 2668, 1: 39}\n",
      "            |- Split(feature=2, value=4.1248, score=0.9847), Classes: {0: 2647, 1: 25}\n",
      "            |  |- 0, Classes: {0: 2643, 1: 18}\n",
      "            |  `- 1, Classes: {1: 7, 0: 4}\n",
      "            `- Split(feature=25, value=0.3929, score=0.76), Classes: {1: 14, 0: 21}\n",
      "               |- 1, Classes: {1: 12, 0: 3}\n",
      "               `- 0, Classes: {0: 18, 1: 2}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\n",
       "Decision tree trained with 5998 samples, 2 classes"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Single model training for visualisation of the architecture.\n",
    "new_scaled, new_labels = unison_shuffled_copies(scaled_transfrom, labels)\n",
    "dt = tr.DecisionTree(new_scaled, new_labels, tr.CostFunction.gini, 5)\n",
    "dt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we define a cross-validation function for the trees classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10-fold cross-validation for decision trees.\n",
    "def dt_10fold(depth):\n",
    "    # Shuffles dataset.\n",
    "    new_scaled, new_labels = unison_shuffled_copies(scaled_transfrom, labels)\n",
    "    n = len(new_scaled) / 10  # Size of fold.\n",
    "    tot_result = 0\n",
    "    # Trains and test on each fold.\n",
    "    for i in range(10):\n",
    "        # Training data and labels.\n",
    "        x_train_2 = np.concatenate((new_scaled[0:int(i * n)], new_scaled[int((i+1) * n):]), axis=0)\n",
    "        x_labs_train = np.concatenate((new_labels[0:int(i * n)], new_labels[int((i+1) * n):]), axis=0)\n",
    "        # Test data and labels.\n",
    "        x_test_2 = new_scaled[int(i * n):int((i+1) * n)]\n",
    "        x_labs_test = new_labels[int(i * n):int((i+1) * n)]\n",
    "        # Model generation.\n",
    "        dt = tr.DecisionTree(x_train_2, x_labs_train, tr.CostFunction.gini, depth)\n",
    "        result = 0  # Fold result init.\n",
    "        for elt, lab in zip(x_test_2, x_labs_test):\n",
    "            result += dt.predict(elt) == lab\n",
    "            result /= len(x_test_2)\n",
    "            tot_result += result  # Adds fold result to total result.\n",
    "    # Averages the results.\n",
    "    final_res = tot_result / 10\n",
    "    return (final_res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then try to generate trees with different depths to see how it affects the learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9624206485966618\n",
      "0.9764493251495863\n",
      "0.9791187681849822\n",
      "0.9801243460425197\n",
      "0.9811257363195101\n"
     ]
    }
   ],
   "source": [
    "tree_res = []\n",
    "dep = [2, 3, 4, 5, 6]\n",
    "for d in dep:\n",
    "    tree_res.append(dt_10fold(d))\n",
    "    print(tree_res[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data segmentation, 75% training, 25% test.\n",
    "new_scaled, new_labels = unison_shuffled_copies(scaled_transfrom, labels)\n",
    "training_data, test_data = new_scaled[0:4500], new_scaled[4500:]\n",
    "training_labels, test_labels = new_labels[0:4500], new_labels[4500:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random forest\n",
    "predictions = [Counter({0: 0, 1: 0}) for elt in test_data]\n",
    "for i in range(300):\n",
    "    weak_tree = tr.DecisionTree(training_data, training_labels, tr.CostFunction.gini, 5, True)\n",
    "    for j, elt in enumerate(test_data):\n",
    "        y_hat = weak_tree.predict(elt)\n",
    "        predictions[j][y_hat] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the most common prediciton for each point of the test set.\n",
    "final_pred = np.array([elt.most_common(1)[0][0] for elt in predictions])\n",
    "# Compares to true values and get the proportion of correctly predicted.\n",
    "accuracy = sum([true == expected for true, expected in zip(test_labels, final_pred)]) / len(final_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9766355140186916"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy of random forest is still below the neural network's one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}