TitanicKaggleCompetition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Getting started: Kaggle's Titanic Competition\n",
    "\n",
    "Kaggle is already established as the best place which hosts machine learning competitions. If you do not know it already, then it's time to do it.\n",
    "\n",
    "__[Titanic Competition](https://www.kaggle.com/c/titanic)__ is perhaps the first competition which one should try. Of course, if you are already an experienced data scientist, than you can skip to an advanced competition.\n",
    "\n",
    "The purpose of the competition is to learn if a passenger has survived or not. We illustrate some steps and ideas one can apply to compete in this learning competition using the available tools one can find in rapaio library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[32mAdd /home/ati/work/rapaio-notebooks/././rapaio-core-6.0.0.jar to classpath\u001b[0m"
     ]
    }
   ],
   "source": [
    "%load ./rapaio-bootstrap.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Get the data\n",
    "\n",
    "The purpose of the competition is to predict which passengers have survived or not. The available data has two parts. The first part consists in a data set which contains what happened with some passengers and some related information like sex, cabin, age, class, etc. This data set contains information regarding their survival. The purpose why this data set contains survival data is because it will be used to train a model which learns how to decide if a passenger survives or not. This is the `train.csv`. The other file is a data set which contains data about another set of passenger, this time without knowing if they survived or not. They contain, however an identification number. This data set is `test.csv` and this is used to make predictions. Those predictions should be similar with the provided `gendermodel.csv`.\n",
    "\n",
    "We also have to take a look of the data description provided on __[contest dedicated page](https://www.kaggle.com/c/titanic/data)__:\n",
    "\n",
    "```\n",
    "VARIABLE DESCRIPTIONS:\n",
    "survival Survival\n",
    "(0 = No; 1 = Yes)\n",
    "pclass Passenger Class\n",
    "(1 = 1st; 2 = 2nd; 3 = 3rd)\n",
    "name Name\n",
    "sex Sex\n",
    "age Age\n",
    "sibsp Number of Siblings/Spouses Aboard\n",
    "parch Number of Parents/Children Aboard\n",
    "ticket Ticket Number\n",
    "fare Passenger Fare\n",
    "cabin Cabin\n",
    "embarked Port of Embarkation\n",
    "(C = Cherbourg; Q = Queenstown; S = Southampton)\n",
    "\n",
    "SPECIAL NOTES:\n",
    "Pclass is a proxy for socio-economic status (SES)\n",
    "1st ~ Upper; 2nd ~ Middle; 3rd ~ Lower\n",
    "\n",
    "Age is in Years; Fractional if Age less than One (1)\n",
    "If the Age is Estimated, it is in the form xx.5\n",
    "\n",
    "With respect to the family relation variables (i.e. sibsp and parch)\n",
    "some relations were ignored. The following are the definitions used\n",
    "for sibsp and parch.\n",
    "\n",
    "Sibling: Brother, Sister, Stepbrother, or Stepsister of Passenger Aboard Titanic\n",
    "Spouse: Husband or Wife of Passenger Aboard Titanic (Mistresses and Fiances Ignored)\n",
    "Parent: Mother or Father of Passenger Aboard Titanic\n",
    "Child: Son, Daughter, Stepson, or Stepdaughter of Passenger Aboard Titanic\n",
    "\n",
    "Other family relatives excluded from this study include cousins,\n",
    "nephews/nieces, aunts/uncles, and in-laws. Some children travelled\n",
    "only with a nanny, therefore parch=0 for them. As well, some\n",
    "travelled with very close friends or neighbors in a village, however,\n",
    "the definitions do not support such relations.\n",
    "```\n",
    "\n",
    "The first step in our adventure is to download those 3 data file in *csv* format. You can do it from __[data section](https://www.kaggle.com/c/titanic/data)__ of the competition. Let's suppose you downloaded somewhere in a local folder. We will name this folder `data` folder, and actually it can have any name you would like.\n",
    "\n",
    "### 1.2 Read train data from csv file\n",
    "\n",
    "Because the data is small we can load the whole data in memory with no problems.\n",
    "\n",
    "Let's see how we can load the data into memory. In rapaio the sets of data are loaded into the form of *frames* (`rapaio.data.Frame`). A frame is basically a tabular data, with columns for each variable (feature) and rows for each instance (in our case for each passenger).\n",
    "\n",
    "\n",
    "A first try of loading the train data set and see what has happened is the following:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 183/891\n",
      "* varCount: 12\n",
      "* varNames: \n",
      "\n",
      " 0. PassengerId : int |  4.         Sex : nom |  8.      Ticket : nom | \n",
      " 1.    Survived : bin |  5.         Age : dbl |  9.        Fare : dbl | \n",
      " 2.      Pclass : int |  6.       SibSp : int | 10.       Cabin : nom | \n",
      " 3.        Name : nom |  7.       Parch : int | 11.    Embarked : nom | \n",
      "\n",
      "* summary: \n",
      " PassengerId [int]        Survived [bin]    Pclass [int]     \n",
      "      Min. :   1.0000000 0 :         549    Min. : 1.0000000 \n",
      "   1st Qu. : 223.5000000 1 :         342 1st Qu. : 2.0000000 \n",
      "    Median : 446.0000000 NAs :         0  Median : 3.0000000 \n",
      "      Mean : 446.0000000                    Mean : 2.3086420 \n",
      "   2nd Qu. : 668.5000000                 2nd Qu. : 3.0000000 \n",
      "      Max. : 891.0000000                    Max. : 3.0000000 \n",
      "                                                             \n",
      "\n",
      "                                                   Name [nom]      Sex [nom]       Age \n",
      "\"Jacobsohn, Mrs. Sidney Samuel (Amy Frances Christy)\" :     1   male :   577    Min. : \n",
      "                                \"Toomey, Miss. Ellen\" :     1 female :   314 1st Qu. : \n",
      "                          \"de Pelsmaeker, Mr. Alfons\" :     1                 Median : \n",
      "                          \"Jensen, Mr. Svend Lauritz\" :     1                   Mean : \n",
      "                                \"Morley, Mr. William\" :     1                2nd Qu. : \n",
      "                                              (Other) :   886                   Max. : \n",
      "                                                                                 NAs : \n",
      "\n",
      "[dbl]           SibSp [int]         Parch [int]         Ticket [nom]      Fare [dbl]       \n",
      "  0.4200000    Min. : 0.0000000    Min. : 0.0000000 CA. 2343 :     7    Min. :   0.0000000 \n",
      " 20.1250000 1st Qu. : 0.0000000 1st Qu. : 0.0000000   347082 :     7 1st Qu. :   7.9104000 \n",
      " 28.0000000  Median : 0.0000000  Median : 0.0000000     1601 :     7  Median :  14.4542000 \n",
      " 29.6991176    Mean : 0.5230079    Mean : 0.3815937   347088 :     6    Mean :  32.2042080 \n",
      " 38.0000000 2nd Qu. : 1.0000000 2nd Qu. : 0.0000000  3101295 :     6 2nd Qu. :  31.0000000 \n",
      " 80.0000000    Max. : 8.0000000    Max. : 6.0000000  (Other) :   858    Max. : 512.3292000 \n",
      "177.0000000                                                                                \n",
      "\n",
      "        Cabin [nom]  Embarked [nom] \n",
      "C23 C25 C27 :     4       S :   644 \n",
      "    B96 B98 :     4       C :   168 \n",
      "         G6 :     4       Q :    77 \n",
      "         F2 :     3                 \n",
      "    (Other) :   189                 \n",
      "        NAs :   687     NAs :     2 \n",
      "                                    \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "String urlTrain = \"./data/titanic/train.csv\";\n",
    "String urlTest = \"./data/titanic/test.csv\";\n",
    "Csv.instance().read(urlTrain).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we interpret the output of the frame's summary?\n",
    "\n",
    "* We loaded a frame which has $891$ rows and $12$ columns (variables)\n",
    "* From all the rows, $183$ are complete (non missing data)\n",
    "* The name of the variables are listed, together with their types\n",
    "* It follows a data summary for the frame: 6 number summary for numeric variables, most frequent levels for nominal variables\n",
    "\n",
    "Let's inspect each variable and see how it fits our needs.\n",
    "\n",
    "**PassengedId**\n",
    "\n",
    "The type for this variable is index (integer values). This field looks like an identifier for the passenger, so from our point of view the sorting is not required. What we can do, but is not required, is to change the field type to nominal. Anyway, we do not need this field for learning since it should be unique for each instance, thus the predictive power is null. We will ignore it for now since we will not consider it for learning\n",
    "\n",
    "**Survived**\n",
    "\n",
    "This is our target variable. It is parsed as binary, but since we do classification, we will change it's type to nominal. We do that directly from the csv parsing, by indicating that we want Survived parsed as nominal variable:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 183/891\n",
      "* varCount: 12\n",
      "* varNames: \n",
      "\n",
      " 0. PassengerId : int |  4.         Sex : nom |  8.      Ticket : nom | \n",
      " 1.    Survived : nom |  5.         Age : dbl |  9.        Fare : dbl | \n",
      " 2.      Pclass : int |  6.       SibSp : int | 10.       Cabin : nom | \n",
      " 3.        Name : nom |  7.       Parch : int | 11.    Embarked : nom | \n",
      "\n",
      "* summary: \n",
      " PassengerId [int]        Survived [nom]    Pclass [int]     \n",
      "      Min. :   1.0000000       0 :   549    Min. : 1.0000000 \n",
      "   1st Qu. : 223.5000000       1 :   342 1st Qu. : 2.0000000 \n",
      "    Median : 446.0000000                  Median : 3.0000000 \n",
      "      Mean : 446.0000000                    Mean : 2.3086420 \n",
      "   2nd Qu. : 668.5000000                 2nd Qu. : 3.0000000 \n",
      "      Max. : 891.0000000                    Max. : 3.0000000 \n",
      "                                                             \n",
      "\n",
      "                                                   Name [nom]      Sex [nom]       Age \n",
      "\"Jacobsohn, Mrs. Sidney Samuel (Amy Frances Christy)\" :     1   male :   577    Min. : \n",
      "                                \"Toomey, Miss. Ellen\" :     1 female :   314 1st Qu. : \n",
      "                          \"de Pelsmaeker, Mr. Alfons\" :     1                 Median : \n",
      "                          \"Jensen, Mr. Svend Lauritz\" :     1                   Mean : \n",
      "                                \"Morley, Mr. William\" :     1                2nd Qu. : \n",
      "                                              (Other) :   886                   Max. : \n",
      "                                                                                 NAs : \n",
      "\n",
      "[dbl]           SibSp [int]         Parch [int]         Ticket [nom]      Fare [dbl]       \n",
      "  0.4200000    Min. : 0.0000000    Min. : 0.0000000 CA. 2343 :     7    Min. :   0.0000000 \n",
      " 20.1250000 1st Qu. : 0.0000000 1st Qu. : 0.0000000   347082 :     7 1st Qu. :   7.9104000 \n",
      " 28.0000000  Median : 0.0000000  Median : 0.0000000     1601 :     7  Median :  14.4542000 \n",
      " 29.6991176    Mean : 0.5230079    Mean : 0.3815937   347088 :     6    Mean :  32.2042080 \n",
      " 38.0000000 2nd Qu. : 1.0000000 2nd Qu. : 0.0000000  3101295 :     6 2nd Qu. :  31.0000000 \n",
      " 80.0000000    Max. : 8.0000000    Max. : 6.0000000  (Other) :   858    Max. : 512.3292000 \n",
      "177.0000000                                                                                \n",
      "\n",
      "        Cabin [nom]  Embarked [nom] \n",
      "C23 C25 C27 :     4       S :   644 \n",
      "    B96 B98 :     4       C :   168 \n",
      "         G6 :     4       Q :    77 \n",
      "         F2 :     3                 \n",
      "    (Other) :   189                 \n",
      "        NAs :   687     NAs :     2 \n",
      "                                    \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Csv.instance()\n",
    ".types.add(VarType.NOMINAL, \"Survived\")\n",
    ".read(urlTrain)\n",
    ".printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And notice how type of the `Survived` variable changed to nominal.\n",
    "\n",
    "\n",
    "**Pclass**\n",
    "\n",
    "This variable has index type. We can keep it like it is or we can change it to nominal. Both ways can be useful. For example parsed as index could give an interpretation to the order. We can say that somehow, because of ordering class 1 is lower than class 2, and class 2 is between classes 1 and 3. At the same time we can keep it as nominal if we do not want to use the ordering. Let's choose nominal for now, considering that 1,2 and 3 are just labels for type of tickets, with no other meaning attached. We proceed in the same way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 183/891\n",
      "* varCount: 12\n",
      "* varNames: \n",
      "\n",
      " 0. PassengerId : int |  4.         Sex : nom |  8.      Ticket : nom | \n",
      " 1.    Survived : nom |  5.         Age : dbl |  9.        Fare : dbl | \n",
      " 2.      Pclass : nom |  6.       SibSp : int | 10.       Cabin : nom | \n",
      " 3.        Name : nom |  7.       Parch : int | 11.    Embarked : nom | \n",
      "\n",
      "* summary: \n",
      " PassengerId [int]        Survived [nom]  Pclass [nom] \n",
      "      Min. :   1.0000000       0 :   549     3 :   491 \n",
      "   1st Qu. : 223.5000000       1 :   342     1 :   216 \n",
      "    Median : 446.0000000                     2 :   184 \n",
      "      Mean : 446.0000000                               \n",
      "   2nd Qu. : 668.5000000                               \n",
      "      Max. : 891.0000000                               \n",
      "                                                       \n",
      "\n",
      "                                                   Name [nom]      Sex [nom]       Age \n",
      "\"Jacobsohn, Mrs. Sidney Samuel (Amy Frances Christy)\" :     1   male :   577    Min. : \n",
      "                                \"Toomey, Miss. Ellen\" :     1 female :   314 1st Qu. : \n",
      "                          \"de Pelsmaeker, Mr. Alfons\" :     1                 Median : \n",
      "                          \"Jensen, Mr. Svend Lauritz\" :     1                   Mean : \n",
      "                                \"Morley, Mr. William\" :     1                2nd Qu. : \n",
      "                                              (Other) :   886                   Max. : \n",
      "                                                                                 NAs : \n",
      "\n",
      "[dbl]           SibSp [int]         Parch [int]         Ticket [nom]      Fare [dbl]       \n",
      "  0.4200000    Min. : 0.0000000    Min. : 0.0000000 CA. 2343 :     7    Min. :   0.0000000 \n",
      " 20.1250000 1st Qu. : 0.0000000 1st Qu. : 0.0000000   347082 :     7 1st Qu. :   7.9104000 \n",
      " 28.0000000  Median : 0.0000000  Median : 0.0000000     1601 :     7  Median :  14.4542000 \n",
      " 29.6991176    Mean : 0.5230079    Mean : 0.3815937   347088 :     6    Mean :  32.2042080 \n",
      " 38.0000000 2nd Qu. : 1.0000000 2nd Qu. : 0.0000000  3101295 :     6 2nd Qu. :  31.0000000 \n",
      " 80.0000000    Max. : 8.0000000    Max. : 6.0000000  (Other) :   858    Max. : 512.3292000 \n",
      "177.0000000                                                                                \n",
      "\n",
      "        Cabin [nom]  Embarked [nom] \n",
      "C23 C25 C27 :     4       S :   644 \n",
      "    B96 B98 :     4       C :   168 \n",
      "         G6 :     4       Q :    77 \n",
      "         F2 :     3                 \n",
      "    (Other) :   189                 \n",
      "        NAs :   687     NAs :     2 \n",
      "                                    \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Csv.instance()\n",
    ".types.add(VarType.NOMINAL, \"Survived\", \"Pclass\")\n",
    ".read(urlTrain)\n",
    ".printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we append the variable name after `Survived`. This is possible since the `withTypes` method specify a type, and follows a dynamic array of strings, for the names of variables.\n",
    "\n",
    "**Name**\n",
    "\n",
    "This is the passenger names and the values are unique. As it is, the predictive power of this field is null. We keep it as it is. Note that it contains valuable information, but not in this direct form.\n",
    "\n",
    "**Sex**\n",
    "\n",
    "This field specifies the gender of the passenger. We have $577$ males and $314$ females.\n",
    "\n",
    "**Age**\n",
    "\n",
    "This field specifies the age of an passenger. We would expect that to parse this variable as numeric or at leas index, but is nominal. Why that happened? Notice that the values looks like numbers. But the first value (the most frequent one, $117$ instances) has nothing specified. Well, the variable is nominal has to do with how *Csv* parsing handles missing values. By default, the *csv* parsing considers as missing values only the string \"?\". But the most frequent value in this field is empty string \"\". This means that empty string is not considered a missing value. Because empty string can't produce numbers from parsing, the variable is *promoted* to nominal.\n",
    "\n",
    "We can customize the missing value handling by specifying the valid strings for that purpose. We use `.useNAValues(String...naValues)` to tell the parser all the valid strings which are missing values. In our case we want just the empty string to be a missing value. When the parser will found an empty string it will set the variable value as missing value. It will *not promote* variable to nominal, since a missing value is a legal value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 183/891\n",
      "* varCount: 12\n",
      "* varNames: \n",
      "\n",
      " 0. PassengerId : int |  4.         Sex : nom |  8.      Ticket : nom | \n",
      " 1.    Survived : nom |  5.         Age : dbl |  9.        Fare : dbl | \n",
      " 2.      Pclass : nom |  6.       SibSp : int | 10.       Cabin : nom | \n",
      " 3.        Name : nom |  7.       Parch : int | 11.    Embarked : nom | \n",
      "\n",
      "* summary: \n",
      " PassengerId [int]        Survived [nom]  Pclass [nom] \n",
      "      Min. :   1.0000000       0 :   549     3 :   491 \n",
      "   1st Qu. : 223.5000000       1 :   342     1 :   216 \n",
      "    Median : 446.0000000                     2 :   184 \n",
      "      Mean : 446.0000000                               \n",
      "   2nd Qu. : 668.5000000                               \n",
      "      Max. : 891.0000000                               \n",
      "                                                       \n",
      "\n",
      "                                                   Name [nom]      Sex [nom]       Age \n",
      "\"Jacobsohn, Mrs. Sidney Samuel (Amy Frances Christy)\" :     1   male :   577    Min. : \n",
      "                                \"Toomey, Miss. Ellen\" :     1 female :   314 1st Qu. : \n",
      "                          \"de Pelsmaeker, Mr. Alfons\" :     1                 Median : \n",
      "                          \"Jensen, Mr. Svend Lauritz\" :     1                   Mean : \n",
      "                                \"Morley, Mr. William\" :     1                2nd Qu. : \n",
      "                                              (Other) :   886                   Max. : \n",
      "                                                                                 NAs : \n",
      "\n",
      "[dbl]           SibSp [int]         Parch [int]         Ticket [nom]      Fare [dbl]       \n",
      "  0.4200000    Min. : 0.0000000    Min. : 0.0000000 CA. 2343 :     7    Min. :   0.0000000 \n",
      " 20.1250000 1st Qu. : 0.0000000 1st Qu. : 0.0000000   347082 :     7 1st Qu. :   7.9104000 \n",
      " 28.0000000  Median : 0.0000000  Median : 0.0000000     1601 :     7  Median :  14.4542000 \n",
      " 29.6991176    Mean : 0.5230079    Mean : 0.3815937   347088 :     6    Mean :  32.2042080 \n",
      " 38.0000000 2nd Qu. : 1.0000000 2nd Qu. : 0.0000000  3101295 :     6 2nd Qu. :  31.0000000 \n",
      " 80.0000000    Max. : 8.0000000    Max. : 6.0000000  (Other) :   858    Max. : 512.3292000 \n",
      "177.0000000                                                                                \n",
      "\n",
      "        Cabin [nom]  Embarked [nom] \n",
      "C23 C25 C27 :     4       S :   644 \n",
      "    B96 B98 :     4       C :   168 \n",
      "         G6 :     4       Q :    77 \n",
      "         F2 :     3                 \n",
      "    (Other) :   189                 \n",
      "        NAs :   687     NAs :     2 \n",
      "                                    \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Csv.instance()\n",
    ".naValues.add(\"\")\n",
    ".types.add(VarType.NOMINAL, \"Survived\", \"Pclass\")\n",
    ".read(urlTrain)\n",
    ".printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice what happened: *Age* field is now numeric and it contains $177$ missing values.\n",
    "\n",
    "**SibSp**\n",
    "\n",
    "It's meaning is \"siblings/spouses\". It's parsed as index, which is natural. In pathological cases with sick imagination we can consider a \"quarter of a wife\" for example.\n",
    "\n",
    "**Parch**\n",
    "\n",
    "It's meaning is \"parents/children\". It is naturally parsed as index.\n",
    "\n",
    "**Ticket**\n",
    "\n",
    "This is the code of the ticket. Probably a family can have the same ticket, thus must be the reason why the frequencies have values up to $$7$$. This field is nominal. It has low predictive power used directly. Perhaps contains valuable information, but used directly in row format would not help much.\n",
    "\n",
    "**Fare**\n",
    "\n",
    "This is the price for passenger fare and should be numeric, like it is.\n",
    "\n",
    "**Cabin**\n",
    "\n",
    "Code of the passenger's cabin, parsed as nominal. Same notes as for `Ticket` variable.\n",
    "\n",
    "**Embarked**\n",
    "\n",
    "Code for the embarking city, which could be: C = Cherbourg, Q = Queenstown, S = Southampton. It's parsed as nominal and has $2$ missing values.\n",
    "\n",
    "If we are content with our parsing, we load data into a data frame for later use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 183/891\n",
      "* varCount: 12\n",
      "* varNames: \n",
      "\n",
      " 0. PassengerId : int |  4.         Sex : nom |  8.      Ticket : nom | \n",
      " 1.    Survived : nom |  5.         Age : dbl |  9.        Fare : dbl | \n",
      " 2.      Pclass : nom |  6.       SibSp : int | 10.       Cabin : nom | \n",
      " 3.        Name : nom |  7.       Parch : int | 11.    Embarked : nom | \n",
      "\n",
      "* summary: \n",
      " PassengerId [int]        Survived [nom]  Pclass [nom] \n",
      "      Min. :   1.0000000       0 :   549     3 :   491 \n",
      "   1st Qu. : 223.5000000       1 :   342     1 :   216 \n",
      "    Median : 446.0000000                     2 :   184 \n",
      "      Mean : 446.0000000                               \n",
      "   2nd Qu. : 668.5000000                               \n",
      "      Max. : 891.0000000                               \n",
      "                                                       \n",
      "\n",
      "                                                   Name [nom]      Sex [nom]       Age \n",
      "\"Jacobsohn, Mrs. Sidney Samuel (Amy Frances Christy)\" :     1   male :   577    Min. : \n",
      "                                \"Toomey, Miss. Ellen\" :     1 female :   314 1st Qu. : \n",
      "                          \"de Pelsmaeker, Mr. Alfons\" :     1                 Median : \n",
      "                          \"Jensen, Mr. Svend Lauritz\" :     1                   Mean : \n",
      "                                \"Morley, Mr. William\" :     1                2nd Qu. : \n",
      "                                              (Other) :   886                   Max. : \n",
      "                                                                                 NAs : \n",
      "\n",
      "[dbl]           SibSp [int]         Parch [int]         Ticket [nom]      Fare [dbl]       \n",
      "  0.4200000    Min. : 0.0000000    Min. : 0.0000000 CA. 2343 :     7    Min. :   0.0000000 \n",
      " 20.1250000 1st Qu. : 0.0000000 1st Qu. : 0.0000000   347082 :     7 1st Qu. :   7.9104000 \n",
      " 28.0000000  Median : 0.0000000  Median : 0.0000000     1601 :     7  Median :  14.4542000 \n",
      " 29.6991176    Mean : 0.5230079    Mean : 0.3815937   347088 :     6    Mean :  32.2042080 \n",
      " 38.0000000 2nd Qu. : 1.0000000 2nd Qu. : 0.0000000  3101295 :     6 2nd Qu. :  31.0000000 \n",
      " 80.0000000    Max. : 8.0000000    Max. : 6.0000000  (Other) :   858    Max. : 512.3292000 \n",
      "177.0000000                                                                                \n",
      "\n",
      "        Cabin [nom]  Embarked [nom] \n",
      "C23 C25 C27 :     4       S :   644 \n",
      "    B96 B98 :     4       C :   168 \n",
      "         G6 :     4       Q :    77 \n",
      "         F2 :     3                 \n",
      "    (Other) :   189                 \n",
      "        NAs :   687     NAs :     2 \n",
      "                                    \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Frame train = Csv.instance().naValues.add(\"\").types.add(VarType.NOMINAL, \"Survived\", \"Pclass\").read(urlTrain);\n",
    "train.printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Read test data from *csv* file\n",
    "\n",
    "Once we have a training frame we can load also the test data. We do that to take a look at the frame and because data is small and there is no memory or time problem cost associated with it. To avoid adding again the *csv* options and to get identical levels nominal variables, we use a different way to parse the data set. We specify variable types by frame templates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Frame test = Csv.instance().naValues.add(\"\").template.set(train).read(urlTest);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead to specify again the preferred types for variables, we use train frame as a template for variable types. This has also the side effect that the encoding of categorical variables is identical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 418\n",
      "* complete: 87/418\n",
      "* varCount: 11\n",
      "* varNames: \n",
      "\n",
      " 0. PassengerId : int |  4.         Age : dbl |  8.        Fare : dbl | \n",
      " 1.      Pclass : nom |  5.       SibSp : int |  9.       Cabin : nom | \n",
      " 2.        Name : nom |  6.       Parch : int | 10.    Embarked : nom | \n",
      " 3.         Sex : nom |  7.      Ticket : nom | \n",
      "\n",
      "* summary: \n",
      " PassengerId [int]          Pclass [nom]                                   Name [nom]      Sex \n",
      "      Min. :   892.0000000     3 :   218                \"Birnbaum, Mr. Jakob\" :     1   male : \n",
      "   1st Qu. :   996.2500000     1 :   107 \"Willer, Mr. Aaron (Abi Weller\"\")\"\"\" :     1 female : \n",
      "    Median : 1,100.5000000     2 :    93          \"Olsson, Mr. Oscar Wilhelm\" :     1          \n",
      "      Mean : 1,100.5000000                               \"Barry, Miss. Julia\" :     1          \n",
      "   2nd Qu. : 1,204.7500000                                 \"Thomas, Mr. John\" :     1          \n",
      "      Max. : 1,309.0000000                                            (Other) :   413          \n",
      "                                                                                               \n",
      "\n",
      "[nom]       Age [dbl]          SibSp [int]         Parch [int]         Ticket [nom]      Fare \n",
      "  266    Min. :  0.1700000    Min. : 0.0000000    Min. : 0.0000000 PC 17608 :     5    Min. : \n",
      "  152 1st Qu. : 21.0000000 1st Qu. : 0.0000000 1st Qu. : 0.0000000 CA. 2343 :     4 1st Qu. : \n",
      "       Median : 27.0000000  Median : 0.0000000  Median : 0.0000000   113503 :     4  Median : \n",
      "         Mean : 30.2725904    Mean : 0.4473684    Mean : 0.3923445    16966 :     3    Mean : \n",
      "      2nd Qu. : 39.0000000 2nd Qu. : 1.0000000 2nd Qu. : 0.0000000   347077 :     3 2nd Qu. : \n",
      "         Max. : 76.0000000    Max. : 8.0000000    Max. : 9.0000000  (Other) :   399    Max. : \n",
      "          NAs : 86.0000000                                                              NAs : \n",
      "\n",
      "[dbl]                   Cabin [nom]  Embarked [nom] \n",
      "  0.0000000 B57 B59 B63 B66 :     3       S :   270 \n",
      "  7.8958000            C116 :     2       C :   102 \n",
      " 14.4542000            C101 :     2       Q :    46 \n",
      " 35.6271885              F4 :     2                 \n",
      " 31.5000000         (Other) :    82                 \n",
      "512.3292000             NAs :   327                 \n",
      "  1.0000000                                         \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test.printSummary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can note that we don't have *Survived* variable anymore. This is correct since this is what we have to predict. Note also that the types for the remaining variables are the same with training data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build simple models\n",
    "\n",
    "### 2.1 Build a majority model\n",
    "\n",
    "To make a first submission we will build a very simple model, which classifies with a single value all instances. This value is the majority label.\n",
    "\n",
    "Let's inspect at how target variable look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0   1 \n",
      "  -   - \n",
      "549 342 \n"
     ]
    }
   ],
   "source": [
    "DensityVector.fromLevelCounts(false, train.rvar(\"Survived\")).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we already new from the summary, the number of passengers who didn't survived is lower than those who did. Let's see percentages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        0         1 \n",
      "        -         - \n",
      "0.6161616 0.3838384 \n"
     ]
    }
   ],
   "source": [
    "DensityVector.fromLevelCounts(false, train.rvar(\"Survived\")).normalize().printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A similar result can be obtined using group functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group by: Survived\n",
      "group count: 2\n",
      "group by functions: GroupByFunction{name=count,varNames=[Survived]}\n",
      "\n",
      "    Survived Survived_count_N1  \n",
      "[0]        0 0.6161616161616161 \n",
      "[1]        1 0.3838383838383838 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "Group.from(train, \"Survived\").aggregate(Group.count(1, \"Survived\")).printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that there are about $61\\%$ of passengers who did not survived. We will create a submit data set, which we will save for later submission. How we can do that?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "VarNominal prediction = VarNominal.from(test.rowCount(), row -> \"0\").name(\"Survived\");\n",
    "Frame submit = SolidFrame.byVars(test.rvar(\"PassengerId\"), prediction);\n",
    "\n",
    "Csv.instance().quotes.set(false).write(submit, \"majority_submit.csv\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first line we created a new nominal variable. The size of the new variable is the number of rows from the test frame. For each row we produce the same label `\"0\"`. We name this variable `Survived`.\n",
    "\n",
    "In the second line we created a new frame taking the variable named `PassengerId` from the test data set and the new prediction variable.\n",
    "\n",
    "In the last line we wrote a new csv file with the csv parsing utility, taking care to not write quotes. We can submit this file and see which are the results.\n",
    "\n",
    "![Submission result with majority classifierModel](images/titanic-majority-submit.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Build a simple gender model\n",
    "\n",
    "It has been said that \"women and children first\" really happened during Titanic tragedy. If this was true or not, we do not know. But we can use data to see if we are hearing the same story. For now we will take the gender and see if it had an influence. We will build a contingency table for variables `Sex` and `Survived`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         0   1 total \n",
      "  male 468 109  577  \n",
      "female  81 233  314  \n",
      " total 549 342  891  \n"
     ]
    }
   ],
   "source": [
    "DensityTable.fromLabels(false, train.rvar(\"Sex\"), train.rvar(\"Survived\"), null).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On rows we have levels of `Sex` variable. On columns we have levels of `Sex` variable. Cells are computed as counts. What we see is that there are a lot of men who did not survived and a lot of women who does. We will normalize on rows to take a closer look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "               0         1 total \n",
      "  male 0.8110919 0.1889081   1   \n",
      "female 0.2579618 0.7420382   1   \n",
      " total 1.0690536 0.9309464   2   \n"
     ]
    }
   ],
   "source": [
    "DensityTable.fromLabels(false, train.rvar(\"Sex\"), train.rvar(\"Survived\"), null).normalizeOnRows().printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that men survived with a rate of $0.19$ and women with $0.74$. The values are so obvious, we need no hypothesis testing to check that this variable is significant for classification. We will build a simple model where we predict as survived all the women and not survived all the men."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "Var prediction = VarNominal.from(test.rowCount(), row -> test.getLabel(row, \"Sex\").equals(\"male\") ? \"0\" : \"1\").name(\"Survived\");\n",
    "Frame submit = SolidFrame.byVars(test.rvar(\"PassengerId\"), prediction);\n",
    "Csv.instance().write(submit, \"gender_submit.csv\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__![Submission result with gender classifierModel](images/titanic-gender-submit.png)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Tree model\n",
    "\n",
    "Building models in the manual way is often not the way to go. This process is tedious and time consuming. There are already built automated procedures, which incorporate miscellaneous approaches to learn a classifierModel. One of the often used models is the decision tree. Decision trees are greedy local approximations build in a recursive greedy fashion. Often the split decision at node level uses a single feature. At leave nodes a simple majority classifierModel creates the classification rule.\n",
    "\n",
    "### 3.1 Gender model with decision tree\n",
    "Initially we will build a CART decision tree using as input feature the *Sex* variable. We do this to exemplify how a manual rule can be created in an automated fashion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CTree model\n",
      "================\n",
      "\n",
      "Description:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "\n",
      "Capabilities:\n",
      "types inputs/targets: NOMINAL,INT,DOUBLE,BINARY/NOMINAL\n",
      "counts inputs/targets: [1,1000000] / [1,1]\n",
      "missing inputs/targets: true/false\n",
      "\n",
      "Learned model:\n",
      "\n",
      "total number of nodes: 3\n",
      "total number of leaves: 2\n",
      "description:\n",
      "split, n/err, classes (densities) [* if is leaf / purity if not]\n",
      "\n",
      "|- 1. root    891/342 0 (0.616 0.384 ) [0.139648]\n",
      "|   |- 2. Sex in ['male']    577/109 0 (0.811 0.189 ) *\n",
      "|   |- 3. Sex not in ['male']    314/81 1 (0.258 0.742 ) *\n"
     ]
    }
   ],
   "source": [
    "Frame tr = train.mapVars(\"Survived,Sex\");\n",
    "CTree tree = CTree.newCART();\n",
    "tree.fit(tr, \"Survived\");\n",
    "tree.printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a closer look at the last three rows from the output, one can identify our manual rule. Basically the interpretation is: *\"all the females survived, all the males did not\"*. For exemplification purposes we build also the submit file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.7867416 0.0477342 \n",
      "[1]   train Accuracy 0.7867563 0.0052978 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "// do a cross validation\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", tree, 10, Accuracy.newMetric(true)).run().printContent();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "// fit the tree to test data frame\n",
    "ClassifierResult pred = tree.predict(test);\n",
    "// build teh submission\n",
    "Frame submit = SolidFrame.byVars(test.rvar(\"PassengerId\"),pred.firstClasses().name(\"Survived\"));\n",
    "// write to a submit file\n",
    "Csv.instance().write(submit, \"tree1-model.csv\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Enrich tree by using other features\n",
    "\n",
    "Our training data set has more than a single input feature. Thus We can state we didn't use all the information available. We will add now the class and embarking port and see how it behaves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CTree model\n",
      "================\n",
      "\n",
      "Description:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "\n",
      "Capabilities:\n",
      "types inputs/targets: NOMINAL,INT,DOUBLE,BINARY/NOMINAL\n",
      "counts inputs/targets: [1,1000000] / [1,1]\n",
      "missing inputs/targets: true/false\n",
      "\n",
      "Learned model:\n",
      "\n",
      "total number of nodes: 31\n",
      "total number of leaves: 16\n",
      "description:\n",
      "split, n/err, classes (densities) [* if is leaf / purity if not]\n",
      "\n",
      "|- 1. root    891/342 0 (0.616 0.384 ) [0.139648]\n",
      "|   |- 2. Sex in ['male']    577/109 0 (0.811 0.189 ) [0.0173642]\n",
      "|   |   |- 4. Pclass in ['2','3']    455/64 0 (0.859 0.141 ) [0.0008311]\n",
      "|   |   |   |- 8. Embarked in ['S','C']    415/61 0 (0.853 0.147 ) [0.0018473]\n",
      "|   |   |   |   |- 16. Embarked in ['S']    362/49 0 (0.865 0.135 ) [0.0002721]\n",
      "|   |   |   |   |   |- 26. Pclass in ['3']    265/34 0 (0.872 0.128 ) *\n",
      "|   |   |   |   |   |- 27. Pclass not in ['3']    97/15 0 (0.845 0.155 ) *\n",
      "|   |   |   |   |- 17. Embarked not in ['S']    53/12 0 (0.774 0.226 ) [0.0003245]\n",
      "|   |   |   |   |   |- 28. Pclass in ['3']    43/10 0 (0.767 0.233 ) *\n",
      "|   |   |   |   |   |- 29. Pclass not in ['3']    10/2 0 (0.8 0.2 ) *\n",
      "|   |   |   |- 9. Embarked not in ['S','C']    40/3 0 (0.925 0.075 ) *\n",
      "|   |   |- 5. Pclass not in ['2','3']    122/45 0 (0.631 0.369 ) [0.0022488]\n",
      "|   |   |   |- 10. Embarked in ['S','C']    121/45 0 (0.628 0.372 ) [0.0011482]\n",
      "|   |   |   |   |- 18. Embarked in ['S']    79/28 0 (0.646 0.354 ) *\n",
      "|   |   |   |   |- 19. Embarked not in ['S']    42/17 0 (0.595 0.405 ) *\n",
      "|   |   |   |- 11. Embarked not in ['S','C']    1/0 0 (1 0 ) *\n",
      "|   |- 3. Sex not in ['male']    314/81 1 (0.258 0.742 ) [0.0992456]\n",
      "|   |   |- 6. Pclass in ['3']    144/72 0 (0.5 0.5 ) [0.0491071]\n",
      "|   |   |   |- 12. Embarked in ['S']    88/33 0 (0.625 0.375 ) *\n",
      "|   |   |   |- 13. Embarked not in ['S']    56/17 1 (0.304 0.696 ) [0.00273]\n",
      "|   |   |   |   |- 20. Embarked in ['Q']    33/9 1 (0.273 0.727 ) *\n",
      "|   |   |   |   |- 21. Embarked not in ['Q']    23/8 1 (0.348 0.652 ) *\n",
      "|   |   |- 7. Pclass not in ['3']    170/9 1 (0.053 0.947 ) [0.0011101]\n",
      "|   |   |   |- 14. Embarked in ['S']    115/8 1 (0.07 0.93 ) [0.0011152]\n",
      "|   |   |   |   |- 22. Pclass in ['2']    67/6 1 (0.09 0.91 ) *\n",
      "|   |   |   |   |- 23. Pclass not in ['2']    48/2 1 (0.042 0.958 ) *\n",
      "|   |   |   |- 15. Embarked not in ['S']    55/1 1 (0.018 0.982 ) [0.0001294]\n",
      "|   |   |   |   |- 24. Pclass in ['1']    46/1 1 (0.022 0.978 ) [0.000024]\n",
      "|   |   |   |   |   |- 30. Embarked in ['C']    43/1 1 (0.023 0.977 ) *\n",
      "|   |   |   |   |   |- 31. Embarked not in ['C']    3/0 1 (0 1 ) *\n",
      "|   |   |   |   |- 25. Pclass not in ['1']    9/0 1 (0 1 ) *\n",
      "Model:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8114607 0.0428926 \n",
      "[1]   train Accuracy 0.811448  0.0047604 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "Frame tr = train.mapVars(\"Survived,Sex,Pclass,Embarked\");\n",
    "\n",
    "CTree tree = CTree.newCART();\n",
    "tree.fit(tr, \"Survived\");\n",
    "tree.printSummary();\n",
    "\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", tree, 10, Accuracy.newMetric(true)).run().printContent();\n",
    "\n",
    "ClassifierResult pred = tree.predict(test);\n",
    "Frame submit = SolidFrame.byVars(test.rvar(\"PassengerId\"),pred.firstClasses().name(\"Survived\"));\n",
    "Csv.instance().quotes.set(false).write(submit, \"tree2-model.csv\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tree is much richer and there are more chances to be better. This is what happened after submission.\n",
    "\n",
    "__![Results after submission of enriched tree](images/titanic-tree2-submit.png)__\n",
    "\n",
    "**Nice!**. We advanced $704$ positions and improved our score with $0.01435$. On public leader board we have a nice $0.77990$ accuracy score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Overfitting with trees\n",
    "\n",
    "What about using other input features to improve our prediction accuracy? There are some of them which we can include directly, with no changes: *Age*,*Fare*,*SibSp* and *Parch*.\n",
    "\n",
    "We can change the script slightly, to include those new input features. But we can do better, we can use cross-validation to estimate what will happen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8114607 0.0320342 \n",
      "[1]   train Accuracy 0.811448  0.0035556 \n",
      "\n",
      "Model:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.7755556 0.0287869 \n",
      "[1]   train Accuracy 0.9320357 0.0034567 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "CTree tree = CTree.newCART();\n",
    "ClassifierEvaluation.cv(train.mapVars(\"Survived,Sex,Pclass,Embarked\"),\"Survived\", tree, 10, Accuracy.newMetric(true)).run().printContent();\n",
    "ClassifierEvaluation.cv(train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch\"),\"Survived\", tree, 10, Accuracy.newMetric(true)).run().printContent();;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the 10-crossfold estimator of the accuracy has dropped with a large quantity. What happens? We can have an idea if we take a look at the learned tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CTree model\n",
      "================\n",
      "\n",
      "Description:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "\n",
      "Capabilities:\n",
      "types inputs/targets: NOMINAL,INT,DOUBLE,BINARY/NOMINAL\n",
      "counts inputs/targets: [1,1000000] / [1,1]\n",
      "missing inputs/targets: true/false\n",
      "\n",
      "Learned model:\n",
      "\n",
      "total number of nodes: 393\n",
      "total number of leaves: 197\n",
      "description:\n",
      "split, n/err, classes (densities) [* if is leaf / purity if not]\n",
      "\n",
      "|- 1. root    891/342 0 (0.616 0.384 ) [0.139648]\n",
      "|   |- 2. Sex in ['male']    577/109 0 (0.811 0.189 ) [0.0238166]\n",
      "|   |   |- 4. Age<=6.5    91/25 0 (0.725 0.275 ) [0.0967065]\n",
      "|   |   |   |- 8. Parch<=0.5    60/7 0 (0.883 0.117 ) [0.0273851]\n",
      "|   |   |   |   |- 16. Fare<=22.4646    49/3 0 (0.939 0.061 ) [0.0022213]\n",
      "|   |   |   |   |   |- 32. Pclass in ['3']    42/2 0 (0.952 0.048 ) [0.0060469]\n",
      "|   |   |   |   |   |   |- 56. Fare<=7.8021    18/2 0 (0.889 0.111 ) [0.0378086]\n",
      "|   |   |   |   |   |   |   |- 92. Fare<=7.7625    16/1 0 (0.938 0.062 ) [0.0078125]\n",
      "|   |   |   |   |   |   |   |   |- 132. Fare<=7.2396    8/1 0 (0.875 0.125 ) [0.03125]\n",
      "|   |   |   |   |   |   |   |   |   |- 174. Fare<=7.2271    4/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 175. Fare>7.2271    4/1 0 (0.75 0.25 ) *\n",
      "|   |   |   |   |   |   |   |   |- 133. Fare>7.2396    8/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 93. Fare>7.7625    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |- 57. Fare>7.8021    24/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 33. Pclass not in ['3']    7/1 0 (0.857 0.143 ) [0.1020408]\n",
      "|   |   |   |   |   |   |- 58. Fare<=6.5    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 59. Fare>6.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |- 17. Fare>22.4646    11/4 0 (0.636 0.364 ) [0.0809917]\n",
      "|   |   |   |   |   |- 34. SibSp<=1    10/3 0 (0.7 0.3 ) [0.045]\n",
      "|   |   |   |   |   |   |- 60. Fare<=28.275    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 61. Fare>28.275    8/3 0 (0.625 0.375 ) [0.1020833]\n",
      "|   |   |   |   |   |   |   |- 94. Fare<=37.55    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |- 134. Fare<=33.25    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |- 135. Fare>33.25    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 95. Fare>37.55    5/1 0 (0.8 0.2 ) [0.0533333]\n",
      "|   |   |   |   |   |   |   |   |- 136. Fare<=54.2479    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 137. Fare>54.2479    3/1 0 (0.667 0.333 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |- 176. Fare<=139.1375    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 177. Fare>139.1375    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 35. SibSp>1    1/0 1 (0 1 ) *\n",
      "|   |   |   |- 9. Parch>0.5    31/13 1 (0.419 0.581 ) [0.4270849]\n",
      "|   |   |   |   |- 18. SibSp<=2.5    17/0 1 (0 1 ) *\n",
      "|   |   |   |   |- 19. SibSp>2.5    14/1 0 (0.929 0.071 ) [0.0308642]\n",
      "|   |   |   |   |   |- 36. Age<=2.5    8/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 37. Age>2.5    6/1 0 (0.833 0.167 ) [0.1111111]\n",
      "|   |   |   |   |   |   |- 62. Fare<=31.33125    4/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 63. Fare>31.33125    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |- 5. Age>6.5    486/84 0 (0.827 0.173 ) [0.0228654]\n",
      "|   |   |   |- 10. Pclass in ['2','3']    375/43 0 (0.885 0.115 ) [0.0044611]\n",
      "|   |   |   |   |- 20. Fare<=51.6979    365/39 0 (0.893 0.107 ) [0.0037998]\n",
      "|   |   |   |   |   |- 38. Age<=13    27/4 0 (0.852 0.148 ) [0.0808081]\n",
      "|   |   |   |   |   |   |- 64. Age<=11.5    19/3 0 (0.842 0.158 ) [0.0787935]\n",
      "|   |   |   |   |   |   |   |- 96. Pclass in ['3']    18/2 0 (0.889 0.111 ) [0.0342857]\n",
      "|   |   |   |   |   |   |   |   |- 138. Age<=8.5    6/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 139. Age>8.5    12/2 0 (0.833 0.167 ) [0.122449]\n",
      "|   |   |   |   |   |   |   |   |   |- 178. Age<=9.5    5/2 0 (0.6 0.4 ) [0.2133333]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 216. SibSp<=2.5    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 254. Parch<=1.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 255. Parch>1.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 217. SibSp>2.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 179. Age>9.5    7/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 97. Pclass not in ['3']    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 65. Age>11.5    8/1 0 (0.875 0.125 ) [0.21875]\n",
      "|   |   |   |   |   |   |   |- 98. SibSp<=0.5    7/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 99. SibSp>0.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |- 39. Age>13    338/35 0 (0.896 0.104 ) [0.0022179]\n",
      "|   |   |   |   |   |   |- 66. Age<=32.25    218/27 0 (0.876 0.124 ) [0.008204]\n",
      "|   |   |   |   |   |   |   |- 100. Age<=30.75    193/20 0 (0.896 0.104 ) [0.0045753]\n",
      "|   |   |   |   |   |   |   |   |- 140. Embarked in ['S']    166/14 0 (0.916 0.084 ) [0.006189]\n",
      "|   |   |   |   |   |   |   |   |   |- 180. Fare<=7.0104    5/2 0 (0.6 0.4 ) [0.5]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 218. Age<=22    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 219. Age>22    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 181. Fare>7.0104    161/12 0 (0.925 0.075 ) [0.0040488]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 220. Fare<=11    118/12 0 (0.898 0.102 ) [0.0030607]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 256. Fare<=7.9104    62/4 0 (0.935 0.065 ) [0.0052577]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 282. Fare<=7.7979    38/4 0 (0.895 0.105 ) [0.019506]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 306. Fare<=7.7854    32/2 0 (0.938 0.062 ) [0.0117077]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 328. Age<=23    17/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 329. Age>23    15/2 0 (0.867 0.133 ) [0.0210388]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 352. Age<=25.5    9/2 0 (0.778 0.222 ) [0.0493827]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 364. Fare<=7.09585    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 365. Fare>7.09585    6/2 0 (0.667 0.333 ) [0.0277778]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 376. SibSp<=0.5    4/1 0 (0.75 0.25 ) [0.125]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 382. Age<=24.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 383. Age>24.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 377. SibSp>0.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 353. Age>25.5    6/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 307. Fare>7.7854    6/2 0 (0.667 0.333 ) [0.0444444]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 330. Age<=27.5    5/2 0 (0.6 0.4 ) [0.18]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 354. Age<=25.5    4/1 0 (0.75 0.25 ) [0.125]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 366. Age<=21.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 367. Age>21.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 355. Age>25.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 331. Age>27.5    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 283. Fare>7.7979    24/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 257. Fare>7.9104    56/8 0 (0.857 0.143 ) [0.0091837]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 284. Age<=19.5    16/4 0 (0.75 0.25 ) [0.1113636]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 308. Fare<=8.10415    5/2 1 (0.4 0.6 ) [0.0133333]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 332. Age<=17    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 333. Age>17    3/1 1 (0.333 0.667 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 309. Fare>8.10415    11/1 0 (0.909 0.091 ) [0.0440771]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 334. Fare<=10.3354    8/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 335. Fare>10.3354    3/1 0 (0.667 0.333 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 285. Age>19.5    40/4 0 (0.9 0.1 ) [0.0140659]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 310. Age<=26.5    26/1 0 (0.962 0.038 ) [0.035503]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 336. SibSp<=0.5    24/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 337. SibSp>0.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 311. Age>26.5    14/3 0 (0.786 0.214 ) [0.0510204]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 338. Fare<=8.35625    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 339. Fare>8.35625    9/3 0 (0.667 0.333 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 356. Fare<=10    6/3 0 (0.5 0.5 ) [0.1]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 368. Age<=29.5    5/2 0 (0.6 0.4 ) [0.0133333]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 378. Fare<=9.49165    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 379. Fare>9.49165    3/1 0 (0.667 0.333 ) [0.4444444]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 384. Age<=28.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 385. Age>28.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 369. Age>29.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 357. Fare>10    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 221. Fare>11    43/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 141. Embarked not in ['S']    27/6 0 (0.778 0.222 ) [0.0274972]\n",
      "|   |   |   |   |   |   |   |   |   |- 182. Fare<=7.74585    16/2 0 (0.875 0.125 ) [0.0306122]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 222. Age<=22.75    8/2 0 (0.75 0.25 ) [0.075]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 258. Age<=19.5    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 259. Age>19.5    5/2 0 (0.6 0.4 ) [0.08]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 286. Fare<=7.48125    4/2 0 (0.5 0.5 ) [0]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 312. Age<=21    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 313. Age>21    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 287. Fare>7.48125    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 223. Age>22.75    8/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 183. Fare>7.74585    11/4 0 (0.636 0.364 ) [0.1799816]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 224. Fare<=11.175    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 225. Fare>11.175    9/2 0 (0.778 0.222 ) [0.0790123]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 260. Fare<=15.62085    4/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 261. Fare>15.62085    5/2 0 (0.6 0.4 ) [0.48]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 288. Fare<=21.39375    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 289. Fare>21.39375    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 101. Age>30.75    25/7 0 (0.72 0.28 ) [0.0213818]\n",
      "|   |   |   |   |   |   |   |   |- 142. Parch<=0.5    22/7 0 (0.682 0.318 ) [0.0442739]\n",
      "|   |   |   |   |   |   |   |   |   |- 184. Fare<=21.05    21/6 0 (0.714 0.286 ) [0.0171858]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 226. Fare<=14.425    19/6 0 (0.684 0.316 ) [0.052016]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 262. Fare<=11.75    18/5 0 (0.722 0.278 ) [0.0308642]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 290. Fare<=8.20625    15/5 0 (0.667 0.333 ) [0.037037]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 314. Fare<=7.9104    9/2 0 (0.778 0.222 ) [0.0533333]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 340. Age<=31.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 341. Age>31.5    7/2 0 (0.714 0.286 ) [0.0653061]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 358. Fare<=7.875    5/2 0 (0.6 0.4 ) [0.18]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 370. Embarked in ['Q']    4/1 0 (0.75 0.25 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 371. Embarked not in ['Q']    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 359. Fare>7.875    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 315. Fare>7.9104    6/3 0 (0.5 0.5 ) [0]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 342. Fare<=7.9875    4/2 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 343. Fare>7.9875    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 291. Fare>8.20625    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 263. Fare>11.75    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 227. Fare>14.425    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 185. Fare>21.05    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 143. Parch>0.5    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 67. Age>32.25    120/8 0 (0.933 0.067 ) [0.0047863]\n",
      "|   |   |   |   |   |   |   |- 102. Fare<=7.9104    42/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 103. Fare>7.9104    78/8 0 (0.897 0.103 ) [0.0170762]\n",
      "|   |   |   |   |   |   |   |   |- 144. Fare<=7.9875    4/2 0 (0.5 0.5 ) [0.1666667]\n",
      "|   |   |   |   |   |   |   |   |   |- 186. SibSp<=1    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 228. Age<=41.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 229. Age>41.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 187. SibSp>1    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 145. Fare>7.9875    74/6 0 (0.919 0.081 ) [0.0084733]\n",
      "|   |   |   |   |   |   |   |   |   |- 188. Fare<=13.93125    45/6 0 (0.867 0.133 ) [0.0341414]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 230. Fare<=13.68125    44/5 0 (0.886 0.114 ) [0.0088288]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 264. Age<=60.5    38/4 0 (0.895 0.105 ) [0.0055627]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 292. Age<=46    26/3 0 (0.885 0.115 ) [0.0626036]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 316. Age<=44.5    25/2 0 (0.92 0.08 ) [0.0272]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 344. Fare<=12.9375    17/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 345. Fare>12.9375    8/2 0 (0.75 0.25 ) [0.0416667]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 360. Age<=40.5    6/1 0 (0.833 0.167 ) [0.0555556]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 372. Age<=35    3/1 0 (0.667 0.333 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 373. Age>35    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 361. Age>40.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 317. Age>44.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 293. Age>46    12/1 0 (0.917 0.083 ) [0.0277778]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 318. Fare<=8.55625    4/1 0 (0.75 0.25 ) [0.375]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 346. Fare<=8.08125    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 347. Fare>8.08125    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 319. Fare>8.55625    8/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 265. Age>60.5    6/1 0 (0.833 0.167 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 294. Age<=68    5/1 0 (0.8 0.2 ) [0.12]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 320. Fare<=10    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 321. Fare>10    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 295. Age>68    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 231. Fare>13.68125    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 189. Fare>13.93125    29/0 0 (1 0 ) *\n",
      "|   |   |   |   |- 21. Fare>51.6979    10/4 0 (0.6 0.4 ) [0.32]\n",
      "|   |   |   |   |   |- 40. Fare<=64.9979    5/1 1 (0.2 0.8 ) [0.125]\n",
      "|   |   |   |   |   |   |- 68. Age<=30    3/1 1 (0.333 0.667 ) *\n",
      "|   |   |   |   |   |   |- 69. Age>30    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |- 41. Fare>64.9979    5/0 0 (1 0 ) *\n",
      "|   |   |   |- 11. Pclass not in ['2','3']    111/41 0 (0.631 0.369 ) [0.0349964]\n",
      "|   |   |   |   |- 22. Age<=53    84/37 0 (0.56 0.44 ) [0.0298491]\n",
      "|   |   |   |   |   |- 42. Fare<=26.10625    6/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 43. Fare>26.10625    78/37 0 (0.526 0.474 ) [0.0711404]\n",
      "|   |   |   |   |   |   |- 70. Fare<=27.1354    12/1 1 (0.083 0.917 ) [0.0289256]\n",
      "|   |   |   |   |   |   |   |- 104. Age<=43.5    8/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 105. Age>43.5    4/1 1 (0.25 0.75 ) [0.125]\n",
      "|   |   |   |   |   |   |   |   |- 146. Age<=46.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |- 147. Age>46.5    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 71. Fare>27.1354    66/26 0 (0.606 0.394 ) [0.0254407]\n",
      "|   |   |   |   |   |   |   |- 106. Fare<=29.85    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 107. Fare>29.85    61/26 0 (0.574 0.426 ) [0.0481409]\n",
      "|   |   |   |   |   |   |   |   |- 148. Fare<=32.5104    7/1 1 (0.143 0.857 ) [0.1020408]\n",
      "|   |   |   |   |   |   |   |   |   |- 190. Fare<=30.25    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 191. Fare>30.25    5/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 149. Fare>32.5104    54/20 0 (0.63 0.37 ) [0.0304949]\n",
      "|   |   |   |   |   |   |   |   |   |- 192. Fare<=387.6646    52/18 0 (0.654 0.346 ) [0.0396091]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 232. SibSp<=0.5    22/4 0 (0.818 0.182 ) [0.0797784]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 266. Age<=29.5    11/4 0 (0.636 0.364 ) [0.1706021]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 296. Fare<=77.00835    4/1 1 (0.25 0.75 ) [0.125]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 322. Embarked in ['S']    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 323. Embarked not in ['S']    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 297. Fare>77.00835    7/1 0 (0.857 0.143 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 324. Age<=21.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 325. Age>21.5    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 267. Age>29.5    11/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 233. SibSp>0.5    30/14 0 (0.533 0.467 ) [0.0247009]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 268. Fare<=115.44165    26/11 0 (0.577 0.423 ) [0.0466941]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 298. Age<=49.5    23/11 0 (0.522 0.478 ) [0.0575089]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 326. Age<=47    18/7 0 (0.611 0.389 ) [0.0604938]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 348. Age<=43    15/7 0 (0.533 0.467 ) [0.0670085]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 362. Age<=22    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 363. Age>22    13/6 1 (0.462 0.538 ) [0.1278107]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 374. Age<=27.5    3/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 375. Age>27.5    10/4 0 (0.6 0.4 ) [0.08]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 380. Age<=36.5    5/1 0 (0.8 0.2 ) [0.12]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 386. Fare<=61.8    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 387. Fare>61.8    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 381. Age>36.5    5/2 1 (0.4 0.6 ) [0.08]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 388. Parch<=0.5    4/2 0 (0.5 0.5 ) [0.1666667]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 390. Fare<=71.55    3/1 0 (0.667 0.333 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 392. Fare<=52.8271    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 393. Fare>52.8271    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 391. Fare>71.55    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 389. Parch>0.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 349. Age>43    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |- 327. Age>47    5/1 1 (0.2 0.8 ) [0.32]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 350. Fare<=99.99375    4/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |   |   |- 351. Fare>99.99375    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 299. Age>49.5    3/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 269. Fare>115.44165    4/1 1 (0.25 0.75 ) [0.375]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 300. SibSp<=2.5    3/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 301. SibSp>2.5    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 193. Fare>387.6646    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |- 23. Age>53    27/4 0 (0.852 0.148 ) [0.071035]\n",
      "|   |   |   |   |   |- 44. Age<=75.5    22/2 0 (0.909 0.091 ) [0.0199546]\n",
      "|   |   |   |   |   |   |- 72. Age<=60.5    10/2 0 (0.8 0.2 ) [0.1422222]\n",
      "|   |   |   |   |   |   |   |- 108. SibSp<=0.5    9/1 0 (0.889 0.111 ) [0.0308642]\n",
      "|   |   |   |   |   |   |   |   |- 150. Fare<=33.0979    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 151. Fare>33.0979    4/1 0 (0.75 0.25 ) [0.125]\n",
      "|   |   |   |   |   |   |   |   |   |- 194. Parch<=0.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 195. Parch>0.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 109. SibSp>0.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 73. Age>60.5    12/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 45. Age>75.5    5/2 0 (0.6 0.4 ) [0.2133333]\n",
      "|   |   |   |   |   |   |- 74. Fare<=28.7104    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 75. Fare>28.7104    3/1 1 (0.333 0.667 ) [0.4444444]\n",
      "|   |   |   |   |   |   |   |- 110. Fare<=30.3479    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 111. Fare>30.3479    1/0 0 (1 0 ) *\n",
      "|   |- 3. Sex not in ['male']    314/81 1 (0.258 0.742 ) [0.0992456]\n",
      "|   |   |- 6. Pclass in ['3']    144/72 0 (0.5 0.5 ) [0.0698006]\n",
      "|   |   |   |- 12. Fare<=23.35    117/48 1 (0.41 0.59 ) [0.029742]\n",
      "|   |   |   |   |- 24. Age<=16.5    40/12 1 (0.3 0.7 ) [0.0688095]\n",
      "|   |   |   |   |   |- 46. Embarked in ['S']    12/5 0 (0.583 0.417 ) [0.2480159]\n",
      "|   |   |   |   |   |   |- 76. Fare<=10.7979    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 77. Fare>10.7979    7/2 1 (0.286 0.714 ) [0.4081633]\n",
      "|   |   |   |   |   |   |   |- 112. SibSp<=2    5/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 113. SibSp>2    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 47. Embarked not in ['S']    28/5 1 (0.179 0.821 ) [0.0302095]\n",
      "|   |   |   |   |   |   |- 78. Fare<=15.3729    19/5 1 (0.263 0.737 ) [0.08]\n",
      "|   |   |   |   |   |   |   |- 114. Age<=14.75    11/3 1 (0.273 0.727 ) [0.2350781]\n",
      "|   |   |   |   |   |   |   |   |- 152. Fare<=13.93545    9/1 1 (0.111 0.889 ) [0.0493827]\n",
      "|   |   |   |   |   |   |   |   |   |- 196. Fare<=8.00835    6/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 197. Fare>8.00835    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 234. Fare<=12.3292    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 235. Fare>12.3292    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 153. Fare>13.93545    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 115. Age>14.75    8/2 1 (0.25 0.75 ) [0.0416667]\n",
      "|   |   |   |   |   |   |   |   |- 154. Fare<=7.68125    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |- 155. Fare>7.68125    6/1 1 (0.167 0.833 ) [0.0277778]\n",
      "|   |   |   |   |   |   |   |   |   |- 198. Fare<=7.8896    4/1 1 (0.25 0.75 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 199. Fare>7.8896    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 79. Fare>15.3729    9/0 1 (0 1 ) *\n",
      "|   |   |   |   |- 25. Age>16.5    77/36 1 (0.468 0.532 ) [0.0377804]\n",
      "|   |   |   |   |   |- 48. Fare<=7.8875    28/8 1 (0.286 0.714 ) [0.1482946]\n",
      "|   |   |   |   |   |   |- 80. Age<=28.25    21/5 1 (0.238 0.762 ) [0.0791589]\n",
      "|   |   |   |   |   |   |   |- 116. Age<=18.5    7/2 1 (0.286 0.714 ) [0.0367347]\n",
      "|   |   |   |   |   |   |   |   |- 156. Embarked in ['S']    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |- 157. Embarked not in ['S']    5/1 1 (0.2 0.8 ) [0.12]\n",
      "|   |   |   |   |   |   |   |   |   |- 200. Fare<=7.4896    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 201. Fare>7.4896    3/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 117. Age>18.5    14/3 1 (0.214 0.786 ) [0.0949765]\n",
      "|   |   |   |   |   |   |   |   |- 158. Parch<=1    13/2 1 (0.154 0.846 ) [0.0449954]\n",
      "|   |   |   |   |   |   |   |   |   |- 202. Age<=24    9/1 1 (0.111 0.889 ) [0.0493827]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 236. Embarked in ['Q']    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 270. Fare<=7.8146    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 271. Fare>7.8146    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 237. Embarked not in ['Q']    6/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 203. Age>24    4/1 1 (0.25 0.75 ) [0.125]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 238. Fare<=7.8146    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 239. Fare>7.8146    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 159. Parch>1    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 81. Age>28.25    7/3 1 (0.429 0.571 ) [0.0612245]\n",
      "|   |   |   |   |   |   |   |- 118. Fare<=7.8667    6/3 0 (0.5 0.5 ) [0.1]\n",
      "|   |   |   |   |   |   |   |   |- 160. Fare<=7.8021    5/2 1 (0.4 0.6 ) *\n",
      "|   |   |   |   |   |   |   |   |- 161. Fare>7.8021    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |- 119. Fare>7.8667    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |- 49. Fare>7.8875    49/21 0 (0.571 0.429 ) [0.037415]\n",
      "|   |   |   |   |   |   |- 82. Fare<=15.675    33/11 0 (0.667 0.333 ) [0.0587695]\n",
      "|   |   |   |   |   |   |   |- 120. SibSp<=0.5    22/10 0 (0.545 0.455 ) [0.0967498]\n",
      "|   |   |   |   |   |   |   |   |- 162. Age<=25    9/2 0 (0.778 0.222 ) [0.1234568]\n",
      "|   |   |   |   |   |   |   |   |   |- 204. Age<=19    4/2 0 (0.5 0.5 ) [0.5]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 240. Fare<=12.14795    2/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 241. Fare>12.14795    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 205. Age>19    5/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 163. Age>25    13/5 1 (0.385 0.615 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |- 206. Age<=27.5    5/1 1 (0.2 0.8 ) [0.32]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 242. Fare<=13.8604    4/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 243. Fare>13.8604    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 207. Age>27.5    8/4 0 (0.5 0.5 ) [0.1666667]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 244. Fare<=8.6729    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 245. Fare>8.6729    6/2 1 (0.333 0.667 ) [0.2222222]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 272. Age<=34    3/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 273. Age>34    3/1 0 (0.667 0.333 ) [0.4444444]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 302. Age<=54    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 303. Age>54    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 121. SibSp>0.5    11/1 0 (0.909 0.091 ) [0.1652893]\n",
      "|   |   |   |   |   |   |   |   |- 164. SibSp<=2.5    10/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 165. SibSp>2.5    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 83. Fare>15.675    16/6 1 (0.375 0.625 ) [0.105433]\n",
      "|   |   |   |   |   |   |   |- 122. Age<=21    4/2 0 (0.5 0.5 ) [0.1666667]\n",
      "|   |   |   |   |   |   |   |   |- 166. Embarked in ['S']    3/1 0 (0.667 0.333 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |- 208. SibSp<=1.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 209. SibSp>1.5    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |- 167. Embarked not in ['S']    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 123. Age>21    12/4 1 (0.333 0.667 ) [0.0991736]\n",
      "|   |   |   |   |   |   |   |   |- 168. Age<=25    4/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 169. Age>25    8/4 0 (0.5 0.5 ) [0.1666667]\n",
      "|   |   |   |   |   |   |   |   |   |- 210. Age<=30    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 211. Age>30    6/2 1 (0.333 0.667 ) [0.1777778]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 246. Parch<=1.5    5/1 1 (0.2 0.8 ) [0.12]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 274. Age<=32    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 275. Age>32    3/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 247. Parch>1.5    1/0 0 (1 0 ) *\n",
      "|   |   |   |- 13. Fare>23.35    27/3 0 (0.889 0.111 ) [0.0241975]\n",
      "|   |   |   |   |- 26. Embarked in ['S']    25/2 0 (0.92 0.08 ) [0.0197531]\n",
      "|   |   |   |   |   |- 50. Age<=5.5    6/1 0 (0.833 0.167 ) [0.4444444]\n",
      "|   |   |   |   |   |   |- 84. Age<=3.5    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 85. Age>3.5    4/1 0 (0.75 0.25 ) [0.125]\n",
      "|   |   |   |   |   |   |   |- 124. SibSp<=6    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |- 125. SibSp>6    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |- 51. Age>5.5    19/1 0 (0.947 0.053 ) [0.0207756]\n",
      "|   |   |   |   |   |   |- 86. Parch<=4.5    15/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |- 87. Parch>4.5    4/1 0 (0.75 0.25 ) [0.125]\n",
      "|   |   |   |   |   |   |   |- 126. Age<=40    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |- 127. Age>40    2/0 0 (1 0 ) *\n",
      "|   |   |   |   |- 27. Embarked not in ['S']    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |- 7. Pclass not in ['3']    170/9 1 (0.053 0.947 ) [0.0050089]\n",
      "|   |   |   |- 14. Age<=2.5    8/1 1 (0.125 0.875 ) [0.21875]\n",
      "|   |   |   |   |- 28. Fare<=120.3271    7/0 1 (0 1 ) *\n",
      "|   |   |   |   |- 29. Fare>120.3271    1/0 0 (1 0 ) *\n",
      "|   |   |   |- 15. Age>2.5    162/8 1 (0.049 0.951 ) [0.0041502]\n",
      "|   |   |   |   |- 30. Fare<=28.85625    68/7 1 (0.103 0.897 ) [0.0240213]\n",
      "|   |   |   |   |   |- 52. Fare<=28.23125    67/6 1 (0.09 0.91 ) [0.0103672]\n",
      "|   |   |   |   |   |   |- 88. Age<=56    65/5 1 (0.077 0.923 ) [0.0041497]\n",
      "|   |   |   |   |   |   |   |- 128. SibSp<=0.5    44/2 1 (0.045 0.955 ) [0.0034435]\n",
      "|   |   |   |   |   |   |   |   |- 170. Fare<=13.25    24/2 1 (0.083 0.917 ) [0.0099206]\n",
      "|   |   |   |   |   |   |   |   |   |- 212. Fare<=12.825    10/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 213. Fare>12.825    14/2 1 (0.143 0.857 ) [0.0197897]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 248. Age<=26    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 276. Parch<=1    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 277. Parch>1    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 249. Age>26    11/1 1 (0.091 0.909 ) [0.0440771]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 278. Age<=37    8/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 279. Age>37    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 304. Age<=41    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |   |- 305. Age>41    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 171. Fare>13.25    20/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |- 129. SibSp>0.5    21/3 1 (0.143 0.857 ) [0.0163265]\n",
      "|   |   |   |   |   |   |   |   |- 172. Age<=25    6/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |- 173. Age>25    15/3 1 (0.2 0.8 ) [0.1088889]\n",
      "|   |   |   |   |   |   |   |   |   |- 214. Age<=27.5    3/1 0 (0.667 0.333 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 250. Fare<=23.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 251. Fare>23.5    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |   |   |   |   |   |- 215. Age>27.5    12/1 1 (0.083 0.917 ) [0.0416667]\n",
      "|   |   |   |   |   |   |   |   |   |   |- 252. Age<=43    9/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |- 253. Age>43    3/1 1 (0.333 0.667 ) [0.1111111]\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 280. Fare<=26.125    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |   |   |   |   |- 281. Fare>26.125    1/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 89. Age>56    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |- 53. Fare>28.23125    1/0 0 (1 0 ) *\n",
      "|   |   |   |   |- 31. Fare>28.85625    94/1 1 (0.011 0.989 ) [0.0011921]\n",
      "|   |   |   |   |   |- 54. Parch<=1.5    79/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |- 55. Parch>1.5    15/1 1 (0.067 0.933 ) [0.0177778]\n",
      "|   |   |   |   |   |   |- 90. Age<=24.5    10/0 1 (0 1 ) *\n",
      "|   |   |   |   |   |   |- 91. Age>24.5    5/1 1 (0.2 0.8 ) [0.12]\n",
      "|   |   |   |   |   |   |   |- 130. Age<=33.5    2/1 0 (0.5 0.5 ) *\n",
      "|   |   |   |   |   |   |   |- 131. Age>33.5    3/0 1 (0 1 ) *\n"
     ]
    }
   ],
   "source": [
    "tree.fit(train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch\"), \"Survived\");\n",
    "tree.printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Notice how large is the tree. Basically the tree was full grown and overfit the training data set too much. We can ask ourselves why that happens? Why it happens now, and did not happened when we had fewer inputs? The answer is that it happened also before. But it's consequences were not so drastic.\n",
    "\n",
    "The first tree used for training just $3$ input nominal features. Notice that all three features are nominal. The maximum number of groups which one can form is given by the product of the number of levels for each feature. This total maximal number is $2*3*3=18$. It practically exhausted the discrimination potential of those features. It did overfit in that reduced space of features. When we apply the model to the whole data set, the effect of exhaustion is not seen anymore.\n",
    "\n",
    "The second tree does the same thing, but this time in a richer space, with added input dimensions. Compared with the full feature space, we see the effect.\n",
    "\n",
    "There are two approaches to avoid overfit for a decision tree. The first approach is to stop learning up to the moment when we exhaust the data. The name for this approach is *early stop*. We can do that by specifying some parameters of the tree model:\n",
    "\n",
    "* Set a minimum number of instances for leaf node\n",
    "* Set a maximal depth for the tree\n",
    "* Not implemented yet, but easy to do: complexity threshold, maximal number of nodes in a tree\n",
    "\n",
    "The second approach is to prune the tree. Pruning procedure consists of growing the full tree and later on removing some nodes if they do not provide some type of gain. Currently we implemented only *reduced error pruning strategy*.\n",
    "\n",
    "We will test with 10-fold cross validation an early-stopping strategy to see how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CTree{maxDepth=8,minCount=4,purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8137328 0.0261994 \n",
      "[1]   train Accuracy 0.8698091 0.0072535 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "ClassifierEvaluation.cv(train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch\"),\"Survived\",\n",
    "                        tree.maxDepth.set(8).minCount.set(4), 10, Accuracy.newMetric(true)).run().printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "I tried some values, just to show that we can do something about it, but the progress did not appear. We should try a different approach, and that is an ensemble. Next session contains directions on how to build such an ensemble."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. CForest model\n",
    "\n",
    "Random forests are well-known to work well when the irreducible error from the training data is high. This is probably the case of this Titanic data set. We have reasons to believe that this is the situation since it was a tragedy. A lot of random or not-so-expected things happened. That happened despite of the bravery and the sacrifice of the crew and others.\n",
    "\n",
    "Random forests are the invention of [Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman). The first design was a joint effort together with [Adele Cutler](http://www.math.usu.edu/adele/). The base of random forests is bagging (or **b**ootstrapp **ag**gregation). On top of that, selecting just a random limited number of variables at each node is the core of the algorithm.\n",
    "\n",
    "We will work with random forests for now. This ensemble is mode robust and is capable of obtaining much better results than a single tree. At the same time we will introduce 10-fold cross validation to check our progress and estimate the error produced.\n",
    "\n",
    "In the beginning we will use 10-fold cross validation for estimating the accuracy on public leader board.\n",
    "\n",
    "We will test first with 10 fold cv the tree model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CTree{purity=GiniGain,splitter=Random,varSelector=VarSelector[ALL]}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8114482 0.0499065 \n",
      "[1]   train Accuracy 0.8114478 0.0055383 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "ClassifierEvaluation.cv(train.mapVars(\"Survived,Sex,Pclass,Embarked\"), \"Survived\", CTree.newCART(), 10, Accuracy.newMetric(true)).run().printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Our first random forest\n",
    "\n",
    "The name of the random forest implementation is `CForest`. To build a new ensemble of trees, one have to instantiate it in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "CForest rf = CForest.newModel();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of things which can be customized for a random forest. Among them one can change:\n",
    "\n",
    "* Number of trees for classification\n",
    "* Which kind of weak classifierModel to use (you can customize this customized accordingly, like any other classifierModel)\n",
    "* Number of threads in pool (if you want to use parallelism)\n",
    "* What to do after each running step\n",
    "\n",
    "Let's build one and use ore new cross validation procedure to estimate it's error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CForest{poolSize=3,rowSampler=Bootstrap(p=1),seed=123}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8113483 0.0357612 \n",
      "[1]   train Accuracy 0.8114466 0.0040024 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "Frame tr = train.mapVars(\"Survived,Sex,Pclass,Embarked\");\n",
    "CForest rf = CForest.newModel().runs.set(100).poolSize.set(3).seed.set(123L);\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", rf, 10, Accuracy.newMetric(true)).run().printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, an identical output. This is due to the fact that our variables are already exhausted by the tree. It looks like an underfit. If one consider bias variance trade off, one can see this as high bias. We need to enrich our feature space to improve our performance.\n",
    "\n",
    "Let's be direct and test what would happen if we would use all our directly usable features? This time we will fit also the training data set, to see the distribution of the training error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CForest{poolSize=3,rowSampler=Bootstrap(p=1),seed=123}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8316604 0.0182228 \n",
      "[1]   train Accuracy 0.9280458 0.0030014 \n",
      "\n",
      "> Confusion matrix\n",
      " - Frequency table\n",
      "Ac\\Pr |    0    1 | total \n",
      "----- |    -    - | ----- \n",
      "    0 | >534   15 |   549 \n",
      "    1 |   46 >296 |   342 \n",
      "----- |    -    - | ----- \n",
      "total |  580  311 |   891 \n",
      " - Probability table\n",
      "Ac\\Pr |      0      1 | total \n",
      "----- |      -      - | ----- \n",
      "    0 | >0.599  0.017 | 0.616 \n",
      "    1 |  0.052 >0.332 | 0.384 \n",
      "----- |      -      - | ----- \n",
      "total |  0.651  0.349 | 1.000 \n",
      "\n",
      "\n",
      "Complete cases 891 from 891\n",
      "Acc: 0.9315376         (Accuracy )\n",
      "F1:  0.9459699         (F1 score / F-measure)\n",
      "MCC: 0.8551446         (Matthew correlation coefficient)\n",
      "Pre: 0.9206897         (Precision)\n",
      "Rec: 0.9726776         (Recall)\n",
      "G:   0.9463267         (G-measure)\n"
     ]
    }
   ],
   "source": [
    "Frame tr = train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch\");\n",
    "CForest rf = CForest.newModel()\n",
    "    .runs.set(100)\n",
    "    .poolSize.set(3)\n",
    "    .seed.set(123L);\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", rf, 10, Accuracy.newMetric(true)).run().printContent();\n",
    "\n",
    "rf.fit(tr, \"Survived\");\n",
    "ClassifierResult fit = rf.predict(test);\n",
    "Confusion.from(tr.rvar(\"Survived\"), rf.predict(tr).firstClasses()).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we have a good example of overfit. Why is that? Look at the confusion matrix on the training set. We fit too well the training data. This data set is well known for its high irreducible error. And there is an explanation for that. During the tragic event a lot of exceptional things happened. For example I read somewhere that an old lady which had a dog was not allowed to embark with her pet due to regulations. As a consequence she decided to not leave it and she chose to die with him. It's close to impossible to learn those kind of things, even if the information would be available. \n",
    "\n",
    "We should reduce the error somehow. We can try to decrease the overfit by adding more learners. Let's see if that would be enough for our purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CForest{poolSize=3,rowSampler=Bootstrap(p=1),runs=500,seed=123}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8327341 0.0608184 \n",
      "[1]   train Accuracy 0.9287932 0.0040688 \n",
      "\n",
      "> Confusion matrix\n",
      " - Frequency table\n",
      "Ac\\Pr |    0    1 | total \n",
      "----- |    -    - | ----- \n",
      "    0 | >533   16 |   549 \n",
      "    1 |   48 >294 |   342 \n",
      "----- |    -    - | ----- \n",
      "total |  581  310 |   891 \n",
      " - Probability table\n",
      "Ac\\Pr |      0      1 | total \n",
      "----- |      -      - | ----- \n",
      "    0 | >0.598  0.018 | 0.616 \n",
      "    1 |  0.054 >0.330 | 0.384 \n",
      "----- |      -      - | ----- \n",
      "total |  0.652  0.348 | 1.000 \n",
      "\n",
      "\n",
      "Complete cases 891 from 891\n",
      "Acc: 0.9281706         (Accuracy )\n",
      "F1:  0.9433628         (F1 score / F-measure)\n",
      "MCC: 0.8479548         (Matthew correlation coefficient)\n",
      "Pre: 0.9173838         (Precision)\n",
      "Rec: 0.9708561         (Recall)\n",
      "G:   0.9437413         (G-measure)\n"
     ]
    }
   ],
   "source": [
    "Frame tr = train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch\");\n",
    "CForest rf = CForest.newModel()\n",
    "    .runs.set(500)\n",
    "    .poolSize.set(3)\n",
    "    .seed.set(123L);\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", rf, 10, Accuracy.newMetric(true)).run().printContent();\n",
    "\n",
    "rf.fit(tr, \"Survived\");\n",
    "ClassifierResult fit = rf.predict(test);\n",
    "Confusion.from(tr.rvar(\"Survived\"), rf.predict(tr).firstClasses()).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is slightly better than before. But the difference does not look significantly better than previous. We will use a simple pre-pruning strategy is to limit the number instances in leaf nodes. We set the minimum count to $3$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CForest{model=CTree,rowSampler=Bootstrap(p=1),seed=123}\n",
      "Raw scores:\n",
      "===========\n",
      "     dataset round fold Accuracy       dataset round fold Accuracy       dataset round fold Accuracy  \n",
      " [0]    test     0    0 0.8111111  [7]    test     0    7 0.8426966 [14]   train     0    4 0.9127182 \n",
      " [1]    test     0    1 0.8651685  [8]    test     0    8 0.8539326 [15]   train     0    5 0.9089776 \n",
      " [2]    test     0    2 0.8988764  [9]    test     0    9 0.8089888 [16]   train     0    6 0.9127182 \n",
      " [3]    test     0    3 0.8539326 [10]   train     0    0 0.9076155 [17]   train     0    7 0.9064838 \n",
      " [4]    test     0    4 0.7640449 [11]   train     0    1 0.9002494 [18]   train     0    8 0.9114713 \n",
      " [5]    test     0    5 0.7977528 [12]   train     0    2 0.9027431 [19]   train     0    9 0.9014963 \n",
      " [6]    test     0    6 0.8202247 [13]   train     0    3 0.9077307 \n",
      "\n",
      "Round scores:\n",
      "=============\n",
      "    dataset round Accuracy_mean Accuracy_std \n",
      "[0]    test     0   0.8316729    0.0368012   \n",
      "[1]   train     0   0.9072204    0.0042892   \n",
      "\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8316729 0.0387919 \n",
      "[1]   train Accuracy 0.9072204 0.0045213 \n",
      "\n",
      "> Confusion matrix\n",
      " - Frequency table\n",
      "Ac\\Pr |    0    1 | total \n",
      "----- |    -    - | ----- \n",
      "    0 | >525   24 |   549 \n",
      "    1 |   58 >284 |   342 \n",
      "----- |    -    - | ----- \n",
      "total |  583  308 |   891 \n",
      " - Probability table\n",
      "Ac\\Pr |      0      1 | total \n",
      "----- |      -      - | ----- \n",
      "    0 | >0.589  0.027 | 0.616 \n",
      "    1 |  0.065 >0.319 | 0.384 \n",
      "----- |      -      - | ----- \n",
      "total |  0.654  0.346 | 1.000 \n",
      "\n",
      "\n",
      "Complete cases 891 from 891\n",
      "Acc: 0.9079686         (Accuracy )\n",
      "F1:  0.9275618         (F1 score / F-measure)\n",
      "MCC: 0.8044428         (Matthew correlation coefficient)\n",
      "Pre: 0.9005146         (Precision)\n",
      "Rec: 0.9562842         (Recall)\n",
      "G:   0.9279805         (G-measure)\n"
     ]
    }
   ],
   "source": [
    "Frame tr = train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch\");\n",
    "CForest rf = CForest.newModel()\n",
    "    .model.set(CTree.newCART().minCount.set(3))\n",
    "    .runs.set(100)\n",
    "    .seed.set(123L);\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", rf, 10, Accuracy.newMetric(true)).run().printFullContent();\n",
    "\n",
    "rf.fit(tr, \"Survived\");\n",
    "ClassifierResult fit = rf.predict(test);\n",
    "Confusion.from(tr.rvar(\"Survived\"), rf.predict(tr).firstClasses()).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we changed the classifierModel used by `CForest`. This is the same classifierModel used by default by random forest. We do this because we customized the classifierModel by changing the min count parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That had indeed some effect. However after submitting to competition we did not saw any improvement. We should look forward to engineer a little bit our features for further improvements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Feature engineering\n",
    "\n",
    "### 5.1 Title feature\n",
    "\n",
    "It is clear that we can't use directly the `\"Name\"` variable. This is due to the fact that names are almost unique, and that leads to a tiny generalization power. To understand that we should see that even if we learned that a passenger with a given name survived or not. We can't decide if another passenger survived, using only the name of the new passenger.\n",
    "\n",
    "Lets inspect some of the values from `\"Name\"` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VarNominal [name:\"Name\", rowCount:891]\n",
      " row                             value                            \n",
      "  [0]                  \"Braund, Mr. Owen Harris\"                  \n",
      "  [1]    \"Cumings, Mrs. John Bradley (Florence Briggs Thayer)\"    \n",
      "  [2]                  \"Heikkinen, Miss. Laina\"                   \n",
      "  [3]       \"Futrelle, Mrs. Jacques Heath (Lily May Peel)\"        \n",
      "  [4]                 \"Allen, Mr. William Henry\"                  \n",
      "  [5]                     \"Moran, Mr. James\"                      \n",
      "  [6]                  \"McCarthy, Mr. Timothy J\"                  \n",
      "  [7]              \"Palsson, Master. Gosta Leonard\"               \n",
      "  [8]     \"Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)\"     \n",
      "  [9]            \"Nasser, Mrs. Nicholas (Adele Achem)\"            \n",
      " [10]              \"Sandstrom, Miss. Marguerite Rut\"              \n",
      " [11]                 \"Bonnell, Miss. Elizabeth\"                  \n",
      " [12]              \"Saundercock, Mr. William Henry\"               \n",
      " [13]                \"Andersson, Mr. Anders Johan\"                \n",
      " [14]           \"Vestrom, Miss. Hulda Amanda Adolfina\"            \n",
      " [15]             \"Hewlett, Mrs. (Mary D Kingcome) \"              \n",
      " [16]                   \"Rice, Master. Eugene\"                    \n",
      " [17]               \"Williams, Mr. Charles Eugene\"                \n",
      " [18]  \"Vander Planke, Mrs. Julius (Emelia Maria Vandemoortele)\"  \n",
      " [19]                  \"Masselmani, Mrs. Fatima\"                  \n",
      " [20]                   \"Fynney, Mr. Joseph J\"                    \n",
      " [21]                   \"Beesley, Mr. Lawrence\"                   \n",
      " [22]               \"McGowan, Miss. Anna \"\"Annie\"\"\"               \n",
      " [23]               \"Sloper, Mr. William Thompson\"                \n",
      " [24]               \"Palsson, Miss. Torborg Danira\"               \n",
      " [25] \"Asplund, Mrs. Carl Oscar (Selma Augusta Emilia Johansson)\" \n",
      " [26]                  \"Emir, Mr. Farred Chehab\"                  \n",
      " [27]              \"Fortune, Mr. Charles Alexander\"               \n",
      " [28]              \"O'Dwyer, Miss. Ellen \"\"Nellie\"\"\"              \n",
      " [29]                    \"Todoroff, Mr. Lalio\"                    \n",
      " [30]                 \"Uruchurtu, Don. Manuel E\"                  \n",
      " [31]      \"Spencer, Mrs. William Augustus (Marie Eugenie)\"       \n",
      " [32]                 \"Glynn, Miss. Mary Agatha\"                  \n",
      " [33]                   \"Wheadon, Mr. Edward H\"                   \n",
      " [34]                  \"Meyer, Mr. Edgar Joseph\"                  \n",
      " [35]              \"Holverson, Mr. Alexander Oskar\"               \n",
      " [36]                     \"Mamee, Mr. Hanna\"                      \n",
      " [37]                 \"Cann, Mr. Ernest Charles\"                  \n",
      " [38]            \"Vander Planke, Miss. Augusta Maria\"             \n",
      " [39]                \"Nicola-Yarred, Miss. Jamila\"                \n",
      " [40]      \"Ahlin, Mrs. Johan (Johanna Persdotter Larsson)\"       \n",
      " [41] \"Turpin, Mrs. William John Robert (Dorothy Ann Wonnacott)\"  \n",
      " [42]                    \"Kraeff, Mr. Theodor\"                    \n",
      " [43]         \"Laroche, Miss. Simonne Marie Anne Andree\"          \n",
      " [44]               \"Devaney, Miss. Margaret Delia\"               \n",
      " [45]                 \"Rogers, Mr. William John\"                  \n",
      " [46]                     \"Lennon, Mr. Denis\"                     \n",
      " [47]                 \"O'Driscoll, Miss. Bridget\"                 \n",
      " [48]                    \"Samaan, Mr. Youssef\"                    \n",
      " [49]       \"Arnold-Franchi, Mrs. Josef (Josefine Franchi)\"       \n",
      " [50]                \"Panula, Master. Juha Niilo\"                 \n",
      " [51]               \"Nosworthy, Mr. Richard Cater\"                \n",
      " [52]         \"Harper, Mrs. Henry Sleeper (Myna Haxtun)\"          \n",
      " [53]    \"Faunthorpe, Mrs. Lizzie (Elizabeth Anne Wilkinson)\"     \n",
      " [54]              \"Ostby, Mr. Engelhart Cornelius\"               \n",
      " [55]                     \"Woolner, Mr. Hugh\"                     \n",
      " [56]                     \"Rugg, Miss. Emily\"                     \n",
      " [57]                    \"Novel, Mr. Mansouer\"                    \n",
      " [58]               \"West, Miss. Constance Mirium\"                \n",
      " [59]            \"Goodwin, Master. William Frederick\"             \n",
      " [60]                   \"Sirayanian, Mr. Orsen\"                   \n",
      " [61]                    \"Icard, Miss. Amelie\"                    \n",
      " [62]                \"Harris, Mr. Henry Birkhardt\"                \n",
      " [63]                   \"Skoog, Master. Harald\"                   \n",
      " [64]                   \"Stewart, Mr. Albert A\"                   \n",
      " [65]                 \"Moubarek, Master. Gerios\"                  \n",
      " [66]               \"Nye, Mrs. (Elizabeth Ramell)\"                \n",
      " [67]                 \"Crease, Mr. Ernest James\"                  \n",
      " [68]              \"Andersson, Miss. Erna Alexandra\"              \n",
      " [69]                     \"Kink, Mr. Vincenz\"                     \n",
      " [70]                \"Jenkin, Mr. Stephen Curnow\"                 \n",
      " [71]                \"Goodwin, Miss. Lillian Amy\"                 \n",
      " [72]                   \"Hood, Mr. Ambrose Jr\"                    \n",
      " [73]                \"Chronopoulos, Mr. Apostolos\"                \n",
      " [74]                       \"Bing, Mr. Lee\"                       \n",
      " [75]                  \"Moen, Mr. Sigurd Hansen\"                  \n",
      " [76]                     \"Staneff, Mr. Ivan\"                     \n",
      " [77]                 \"Moutal, Mr. Rahamin Haim\"                  \n",
      " [78]               \"Caldwell, Master. Alden Gates\"               \n",
      " ...                              ...                             \n",
      "[871]     \"Beckwith, Mrs. Richard Leonard (Sallie Monypeny)\"      \n",
      "[872]                 \"Carlsson, Mr. Frans Olof\"                  \n",
      "[873]                \"Vander Cruyssen, Mr. Victor\"                \n",
      "[874]           \"Abelson, Mrs. Samuel (Hannah Wizosky)\"           \n",
      "[875]            \"Najib, Miss. Adele Kiamie \"\"Jane\"\"\"             \n",
      "[876]               \"Gustafsson, Mr. Alfred Ossian\"               \n",
      "[877]                   \"Petroff, Mr. Nedelio\"                    \n",
      "[878]                    \"Laleff, Mr. Kristo\"                     \n",
      "[879]       \"Potter, Mrs. Thomas Jr (Lily Alexenia Wilson)\"       \n",
      "[880]       \"Shelley, Mrs. William (Imanita Parrish Hall)\"        \n",
      "[881]                    \"Markun, Mr. Johann\"                     \n",
      "[882]               \"Dahlberg, Miss. Gerda Ulrika\"                \n",
      "[883]               \"Banfield, Mr. Frederick James\"               \n",
      "[884]                  \"Sutehall, Mr. Henry Jr\"                   \n",
      "[885]           \"Rice, Mrs. William (Margaret Norton)\"            \n",
      "[886]                   \"Montvila, Rev. Juozas\"                   \n",
      "[887]               \"Graham, Miss. Margaret Edith\"                \n",
      "[888]        \"Johnston, Miss. Catherine Helen \"\"Carrie\"\"\"         \n",
      "[889]                   \"Behr, Mr. Karl Howell\"                   \n",
      "[890]                    \"Dooley, Mr. Patrick\"                    \n",
      "                                                                  \n"
     ]
    }
   ],
   "source": [
    "train.rvar(\"Name\").printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that the names contains title information of the individual. This is valuable, but how can we benefit from that? First of all see that the format of that string is clear: space + title + dot + space. We can try to model a regular expression or we can take a simpler, but manual path. Intuition tells us that there should not be too many keys.\n",
    "\n",
    "We build a set with known keys. After that we filter out names with known titles, and print first twenty of them. We see that we have already `\"Mrs\"` and `\"Mr\"`. Let's find others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"Heikkinen, Miss. Laina\"\n",
      "\"Palsson, Master. Gosta Leonard\"\n",
      "\"Sandstrom, Miss. Marguerite Rut\"\n",
      "\"Bonnell, Miss. Elizabeth\"\n",
      "\"Vestrom, Miss. Hulda Amanda Adolfina\"\n",
      "\"Rice, Master. Eugene\"\n",
      "\"McGowan, Miss. Anna \"\"Annie\"\"\"\n",
      "\"Palsson, Miss. Torborg Danira\"\n",
      "\"O'Dwyer, Miss. Ellen \"\"Nellie\"\"\"\n",
      "\"Uruchurtu, Don. Manuel E\"\n",
      "\"Glynn, Miss. Mary Agatha\"\n",
      "\"Vander Planke, Miss. Augusta Maria\"\n",
      "\"Nicola-Yarred, Miss. Jamila\"\n",
      "\"Laroche, Miss. Simonne Marie Anne Andree\"\n",
      "\"Devaney, Miss. Margaret Delia\"\n",
      "\"O'Driscoll, Miss. Bridget\"\n",
      "\"Panula, Master. Juha Niilo\"\n",
      "\"Rugg, Miss. Emily\"\n",
      "\"West, Miss. Constance Mirium\"\n",
      "\"Goodwin, Master. William Frederick\"\n"
     ]
    }
   ],
   "source": [
    "// build incrementally a set with known keys\n",
    "Set<String> keys = Set.of(\"Mrs\", \"Mr\");\n",
    "\n",
    "boolean inSet(String value, Set<String> set) {\n",
    "    for(String key : set) {\n",
    "        if(value.contains(key))\n",
    "            return false;\n",
    "    }\n",
    "    return true;\n",
    "}\n",
    "\n",
    "// filter out names with known keys\n",
    "// print first twenty to inspect and see other keys\n",
    "train.rvar(\"Name\").stream().mapToString().filter(txt -> inSet(txt,keys))\n",
    "    .limit(20).forEach(WS::println);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reduced our search and found other titles like `\"Miss\"`, `\"Master\"`. We arrive at the following set of keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "?  Mr Mrs Miss Master Don Rev Dr Mme Ms Major Lady Sir Mlle Col Capt Countess Jonkheer \n",
      "-  -- --- ---- ------ --- --- -- --- -- ----- ---- --- ---- --- ---- -------- -------- \n",
      "0 517 125 182    40    1   6  7   1  1    2    1    1   2    2   1      1        1     \n"
     ]
    }
   ],
   "source": [
    "Set<String> keys = Set.of(\"Mrs\", \"Mme\", \"Lady\", \"Countess\", \"Mr\", \"Sir\",\"Don\", \"Ms\", \"Miss\", \"Mlle\", \"Master\", \"Dr\",\"Col\", \"Major\", \"Jonkheer\", \"Capt\", \"Rev\");\n",
    "\n",
    "VarNominal title = train.rvar(\"Name\").stream().mapToString().map(txt -> {\n",
    "    for(String key : keys)\n",
    "        if(txt.contains(\" \" + key + \". \"))\n",
    "            return key;\n",
    "    return \"?\";\n",
    "}).collect(VarNominal.collector());\n",
    "DensityVector.fromLevelCounts(true, title).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that we exhausted training data. This is enough. It is possible that in test data to appear new titles. We will consider them missing values. That is why we return `\"?\"` when no matching is found. Another thing to notice is that some of the labels have few number of appearances. We will merge them in a greater category.\n",
    "\n",
    "Another useful feature built in `rapaio` is filters. There are two types of filters: variable filters and frame filters. The nice part of frame filters is that learning algorithms are able to use frame filters naturally, in order to make feature transformations on data. This kind of filters are called *input filters* from the learning algorithm perspective. It is important that you know that input filters transforms features before train phase and also on fit phase.\n",
    "\n",
    "We will build a learning filter to create a new feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TitleFilter implements Transform {\n",
    "    private final Map<String, List<String>> replaceMap = Map.of(\n",
    "            \"Mrs\", List.of(\"Mrs\", \"Mme\", \"Lady\", \"Countess\"),\n",
    "            \"Mr\", List.of(\"Mr\", \"Sir\", \"Don\", \"Ms\"),\n",
    "            \"Miss\", List.of(\"Miss\", \"Mlle\"),\n",
    "            \"Master\", List.of(\"Master\"),\n",
    "            \"Dr\", List.of(\"Dr\"),\n",
    "            \"Military\", List.of(\"Col\", \"Major\", \"Jonkheer\", \"Capt\"),\n",
    "            \"Rev\", List.of(\"Rev\")\n",
    "    );\n",
    "    private final Function<String, String> titleFun = txt -> {\n",
    "        for (Map.Entry<String, List<String>> e : replaceMap.entrySet()) {\n",
    "            for (int i = 0; i < e.getValue().size(); i++) {\n",
    "                if (txt.contains(\" \" + e.getValue().get(i) + \". \"))\n",
    "                    return e.getKey();\n",
    "            }\n",
    "        }\n",
    "        return \"?\";\n",
    "    };\n",
    "\n",
    "    @Override\n",
    "    public void fit(Frame df) {\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    public Frame apply(Frame df) {\n",
    "        VarNominal title = VarNominal.empty(0, new ArrayList<>(replaceMap.keySet())).name(\"Title\");\n",
    "        df.rvar(\"Name\").stream().mapToString().forEach(name -> title.addLabel(titleFun.apply(name)));\n",
    "        return df.bindVars(title);\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    public TitleFilter newInstance() {\n",
    "        return new TitleFilter();\n",
    "    }\n",
    "\n",
    "    public String[] varNames() {\n",
    "        return new String[0];\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try a new random forest on the reduced data set and also on title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CForest{model=CTree,poolSize=4,rowSampler=Bootstrap(p=1),runs=300,seed=124}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8293758 0.0459095 \n",
      "[1]   train Accuracy 0.9103379 0.0046455 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "Transform[] transform =  new Transform[] {\n",
    "    FrameCopy.transform(), \n",
    "    new TitleFilter()};\n",
    "\n",
    "Frame df_train = train.fapply(transform);\n",
    "Frame df_test = test.apply(transform);\n",
    "\n",
    "CForest rf = CForest.newModel().model.set(CTree.newCART().minCount.set(3))\n",
    "    .runs.set(300)\n",
    "    .poolSize.set(4)\n",
    "    .seed.set(124L);\n",
    "\n",
    "ClassifierEvaluation.cv(df_train.mapVars(\"Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch,Title\"), \"Survived\", rf, 10, Accuracy.newMetric(true)).run().printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that looks slightly better than our best classifierModel. We submit that to kaggle to see if there is any improvement.\n",
    "\n",
    "![Progress after incorporating title name into input features](images/titanic-rf1-model.png)\n",
    "\n",
    "### 5.2 Other features\n",
    "\n",
    "There are various authors which published their work on solving this kaggle competition. Most interesting part of their work is the feature engineering section. I developed here some ideas in order to show how one can do this with the library.\n",
    "\n",
    "#### 5.2.1 Family size\n",
    "\n",
    "Using directly `\"SibSp\"` and `\"Parch\"` fields yields no value for a random forest classifierModel. Studying this two features it looks like those values can be both combined into a single one by summation. This would give us a family size estimator.\n",
    "\n",
    "In order to have an idea of the performance of this new estimator I used a ChiSquare independence test. The idea is to study if those features taken separately worth less than combined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> ChiSqIndependence\n",
      "Pearson's Chi-squared test with Yates' continuity correction\n",
      "\n",
      "X-squared = 31.9279202, df = 6, p-value = 0.0000168\n",
      "Observed data:\n",
      "        1   0  3  4  2 5 8 total \n",
      "    0  97 398 12 15 15 5 7  549  \n",
      "    1 112 210  4  3 13 0 0  342  \n",
      "total 209 608 16 18 28 5 7  891  \n",
      "\n",
      "Expected data:\n",
      "                1           0          3          4          2         5         8 total \n",
      "    0 128.7777778 374.6262626  9.8585859 11.0909091 17.2525253 3.0808081 4.3131313  549  \n",
      "    1  80.2222222 233.3737374  6.1414141  6.9090909 10.7474747 1.9191919 2.6868687  342  \n",
      "total 209         608         16         18         28         5         7          891  \n",
      "> ChiSqIndependence\n",
      "Pearson's Chi-squared test with Yates' continuity correction\n",
      "\n",
      "X-squared = 23.3892671, df = 6, p-value = 0.0006761\n",
      "Observed data:\n",
      "        0   1  2 5 3 4 6 total \n",
      "    0 445  53 40 4 2 4 1  549  \n",
      "    1 233  65 40 1 3 0 0  342  \n",
      "total 678 118 80 5 5 4 1  891  \n",
      "\n",
      "Expected data:\n",
      "                0           1          2         5         3         4         6 total \n",
      "    0 417.7575758  72.7070707 49.2929293 3.0808081 3.0808081 2.4646465 0.6161616  549  \n",
      "    1 260.2424242  45.2929293 30.7070707 1.9191919 1.9191919 1.5353535 0.3838384  342  \n",
      "total 678         118         80         5         5         4         1          891  \n",
      "> ChiSqIndependence\n",
      "Pearson's Chi-squared test with Yates' continuity correction\n",
      "\n",
      "X-squared = 72.6627665, df = 8, p-value =   1.45e-12\n",
      "Observed data:\n",
      "        2   1  5   3  7  6  4 8 11 total \n",
      "    0  72 374 12  43  8 19  8 6 7   549  \n",
      "    1  89 163  3  59  4  3 21 0 0   342  \n",
      "total 161 537 15 102 12 22 29 6 7   891  \n",
      "\n",
      "Expected data:\n",
      "                2           1          5           3          7          6          4         8 \n",
      "    0  99.2020202 330.8787879  9.2424242  62.8484848  7.3939394 13.5555556 17.8686869 3.6969697 \n",
      "    1  61.7979798 206.1212121  5.7575758  39.1515152  4.6060606  8.4444444 11.1313131 2.3030303 \n",
      "total 161         537         15         102         12         22         29         6         \n",
      "\n",
      "       11 total \n",
      "4.3131313  549  \n",
      "2.6868687  342  \n",
      "7          891  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "// convert sibsp and parch to nominal types to be able to use a chi-square test\n",
    "Var sibsp = VarNominal.from(train.rowCount(), row -> train.getLabel(row, \"SibSp\"));\n",
    "Var parch = VarNominal.from(train.rowCount(), row -> train.getLabel(row, \"Parch\"));\n",
    "\n",
    "// test individually each feature\n",
    "ChiSqIndependence.from(train.rvar(\"Survived\"), sibsp, true).printSummary();\n",
    "ChiSqIndependence.from(train.rvar(\"Survived\"), parch, true).printSummary();\n",
    "\n",
    "// build a combined feature by summation, as nominal\n",
    "VarNominal familySize = VarNominal.from(train.rowCount(),\n",
    "row -> \"\" + (1 + train.getInt(row, \"SibSp\") + train.getInt(row, \"Parch\")));\n",
    "\n",
    "// run the chi-square test on sumation\n",
    "ChiSqIndependence.from(train.rvar(\"Survived\"), familySize, true).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How we can interpret the result? The test says that each feature brings value separately. The *p-value* associated with both test could be considered significant. That means there are string evidence that those features are not independent of target class. As a conclusion, those features are useful. The last test is made for their summation. It looks like he test is more significant than the previous two. As a consequence we can use the summation instead of those two values taken independently.\n",
    "\n",
    "### 5.3 Cabin and Ticket\n",
    "\n",
    "It seems that cabin and ticket denominations are not useful as they are. There are some various reasons why not to do so. First of all they have many missing values. But a stringer reason is that both have too many levels to contain solid generalization base for learning.\n",
    "\n",
    "If we take only the first letter from each of those two fields, more generalization can happen. This is probably due to the fact that perhaps there is some localization information encoded in those. Perhaps information about the deck, the comfort level, auxiliary functions is encoded in there. As a conclusion it worth a try so we proceed with this thing.\n",
    "\n",
    "To combine all those things we can do it in a single filter or into many filters applied on data. I chose to do a single custom filter to solve all those problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomFilter implements Transform {\n",
    "\n",
    "    private final Map<String, List<String>> replaceMap = Map.of(\n",
    "            \"Mrs\", List.of(\"Mrs\", \"Mme\", \"Lady\", \"Countess\"),\n",
    "            \"Mr\", List.of(\"Mr\", \"Sir\", \"Don\", \"Ms\"),\n",
    "            \"Miss\", List.of(\"Miss\", \"Mlle\"),\n",
    "            \"Master\", List.of(\"Master\"),\n",
    "            \"Dr\", List.of(\"Dr\"),\n",
    "            \"Military\", List.of(\"Col\", \"Major\", \"Jonkheer\", \"Capt\"),\n",
    "            \"Rev\", List.of(\"Rev\")\n",
    "    );\n",
    "    private final Function<String, String> titleFun = txt -> {\n",
    "        for (var e : replaceMap.entrySet()) {\n",
    "            for (int i = 0; i < e.getValue().size(); i++) {\n",
    "                if (txt.contains(\" \" + e.getValue().get(i) + \". \"))\n",
    "                    return e.getKey();\n",
    "            }\n",
    "        }\n",
    "        return \"?\";\n",
    "    };\n",
    "\n",
    "    public void fit(Frame df) {\n",
    "    }\n",
    "\n",
    "    public Frame apply(Frame df) {\n",
    "        VarNominal title = VarNominal.empty(0, new ArrayList<>(replaceMap.keySet())).name(\"Title\");\n",
    "        df.rvar(\"Name\").stream().mapToString().forEach(name -> title.addLabel(titleFun.apply(name)));\n",
    "\n",
    "        Var famSize = VarDouble.from(df.rowCount(), r -> 1.0 + df.getInt(r, \"SibSp\") + df.getInt(r, \"Parch\")).name(\"FamilySize\");\n",
    "        Var ticket = VarNominal.from(df.rowCount(), r -> df.getLabel(r, \"Ticket\").substring(0, 1).toUpperCase()).name(\"Ticket\");\n",
    "        Var cabin = VarNominal.from(df.rowCount(), r -> df.getLabel(r, \"Cabin\").substring(0, 1).toUpperCase()).name(\"Cabin\");\n",
    "\n",
    "        return df.removeVars(\"Ticket,Cabin\").bindVars(famSize, ticket, cabin, title).copy();\n",
    "    }\n",
    "\n",
    "    public CustomFilter newInstance() {\n",
    "        return new CustomFilter();\n",
    "    }\n",
    "\n",
    "    public String[] varNames() {\n",
    "        return new String[0];\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Another try with random forests\n",
    "\n",
    "So we have some new features and we look to learn from them. We can use a previous classifierModel like random forests to test it before submit.\n",
    "\n",
    "But we know that we are in danger to overfit is rf. One idea is to transform numeric features into nominal ones by a process named discretization. For this purpose we use a filter from the library called `FQuantileDiscrete`. This filter computes a given number of quantile intervals and put labels according with those intervals on numerical values. Let's see how we proceed and how the data looks like:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 183/891\n",
      "* varCount: 12\n",
      "* varNames: \n",
      "\n",
      " 0.   Survived : nom |  4.      SibSp : nom |  8. FamilySize : nom | \n",
      " 1.     Pclass : nom |  5.      Parch : nom |  9.     Ticket : nom | \n",
      " 2.        Sex : nom |  6.       Fare : nom | 10.      Cabin : nom | \n",
      " 3.        Age : nom |  7.   Embarked : nom | 11.      Title : nom | \n",
      "\n",
      "* summary: \n",
      " Survived [nom]  Pclass [nom]      Sex [nom]       Age [nom]    SibSp [nom]    Parch [nom] \n",
      "      0 :   549     3 :   491   male :   577 31.8~36 :    91 -Inf~0 :   608 -Inf~0 :   678 \n",
      "      1 :   342     1 :   216 female :   314   14~19 :    87  0~Inf :   283  0~Inf :   213 \n",
      "                    2 :   184                  41~50 :    78                               \n",
      "                                             -Inf~14 :    77                               \n",
      "                                             (Other) :   381                               \n",
      "                                                 NAs :   177                               \n",
      "                                                                                           \n",
      "\n",
      "           Fare [nom]  Embarked [nom]  FamilySize [nom]    Ticket [nom]     Cabin [nom]     Title \n",
      "   7.854~8.05 :   106       S :   644    -Inf~1 :   537       3 :   301       C :    59      Mr : \n",
      "    -Inf~7.55 :    92       C :   168       1~2 :   161       2 :   183       B :    47    Miss : \n",
      "    27~39.688 :    91       Q :    77       2~3 :   102       1 :   146       D :    33     Mrs : \n",
      "39.688~77.958 :    89                     3~Inf :    91       P :    65       E :    32  Master : \n",
      "    21.679~27 :    89                                         S :    65 (Other) :    33      Dr : \n",
      "      (Other) :   424     NAs :     2                   (Other) :   131     NAs :   687 (Other) : \n",
      "                                                                                                  \n",
      "\n",
      "[nom] \n",
      "  520 \n",
      "  184 \n",
      "  128 \n",
      "   40 \n",
      "    7 \n",
      "   12 \n",
      "      \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "var transform = new Transform[] {\n",
    "    FrameCopy.transform(),\n",
    "    new CustomFilter(),\n",
    "    QuantileTransform.split(VarRange.of(\"Age\"), 10),\n",
    "    QuantileTransform.split(VarRange.of(\"Fare\"), 10),\n",
    "    QuantileTransform.split(VarRange.of(\"SibSp\"), 3),\n",
    "    QuantileTransform.split(VarRange.of(\"Parch\"), 3),\n",
    "    QuantileTransform.split(VarRange.of(\"FamilySize\"), 8),\n",
    "    RemoveVars.remove(VarRange.of(\"PassengerId,Name\"))};\n",
    "\n",
    "// print a summary of the transformed data\n",
    "train.fapply(transform).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that `\"Age\"` values are now intervals and still $177$ missing values.\n",
    "\n",
    "The values chosen for quantile numbers is more or less arbitrary. There is no *good* numbers in general, only for some specific purposes.\n",
    "\n",
    "As promised, we will give a try to another random forest to see if it can better generalize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "CForest{rowSampler=Bootstrap(p=1),runs=200,seed=123}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8182272 0.0423661 \n",
      "[1]   train Accuracy 0.908342  0.0055043 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "CForest rf3 = CForest.newModel().runs.set(200).seed.set(123L);\n",
    "\n",
    "tr = train.apply(transform);\n",
    "\n",
    "rf3.fit(tr, \"Survived\");\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", rf3, 10, Accuracy.newMetric()).run().printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I tried some ideas to make the forest to generalize better.\n",
    "\n",
    "* Smaller bootstrap percentage - this could lead to increased independence of trees\n",
    "* Use `GainRatio` as purity function because sometimes is more conservative\n",
    "* Use `MinGain` to avoid growing trees to have many leaves with a single instance\n",
    "* Use `mCols=4`, number of variables used for testing - more than default value, to improve the quality of each tree\n",
    "\n",
    "These are the results. At a first look might seem like an astonishing result. But we know that the irreducible error for this data set is high and is close to $0.2$. It seems obvious that we failed to reduce the variance and we still overfit a lot using this construct. Since this is a tutorial I will not insist on improving this model, but I think that even if it would be improved, the gain would be very small. Perhaps another approach would be better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. SVM model\n",
    "\n",
    "SVM (Support Vector Machines) is a nice framework to test new ideas for various types of problems. The power of SMVs comes from their kernels. A kernel is basically a transformation of the original space generated by the input features into another space, often with more dimensions. It's like a feature engineering in a singe function.\n",
    "\n",
    "But SVMs have a practical problem. The features needs to be numeric and does not allows missing values. This is not a constrain on the algorithm itself. At any moment one can build a kernel for nominal features. But the implemented ones allows only numeric non missing values and is much simpler to shape our data into this format.\n",
    "\n",
    "How can we do that?\n",
    "\n",
    "### 6.1 Data preparation\n",
    "\n",
    "We can use a filter to impute data for missing values. The filter we use is an imputation with a classifierModel of imputation with a regression. The logic is the following: train a classifierModel from a specified set of input features to predict the field with missing values. The data set inside the filter is filtered to contain only instances with non-missing target values.\n",
    "\n",
    "After we impute the missing values we encode nominal features into numeric features. We can accomplish this task using, again, another filter for this purpose. The name of this filter is `FOneHotEncoding`. What it does is to create a number of numeric feature for level of the nominal variable. Than the values on these numeric variables receives the value of the indicator function. We have $1$ if the level equals the numeric variable's name, $0$ otherwise.\n",
    "\n",
    "After we have numerical variables, is better to make all the variables to be in the same range. This is not a requirement for SMVs in general. The meaning is to give same weight to all the involved variables. As a side effect it makes the algorithm to run faster. This is due to the fact that the convex optimization problem has smaller chances to have a close-to-flat big surfaces.\n",
    "\n",
    "Finally, we will remove the not used variables from the frame in order to be prepared for learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame Summary\n",
      "=============\n",
      "* rowCount: 891\n",
      "* complete: 891/891\n",
      "* varCount: 44\n",
      "* varNames: \n",
      "\n",
      " 0.       Survived : nom | 11.     FamilySize : dbl | 22.       Ticket.F : bin | 33.        Cabin.B : bin | \n",
      " 1.       Pclass.3 : bin | 12.       Ticket.A : bin | 23.       Ticket.L : bin | 34.        Cabin.F : bin | \n",
      " 2.       Pclass.1 : bin | 13.       Ticket.P : bin | 24.       Ticket.9 : bin | 35.        Cabin.T : bin | \n",
      " 3.       Pclass.2 : bin | 14.       Ticket.S : bin | 25.       Ticket.6 : bin | 36.        Title.? : bin | \n",
      " 4.       Sex.male : bin | 15.       Ticket.1 : bin | 26.       Ticket.5 : bin | 37.      Title.Rev : bin | \n",
      " 5.     Sex.female : bin | 16.       Ticket.3 : bin | 27.       Ticket.8 : bin | 38.     Title.Miss : bin | \n",
      " 6.            Age : dbl | 17.       Ticket.2 : bin | 28.        Cabin.C : bin | 39. Title.Military : bin | \n",
      " 7.           Fare : dbl | 18.       Ticket.C : bin | 29.        Cabin.E : bin | 40.       Title.Dr : bin | \n",
      " 8.     Embarked.S : bin | 19.       Ticket.7 : bin | 30.        Cabin.G : bin | 41.       Title.Mr : bin | \n",
      " 9.     Embarked.C : bin | 20.       Ticket.W : bin | 31.        Cabin.D : bin | 42.   Title.Master : bin | \n",
      "10.     Embarked.Q : bin | 21.       Ticket.4 : bin | 32.        Cabin.A : bin | 43.      Title.Mrs : bin | \n",
      "\n",
      "* summary: \n",
      " Survived [nom]  Pclass.3 [bin]  Pclass.1 [bin]  Pclass.2 [bin]  Sex.male [bin]  Sex.female [bin] \n",
      "      0 :   549 0 :         400 0 :         675 0 :         707 0 :         314 0 :           577 \n",
      "      1 :   342 1 :         491 1 :         216 1 :         184 1 :         577 1 :           314 \n",
      "                NAs :         0 NAs :         0 NAs :         0 NAs :         0 NAs :           0 \n",
      "                                                                                                  \n",
      "                                                                                                  \n",
      "                                                                                                  \n",
      "                                                                                                  \n",
      "\n",
      "      Age [dbl]           Fare [dbl]       Embarked.S [bin]  Embarked.C [bin]  Embarked.Q [bin] \n",
      "   Min. : -2.1237953    Min. : -0.6480577 0 :           246 0 :           722 0 :           814 \n",
      "1st Qu. : -0.6285973 1st Qu. : -0.4888737 1 :           645 1 :           169 1 :            77 \n",
      " Median : -0.0837001  Median : -0.3571902 NAs :           0 NAs :           0 NAs :           0 \n",
      "   Mean :  0.0000000    Mean :  0.0000000                                                       \n",
      "2nd Qu. :  0.6065031 2nd Qu. : -0.0242327                                                       \n",
      "   Max. :  3.6579276    Max. :  9.6617401                                                       \n",
      "                                                                                                \n",
      "\n",
      " FamilySize [dbl]       Ticket.A [bin]  Ticket.P [bin]  Ticket.S [bin]  Ticket.1 [bin]  Ticket.3 \n",
      "     Min. : -0.5606599 0 :         862 0 :         826 0 :         826 0 :         745 0 :       \n",
      "  1st Qu. : -0.5606599 1 :          29 1 :          65 1 :          65 1 :         146 1 :       \n",
      "   Median : -0.5606599 NAs :         0 NAs :         0 NAs :         0 NAs :         0 NAs :     \n",
      "     Mean :  0.0000000                                                                           \n",
      "  2nd Qu. :  0.0591267                                                                           \n",
      "     Max. :  5.6372062                                                                           \n",
      "                                                                                                 \n",
      "\n",
      "[bin]  Ticket.2 [bin]  Ticket.C [bin]  Ticket.7 [bin]  Ticket.W [bin]  Ticket.4 [bin]  Ticket.F [bin] \n",
      "  590 0 :         708 0 :         844 0 :         882 0 :         878 0 :         881 0 :         884 \n",
      "  301 1 :         183 1 :          47 1 :           9 1 :          13 1 :          10 1 :           7 \n",
      "    0 NAs :         0 NAs :         0 NAs :         0 NAs :         0 NAs :         0 NAs :         0 \n",
      "                                                                                                      \n",
      "                                                                                                      \n",
      "                                                                                                      \n",
      "                                                                                                      \n",
      "\n",
      " Ticket.L [bin]  Ticket.9 [bin]  Ticket.6 [bin]  Ticket.5 [bin]  Ticket.8 [bin]  Cabin.C [bin] \n",
      "0 :         887 0 :         890 0 :         885 0 :         888 0 :         889 0 :        814 \n",
      "1 :           4 1 :           1 1 :           6 1 :           3 1 :           2 1 :         77 \n",
      "NAs :         0 NAs :         0 NAs :         0 NAs :         0 NAs :         0 NAs :        0 \n",
      "                                                                                               \n",
      "                                                                                               \n",
      "                                                                                               \n",
      "                                                                                               \n",
      "\n",
      " Cabin.E [bin]  Cabin.G [bin]  Cabin.D [bin]  Cabin.A [bin]  Cabin.B [bin]  Cabin.F [bin]  Cabin.T \n",
      "0 :        729 0 :        845 0 :        815 0 :        867 0 :        835 0 :        442 0 :      \n",
      "1 :        162 1 :         46 1 :         76 1 :         24 1 :         56 1 :        449 1 :      \n",
      "NAs :        0 NAs :        0 NAs :        0 NAs :        0 NAs :        0 NAs :        0 NAs :    \n",
      "                                                                                                   \n",
      "                                                                                                   \n",
      "                                                                                                   \n",
      "                                                                                                   \n",
      "\n",
      "[bin]  Title.? [bin]  Title.Rev [bin]  Title.Miss [bin]  Title.Military [bin]  Title.Dr [bin] \n",
      "  890 0 :        891 0 :          885 0 :           707 0 :               885 0 :         884 \n",
      "    1 1 :          0 1 :            6 1 :           184 1 :                 6 1 :           7 \n",
      "    0 NAs :        0 NAs :          0 NAs :           0 NAs :               0 NAs :         0 \n",
      "                                                                                              \n",
      "                                                                                              \n",
      "                                                                                              \n",
      "                                                                                              \n",
      "\n",
      " Title.Mr [bin]  Title.Master [bin]  Title.Mrs [bin] \n",
      "0 :         371 0 :             851 0 :          763 \n",
      "1 :         520 1 :              40 1 :          128 \n",
      "NAs :         0 NAs :             0 NAs :          0 \n",
      "                                                     \n",
      "                                                     \n",
      "                                                     \n",
      "                                                     \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "var transform = new Transform[] {\n",
    "    FrameCopy.transform(),\n",
    "    new CustomFilter(),\n",
    "    ImputeRegression.of(RForest.newRF().runs.set(100), VarRange.of(\"Age,Pclass,Embarked,Sex,Fare,Title\"), \"Age\"),\n",
    "    ImputeClassifier.of(CForest.newModel().runs.set(10), VarRange.of(\"Embarked,Age,Pclass,Sex,Title\"), \"Embarked\"),\n",
    "    ImputeClassifier.of(CForest.newModel().runs.set(100), VarRange.of(\"Age,Pclass,Embarked,Sex,Fare,Ticket\"),\"Ticket\"),\n",
    "    ImputeClassifier.of(CForest.newModel().runs.set(100), VarRange.of(\"Age,Pclass,Embarked,Sex,Fare,Cabin\"), \"Cabin\"),\n",
    "    OneHotEncoding.on(\"Title\"),\n",
    "    OneHotEncoding.on(false, false, \"Embarked,Sex,Ticket,Cabin,Pclass\"),\n",
    "    StandardScaler.on(VarRange.onlyTypes(VarType.DOUBLE)),\n",
    "    RemoveVars.remove(VarRange.of(\"PassengerId,Name,SibSp,Parch\"))};\n",
    "train.fapply(transform).printSummary();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot of content. Notice that we have numerical variables for each ticket first letter, title, cabin first letter, etc.\n",
    "## Train a polynomial SVM\n",
    "A linear kernel is a polynomial kernel with degree 1. We let the $C$ parameter to the default value which is $1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "SVMClassifier{kernel=PolyKernel(exp=1,bias=1,slope=1),probability=true}\n",
      "Raw scores:\n",
      "===========\n",
      "     dataset round fold Accuracy       dataset round fold Accuracy       dataset round fold Accuracy  \n",
      " [0]    test     0    0 0.8333333  [7]    test     0    7 0.8539326 [14]   train     0    4 0.8528678 \n",
      " [1]    test     0    1 0.8764045  [8]    test     0    8 0.7640449 [15]   train     0    5 0.8516209 \n",
      " [2]    test     0    2 0.7977528  [9]    test     0    9 0.8651685 [16]   train     0    6 0.8329177 \n",
      " [3]    test     0    3 0.8089888 [10]   train     0    0 0.8476904 [17]   train     0    7 0.8441397 \n",
      " [4]    test     0    4 0.7977528 [11]   train     0    1 0.8478803 [18]   train     0    8 0.8553616 \n",
      " [5]    test     0    5 0.8426966 [12]   train     0    2 0.8491272 [19]   train     0    9 0.8341646 \n",
      " [6]    test     0    6 0.8876404 [13]   train     0    3 0.840399  \n",
      "\n",
      "Round scores:\n",
      "=============\n",
      "    dataset round Accuracy_mean Accuracy_std \n",
      "[0]    test     0   0.8327715    0.0377521   \n",
      "[1]   train     0   0.8456169    0.0072606   \n",
      "\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8327715 0.0397942 \n",
      "[1]   train Accuracy 0.8456169 0.0076533 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "var model = new SvmClassifier()\n",
    "    .c.set(1.0)\n",
    "    .kernel.set(new PolyKernel(1))\n",
    "    .probability.set(true);\n",
    "\n",
    "var tr = train.apply(transform);\n",
    "model.fit(tr, \"Survived\");\n",
    "\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", model, 10, Accuracy.newMetric()).run().printFullContent();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model:\n",
      "SVMClassifier{cost=0.01,kernel=PolyKernel(exp=3,bias=1,slope=1),probability=true}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8404798 0.055386  \n",
      "[1]   train Accuracy 0.9040102 0.0029642 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "var model = new SvmClassifier()\n",
    "    .c.set(0.01)\n",
    "    .kernel.set(new PolyKernel(3))\n",
    "    .probability.set(true);\n",
    "\n",
    "var tr = train.apply(transform);\n",
    "model.fit(tr, \"Survived\");\n",
    "\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", model, 20, Accuracy.newMetric()).run().printContent();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are not promising. This is better than random but it is not enough for our purpose. There are some explanations for this result. First one could be that if the space would be linear, than the original feature space would be the same as transformed. This means that a classifierModel as random forest would work well if the linear svm would have worked. This might not be true in general, but in this case it looks like a good explanation. We need to be more flexible.\n",
    "\n",
    "To increase the flexibility of the model and to allow features to interact with one another we change the degree of the polynomial kernel. This time we will use `degree=3`. Also, we use $C=0.0001$ to allow for some errors. This parameter is the factor of the slack regularization constraints of the SVM optimization problem. The bigger the value the more is the penalty for wrong decisions. If the space would be linear separable than one can theoretically set this value as high as possible. But we know it is not. Also we know that we have plenty of irreducible error. As a consequence, it looks like we should decrease the value of this parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Confusion matrix\n",
      " - Frequency table\n",
      "Ac\\Pr |    0    1 | total \n",
      "----- |    -    - | ----- \n",
      "    0 | >523   26 |   549 \n",
      "    1 |   60 >282 |   342 \n",
      "----- |    -    - | ----- \n",
      "total |  583  308 |   891 \n",
      " - Probability table\n",
      "Ac\\Pr |      0      1 | total \n",
      "----- |      -      - | ----- \n",
      "    0 | >0.587  0.029 | 0.616 \n",
      "    1 |  0.067 >0.316 | 0.384 \n",
      "----- |      -      - | ----- \n",
      "total |  0.654  0.346 | 1.000 \n",
      "\n",
      "\n",
      "Complete cases 891 from 891\n",
      "Acc: 0.9034792         (Accuracy )\n",
      "F1:  0.9240283         (F1 score / F-measure)\n",
      "MCC: 0.7947377         (Matthew correlation coefficient)\n",
      "Pre: 0.897084         (Precision)\n",
      "Rec: 0.9526412         (Recall)\n",
      "G:   0.9244453         (G-measure)\n",
      "Model:\n",
      "SVMClassifier{cost=0.01,kernel=PolyKernel(exp=3,bias=1,slope=1),probability=true}\n",
      "CV score in training data\n",
      "=========================\n",
      "    dataset  metric    mean       std    \n",
      "[0]    test Accuracy 0.8474242 0.0596819 \n",
      "[1]   train Accuracy 0.9041292 0.0032511 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "var model = new SvmClassifier()\n",
    "    .c.set(.01)\n",
    "    .kernel.set(new PolyKernel(3))\n",
    "    .probability.set(true);\n",
    "\n",
    "var tr = train.apply(transform);\n",
    "model.fit(tr, \"Survived\");\n",
    "\n",
    "Confusion.from(tr.rvar(\"Survived\"), model.predict(tr).firstClasses()).printSummary();\n",
    "\n",
    "ClassifierEvaluation.cv(tr, \"Survived\", model, 20, Accuracy.newMetric()).run().printContent()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time the results are promising. We achieved a training error which is not close to zero and the cross validation errors are close to our desired results. We definitely should try this classifierModel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM1](images/titanic-svm1-submit.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a better score also on public leader board. Which is very fine. Usually in this competition a score in $0.75-0.78$ is fine and one in $0.78-0.81$ is excellent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Stacking classifiers\n",
    "\n",
    "Using random forests or SVMs did not provided us with a result over $0.8$. We have two very different types of models which performed well. For the sole purpose of prediction we use a nice ensemble technique which often provides good prediction performance gain. This technique is called stacking.\n",
    "\n",
    "The idea behind stacking is that one can explore the space of the solutions with different approaches. Each approach (or statistical model) is basically an interpretation of the solution. But often times a proper interpretation is really hard to find. Each interpretation of the solution can have good points and weak points. The idea is to blend those interpretations into a single one in a way that we try somehow to keep what is string from each individual classifierModel.\n",
    "\n",
    "A stacking classifierModel take some base learners and train them on training data. The results of the base learners are used as input for a stacking learner. This stacking learner is trained on the output of base learners and target variable and is finally used for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Confusion matrix\n",
      " - Frequency table\n",
      "Ac\\Pr |    0    1 | total \n",
      "----- |    -    - | ----- \n",
      "    0 | >522   27 |   549 \n",
      "    1 |   83 >259 |   342 \n",
      "----- |    -    - | ----- \n",
      "total |  605  286 |   891 \n",
      " - Probability table\n",
      "Ac\\Pr |      0      1 | total \n",
      "----- |      -      - | ----- \n",
      "    0 | >0.586  0.030 | 0.616 \n",
      "    1 |  0.093 >0.291 | 0.384 \n",
      "----- |      -      - | ----- \n",
      "total |  0.679  0.321 | 1.000 \n",
      "\n",
      "\n",
      "Complete cases 891 from 891\n",
      "Acc: 0.8765432         (Accuracy )\n",
      "F1:  0.9046794         (F1 score / F-measure)\n",
      "MCC: 0.7376516         (Matthew correlation coefficient)\n",
      "Pre: 0.8628099         (Precision)\n",
      "Rec: 0.9508197         (Recall)\n",
      "G:   0.9057465         (G-measure)\n"
     ]
    }
   ],
   "source": [
    "var model = CStacking.newModel()\n",
    "    .learners.add(new SvmClassifier().c.set(1.0).kernel.set(new CauchyKernel(20)).probability.set(true))\n",
    "    .learners.add(CForest.newModel()\n",
    "                  .rowSampler.set(RowSampler.bootstrap(0.07))\n",
    "                  .model.set(CTree.newCART().varSelector.set(VarSelector.fixed(4)).purity.set(Purity.GainRatio).minGain.set(0.001))\n",
    "                  .runs.set(200))\n",
    "    .stackModel.set(CForest.newModel().rowSampler.set(RowSampler.bootstrap(0.3)).runs.set(200)\n",
    "                    .model.set(CTree.newCART().purity.set(Purity.GainRatio).minGain.set(0.05)));\n",
    "\n",
    "var tr = train.apply(transform);\n",
    "model.fit(tr, \"Survived\");\n",
    "\n",
    "Confusion.from(tr.rvar(\"Survived\"), model.predict(tr).firstClasses()).printSummary();\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually one uses a binary logistic regression model but it provided weak results. What looked much better is another random forrest classifierModel. However the stacking model uses a big value for minimum gain parameter because we want to act as an draft average over the results.\n",
    "\n",
    "Again, the results are promising. The space between training error and cross validation error is smaller and our expectations grows.\n",
    "\n",
    "![Stacking with a random forest](images/titanic-stacking1-submit.png)\n",
    "\n",
    "Finally our target performance was achieved!!\n",
    "\n",
    "**Note:**\n",
    "\n",
    "The general advice in real life is to not fight for each piece of performance measure. It really depends on the question one wants to answer. Often measures like ROC or partial ROC are much better than error frequency. We fixed this milestone because we know it is possible and because it looks like a psychological difficulty. (that $0.799$ is outrageous)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "java (Rapaio/j!)",
   "language": "java",
   "name": "jbang-rapaio"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "java",
   "nbconvert_exporter": "script",
   "pygments_lexer": "java",
   "version": "21.0.2+13-LTS"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}