W03_Support_Vector_Machines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A *Support Vector Machine* (SVM) is a commonly used method for linear and non-linear classification, doing regression analysis or identifying outsiders. Because they are used so much, Scikit also has standard tooling for it in-house."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we start importing the most important libraries. In this assignment we use the well-known (and often mentioned) <a href=\"https://en.wikipedia.org/wiki/Iris_flower_data_set\"> iris dataset </a>. This is included in `sklearn.datasets` by default, so we have to import them as well. We also use a number of help functions here, but because there are many this time, we moved them to a separate file (`Helpers.py`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets\n",
    "from sklearn.svm import SVC\n",
    "from Helpers import plot_svc_decision_boundary "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we start by making a visual representation of the dataset. This time, however, we also do something more, to clarify the idea of SVM a bit: In addition to plotting the data points themselves, we also draw three *classifiers* in the same plot. These *classifiers* have already been given, as well as the loading of the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data that is loaded consists of a *dictionary* with keys `data` and` target`. This corresponds to the $ X $ matrix and the $ y $ vector, respectively. The matrix $ X $ contains four columns:\n",
    "\n",
    "- index 0: length of the sepals in centimeters\n",
    "- index 1: sepal width in centimeters\n",
    "- index 2: length of the petal in centimeters\n",
    "- index 3: width of the petal in centimeters\n",
    "\n",
    "In the rest of this Notebook, we want to look at the ratio of the width and length of the petal to determine the type of iris. Initially, we only look at the flowers that are a *Setosa* ($ y = 0 $) or a *Versicolor* ($ y = 1 $), and the flowers that are *not*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follow the instructions in the cell. Prepare the $X$ matrix and the $y$ vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # petal length, petal width\n",
    "y = iris[\"target\"]\n",
    "print (iris['data'][1:3, :])\n",
    "print (y.shape)\n",
    "\n",
    "# create a mask for the type of flowers. See example code below\n",
    "\n",
    "y_demo = np.array([1,2,3,0,0,1])\n",
    "X_demo = np.array( ([1,2],[1,3],[0,1],[0,4], [4,2], [5,2]))\n",
    "\n",
    "mask = (y_demo==0) | (y_demo==1)\n",
    "print (X_demo[mask])\n",
    "print (y_demo[mask])\n",
    "\n",
    "\n",
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot data and *classifiers*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create randomly three classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = np.linspace(0, 5.5, 200)\n",
    "\n",
    "# y-values different classifiers\n",
    "pred_1 = 5*x0 - 20        #C1 - make this green ('g-')\n",
    "pred_2 = x0 - 1.8         #C2 - make this cyan ('c-')\n",
    "pred_3 = 0.1 * x0 + 0.5   #C3 - make this red ('r-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then create a scatter plot showing the different types of flowers and add the classifiers to it. Use the `plot` method from matplotlib to draw these lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the green classifier (`C1`) is so bad that it is not even able to distinguish the Versicolor from the Setosa. The cyan (`C2`) and the red (` 3`) classifiers do, but offer very little space between the border cases: the distance between these border cases and the *boundary* is minimal. These two classifiers are unlikely to identify new data that is close to the decision boundary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An SVM creates a decision-boundary between two (or more) classes, but in doing so tries to keep as much distance as possible between the border cases and the boundary - a technique known as the Large Margin Classification. You can imagine an SVM as a classifier that tries to create as wide a street as possible between the different classes. The middle of that street is then the decision-boundary and the two edges of that street are the distance between the extremes of the different classes on the one hand and the boundary on the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html\"> Support Vector Classifier SVC (SVC) by Scikit Learn </a> to create a decision boundary between two classes. The SVM class has many settings, but the most important for now are `kernel` and the parameter` C`. The `kernel` parameter indicates what kind of classification we want to achieve (linear, logistic, ...). The parameter 'C' has to do with the regularization and determines the width of the 'street'. The smaller the value of `C`, the wider the street but the more instances are incorrectly classified (because this parameter requires a consideration of the performance and the accuracy of our classifier this is a *hyperparameter*)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an SVC with a linear kernel (`kernel = 'linear' `) and a` C` of 10. Then call the `fit ()` method on that SVC and create a plot again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there is a reasonable distance between the *boundary* and the two support vectors. Also note that adding more examples on either side of the boundary does not make the classification better or more accurate: the lines in the figure above are completely determined by the boundary cases on either side of the 'street' - these are the support vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also note that two observations are provided with a kind of 'glow'. This is to indicate that these two act as *Support Vectors*. You also see that these are the two extremes within the data set."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}