03_CrossValidation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CrossValidation\n",
    "[公式ドキュメント](http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "下ごしらえ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from ipywidgets import interact\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris=load_iris()\n",
    "data=iris.data\n",
    "target=iris.target\n",
    "from sklearn.utils import shuffle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "clf=SVC(probability=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 今回のお題"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import cross_validation as cv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- モデルのインスタンス、記述子、ラベル、分割数、ジョブ数(-1でコア数分並列)を指定してあげると一行でクロスバリデーションできる\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "score=cv.cross_val_score(clf,data,target,cv=5,n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- scoreのデフォルトは正答率（accurary）\n",
    "\n",
    "- 他にもあるけどマルチクラス分類では使えないのが多い。\n",
    "\n",
    "- [参照](http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all:  [ 0.96666667  1.          0.96666667  0.96666667  1.        ]\n",
      "Accuracy: 0.9800 (+/- 0.0163)\n"
     ]
    }
   ],
   "source": [
    "print(\"all: \",score)\n",
    "print(\"Accuracy: {0:04.4f} (+/- {1:04.4f})\".format(score.mean(),score.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "単純に入力順にn分割している様子  \n",
    "入力をシャッフルすればOK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.9       ,  1.        ,  0.96666667,  0.96666667,  1.        ])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_shuffle,target_shuffle=shuffle(data,target)\n",
    "cv.cross_val_score(clf,data_shuffle,target_shuffle,n_jobs=-1,cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### SVMのコストパラメータを探索する場合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- まずざっくりとlogscaleでコストを振ってみる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "C_params=np.logspace(-2, 2, 9)\n",
    "scores=list()\n",
    "for C in C_params:\n",
    "    clf.C=C\n",
    "    tmp_score=cv.cross_val_score(clf,data_shuffle,target_shuffle,cv=5,n_jobs=-1)\n",
    "    scores.append(np.mean(tmp_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "コスト探索の可視化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f4cbba27588>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEHCAYAAACumTGlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFG1JREFUeJzt3X+MXWde3/H3mGSBLBN7UK8Q9tTjkmi/iV3h3YUdQREq\nyZaNWehacasSI4WwDhMLyBKVqjIyoKp/VLVHgtomUCnGXgWLxRVhU+9KK3AgViJAWrxkYpLY+yXC\nwesfAbmyk9iNlXrHt3/cZ5yb2ST3OPOM753x+yVFmXPOM9ff+Wrmfu5zzj33GWq320iStKTfBUiS\nBoOBIEkCDARJUmEgSJIAA0GSVBgIkiQAbmoyKCLWATvoBMiezNw+6/gyYC9wG3AJ2JSZR8uxR4Cf\nL0N3Z+auSrVLkirqOUOIiCXAo8A9wBpgY0TcMWvYVmAqM9cCDwC7yveuAR4EfhD4KPBTEfF99cqX\nJNXS5JTROPByZp7IzMvAfmD9rDGrgacBMjOBVRHRAu4EvpqZb2XmNPAssKFa9ZKkapoEwgrgZNf2\nqbKv2xHKE31EjAMrgVHgReBHI2IkIm4BPg3887kWLUmqr9E1hAa2ATsj4jngBWAKmM7Mr0fEduAp\n4OLM/l4P1m6320NDQ5VKk6QbwpyfNJsEwmk6r/hnjJZ9V2XmBWDTzHZEvAIcL8c+D3y+7P9vvHO2\n8a6GhoY4e/ZCg9LUS6s1bC8rsp912c96Wq3hOT9Gk1NGh4HbI2IsIj4E3Ad8qXtARCyNiJvL1xPA\nM5l5sWy3yv9XAvcCX5hz1ZKk6nrOEDJzOiIeBg7y9ttOj0XEZqCdmY/RuXj8eERcAV6i886iGX8c\nEd8NXAZ+MTPfqP5TSJLmbGhAP/667TSyDqfkddnPuuxnPa3W8JyvIXinsiQJMBAkSYWBIEkCDARJ\nUmEgSJIAA0GSVBgIkiTAQJAkFQaCJAkwECRJhYEgSQIMBElSYSBIkgADQZJUGAiSJMBAkCQVBoIk\nCTAQJEmFgSBJAgwESVJhIEiSAANBklQYCJIkAG5qMigi1gE76ATInszcPuv4MmAvcBtwCdiUmUfL\nsf8IPAhcAV4APpuZ/6/aTyBJqqLnDCEilgCPAvcAa4CNEXHHrGFbganMXAs8AOwq37sc+Bzw8cz8\nfjoBdF+98iVJtTQ5ZTQOvJyZJzLzMrAfWD9rzGrgaYDMTGBVRLTKsW8DPhwRNwG3AGeqVC5JqqpJ\nIKwATnZtnyr7uh0BNgBExDiwEhjNzDPAbwLfAE4Dr2Xmn821aElSfY2uITSwDdgZEc/RuU4wBUyX\nawvrgTHgdeCJiPiZzPxCrwdstYYrlSZ7WZf9rMt+Do4mgXCaziv+GaNl31WZeQHYNLMdEceB48A6\n4Hhmniv7vwj8K6BnIJw9e6FBaeql1Rq2lxXZz7rsZz01grVJIBwGbo+IMeBVOheFN3YPiIilwJuZ\neTkiJoBnM/NiRHwD+KGI+A7gLeCT5fEkSQOm5zWEzJwGHgYOAi8B+zPzWERsjoiHyrA7gRcj4hid\ndyM9Ur73r4En6JxCOgIMAY9V/ykkSXM21G63+13Du2k7jazDKXld9rMu+1lPqzU8NNfH8E5lSRJg\nIEiSCgNBkgQYCJKkwkCQJAEGgiSpMBAkSYCBIEkqDARJEmAgSJIKA0GSBBgIkqTCQJAkAQaCJKmo\ntYSmtKidO/caW7Yc4syZEZYvP8fk5N2MjCzrd1lSVQaC1MCWLYc4cOB+Oms8tYF97N59b5+rkury\nlJHUwIkTt9IJA4Chsi0tLgaC1MDY2Ot0ZgYAbcbG3uhnOdK88JSR1MDk5N3AvnIN4TyTk3f1uySp\nOgNBamBkZBm7d9/rGsBa1DxlJEkCDARJUmEgSJKAhtcQImIdsINOgOzJzO2zji8D9gK3AZeATZl5\nNCI+AvwvOm/PGAK+D/iNzNxV70eQJNXQc4YQEUuAR4F7gDXAxoi4Y9awrcBUZq4FHgB2AWTm32Xm\nxzLz48APAP8XeLJi/ZKkSpqcMhoHXs7ME5l5GdgPrJ81ZjXwNEBmJrAqIlqzxvwb4O8z8+Qca5Yk\nzYMmgbAC6H4SP1X2dTsCbACIiHFgJTA6a8xPA3/4wcqUJM23WvchbAN2RsRzwAvAFDA9czAibgY+\nA/xq0wdstYYrlSZ7WZf9rMt+Do4mgXCaziv+GaNl31WZeQHYNLMdEa8Ax7uG/ATwN5l5tmlh3vxT\nhzdS1WU/67Kf9dQI1iaBcBi4PSLGgFeB+4CN3QMiYinwZmZejogJ4JnMvNg1ZCOeLpKkgdbzGkJm\nTgMPAweBl4D9mXksIjZHxENl2J3AixFxjM67kR6Z+f6IuIXOBeUv1i5eklTPULvd7j3q+ms7jazD\nKXld9rMu+1lPqzU81HvU+/NOZUkSYCBIkgoDQZIEGAiSpMIFcqRF5Ny519iy5RAnTtzK2NjrTE7e\nzcjIsn6X9S1m6uysQHduYOu80RgI0iKyZcshDhy4Hxji+efbwD52776332V9i+46Ox+GPJh13mg8\nZSQtIidO3ErnSRZgqGwPnoVS543GQJAWkbGx1+m84gZoMzb2Rj/LeU8Lpc4bjaeMpEVkcvJuYF+5\nhvAGk5N39bukdzVTZ+cawvmBrfNG453Ki5x3gtZlP+uyn/V4p7IkqRoDQZIEGAiSpMJAkCQBBoIk\nqTAQJEmAgSBJKgwESRJgIEiSCgNBkgQYCJKkwkCQJAEGgiSpaPTx1xGxDthBJ0D2ZOb2WceXAXuB\n24BLwKbMPFqOLQV+D/iXwJVy7KvVfgJJUhU9ZwgRsQR4FLgHWANsjIg7Zg3bCkxl5lrgAWBX17Gd\nwFcy805gLXCsRuGSpLqazBDGgZcz8wRAROwH1gNf7xqzGvjvAJmZEbEqIlrAW8CPZubPlWPfBFwa\nSdKCcO7ca2zZcqgsOPQ6k5N3MzKyrN9lzZsmgbACONm1fYpOSHQ7AmwA/jIixoGVwCidU0T/JyI+\nT2d28DXgkcy8NNfCJWm+bdlyiAMH7geGeP75NrCP3bvv7XdZ86bWEprbgJ0R8RzwAjAFTAM3Ax8H\nfikzvxYRO4BfBf5LrwdstYYrlSZ7WZf9rGuQ+3nmzAgwsxDZEGfOjAx0vXPVJBBO03nFP2O07Lsq\nMy8Am2a2I+IV4DjwYeBkZn6tHHoC2NKkMJfVq8MlCuuyn3UNej+XLz8HtOmEQpvly88PbL01gqpJ\nIBwGbo+IMeBV4D5gY/eA8k6iNzPzckRMAM9k5kXgYkScjIiPZObfAZ8Ejs65akm6DiYn7wb2lWsI\nbzA5eVe/S5pXPQMhM6cj4mHgIG+/7fRYRGwG2pn5GHAn8HhEXAFeAh7seohfBv4gIm6mM2v4bO0f\nQpLmw8jIskV9zWC2oXa73e8a3k17UKdlC82gT8kXGvtZl/2sp9UaHuo96v15p7IkCTAQJEmFgSBJ\nAgwESVJhIEiSAANBklQYCJIkwECQJBUGgiQJMBAkSYWBIEkCDARJUmEgSJIAA0GSVBgI6qtz515j\nYuJJPvWpP2di4oucP/9av0uSFpSZv6Ghob1/PdfHqrWmsvSB3GiLmEu1df0NfWKuj+UMQX114sSt\ndC9i3tmW1NQ7/4bmxkBQX42NvU5nEXOANmNjb/SzHGnBeeff0Nx4ykh9daMtYi7VNvM3dODANw/D\npjk9lmsqL3KuWVuX/azLftbjmsqSpGoMBEkSYCBIkopGF5UjYh2wg06A7MnM7bOOLwP2ArcBl4BN\nmXm0HPsH4HXgCnA5M8drFS9JqqfnDCEilgCPAvcAa4CNEXHHrGFbganMXAs8AOzqOnYF+LHM/Jhh\nIEmDq8kpo3Hg5cw8kZmXgf3A+lljVgNPA2RmAqsiolWODTX8dyRJfdTkiXoFcLJr+1TZ1+0IsAEg\nIsaBlcBoOdYGnoqIwxExMbdyJUnzpdaNaduAnRHxHPACMAVMl2M/kpmvlhnDUxFxLDP/otcDtlrD\nlUqTvazLftZlPwdHk0A4TecV/4zRsu+qzLxA1y1yEfEKcLwce7X8/2xEPEnnFFTPQPBmlTq88acu\n+1mX/aynRrA2OWV0GLg9IsYi4kPAfcCXugdExNKIuLl8PQE8k5kXI+KWiPiusv/DwKeAF+dctSSp\nup4zhMycjoiHgYO8/bbTYxGxGWhn5mPAncDjEXEFeAl4sHz79wBPRkS7/Ft/kJkH5+MHkSTNjZ9l\ntMg5Ja/LftZlP+vxs4wkSdUYCJIkwECQJBUDGQjj418e2AXXXRRe0mI1kCumHT78b+nc4Dx4C667\nKLykxWogZwgdg7nguovCS1qsBjgQBnPBdReFl7RYDeQpo0984sssX35+IBdcd1F4SYuVN6Ytct74\nU5f9rMt+1uONaZKkagwESRJgIEiSCgNBkgQYCJKkwkCQJAEGgiSpMBAkSYCBIEkqDARJEmAgSJIK\nA0GSBBgIkqTCQJAkAQ3XQ4iIdcAOOgGyJzO3zzq+DNgL3AZcAjZl5tGu40uArwGnMvMzlWqXJFXU\nc4ZQnswfBe4B1gAbI+KOWcO2AlOZuRZ4ANg16/gjwFEkSQOrySmjceDlzDyRmZeB/cD6WWNWA08D\nZGYCqyKiBRARo8Cngd+rVrV6OnfuNSYmnmR8/MtMTHyR8+df63dJkgZck1NGK4CTXdun6IREtyPA\nBuAvI2IcWAmMAmeB/wH8Z2DpnKtVY1u2HOLAgfuBITprQO9j9+57+1yVpEFWa03lbcDOiHgOeAGY\nAqYj4ieBf8rM5yPix+g8OzXSag1XKu3GdObMCG+3e4gzZ0bsaSX2sS77OTiaBMJpOq/4Z4yWfVdl\n5gVg08x2RBwHjgP3AZ+JiE8D3wkMR8TvZ+bP9vpHXWd1bpYvP0dnZtCZISxfft6eVuAawHXZz3pq\nBGuTQDgM3B4RY8CrdJ7kN3YPiIilwJuZeTkiJoBnM/MinYvNW8uYfw38pyZhoLmbnLwb2MeZMyMs\nX36eycm7+l2SpAHXMxAyczoiHgYO8vbbTo9FxGagnZmPAXcCj0fEFeAl4MH5LFq9jYwsY/fue30F\nJqmxoXa73e8a3k3bJ7E6DIS67Gdd9rOeVmu48TXa9+KdypIkwECQJBUGgiQJMBAkSYWBIEkCDARJ\nUmEgSJIAA0GSVBgIkiTAQJAkFQaCJAkwECRJhYEgSQIMBElSYSBIkgADQZJUGAiSJMBAkCQVBoIk\nCTAQJEmFgSBJAgwESVJhIEiSALipyaCIWAfsoBMgezJz+6zjy4C9wG3AJWBTZh6NiG8HngU+VP6t\nJzLzv1asX5JUSc8ZQkQsAR4F7gHWABsj4o5Zw7YCU5m5FngA2AWQmW8Bd2Xmx4CPAj8REeMV65ck\nVdLklNE48HJmnsjMy8B+YP2sMauBpwEyM4FVEdEq22+WMd9OZ5bQrlG4JKmuJoGwAjjZtX2q7Ot2\nBNgAUGYAK4HRsr0kIqaAfwSeyszDcy1aklRfo2sIDWwDdkbEc8ALwBQwDZCZV4CPRcStwP+OiNWZ\nebTXA7Zaw5VKk72sy37WZT8HR5NAOE3nFf+M0bLvqsy8AGya2Y6IV4Djs8a8ERGHgHVAz0A4e/ZC\ng9LUS6s1bC8rsp912c96agRrk1NGh4HbI2IsIj4E3Ad8qXtARCyNiJvL1xPAM5l5MSL+WUQsLfu/\nE/hx4OtzrlqSVF3PQMjMaeBh4CDwErA/M49FxOaIeKgMuxN4MSKO0Xk30iNl//cChyLieeCrwJ9m\n5ldq/xCSpLkbarcH8k0/baeRdTglr8t+1mU/62m1hofm+hjeqSxJAgwESVJhIEiSAANBklQYCJIk\nwECQJBUGgiQJMBAkSYWBIEkCDARJUmEgSJIAA0GSVBgIkiTAQJAkFQaCJAkwECRJhYEgSQIMBElS\nYSBIkgADQZJUGAiSJMBAkCQVNzUZFBHrgB10AmRPZm6fdXwZsBe4DbgEbMrMoxExCvw+8D3AFWB3\nZu6qWL8kqZKeM4SIWAI8CtwDrAE2RsQds4ZtBaYycy3wADDzpP9N4Fcycw3ww8Avvcv3SpIGQJNT\nRuPAy5l5IjMvA/uB9bPGrAaeBsjMBFZFRCsz/zEzny/7LwLHgBXVqpckVdMkEFYAJ7u2T/GtT+pH\ngA0AETEOrARGuwdExCrgo8BXP2CtkqR51OgaQgPbgJ0R8RzwAjAFTM8cjIjvAp4AHikzhV6GWq3h\nSqXJXtZlP+uyn4OjSSCcpvOKf8Zo2XdVZl4ANs1sR8QrwPHy9U10wmBfZh6Ya8GSpPnRJBAOA7dH\nxBjwKnAfsLF7QEQsBd7MzMsRMQE80zUT2AsczcydFeuWJFU21G63ew4qbzvdydtvO90WEZuBdmY+\nFhE/BDxO562lLwEPZubrEfEjwLN0TiO1y39bM/NP5ufHkSR9UI0CQZK0+HmnsiQJMBAkSYWBIEkC\nDARJUmEgSJKAencqz7uIWA/8JDAM7M3Mp/pc0oIVEf8C+DXg1sz8D/2uZyGLiFuA3wXeonP/zRf6\nXNKC5u9mXdf6vLlgZgiZeSAzHwJ+AfAXZQ4y85XM/Pl+17FIbAD+KDM3A5/pdzELnb+bdV3r8+Z1\nnyFExB7gp4B/yszv79r/vmsudPl14HfmvdAFoEIvNcsH6Oko8Lfl62n0Dv6O1jWHfjZ63uzHDOHz\ndNZWuOr91lyIiPsj4rciYnlEbAO+MvOR2vrAvfzeMnzoeha7QFxTT+l8EvDMJ/vaz291rf2cYS/f\n3TX381qeN697IGTmXwDnZ+1+zzUXMnNfZv4K8O+ATwL/PiIeup41D6o59PKtiPifwEcjYst1LXrA\nXWtPgSfp/E7+DvDl61fpwnCt/YyI7/Z38719gH5+jmt43hyUi8rvtubCePeAzPxt4LevZ1ELVJNe\nnqNzTlHNvGdPM/NNuj7pV428Xz/93bx279fPa3reXDAXlSVJ82tQAqHnmgtqzF7WZ0/rsp91Vetn\nv04ZDfHOi0Y911zQe7KX9dnTuuxnXfPWz+s+Q4iILwB/BXwkIr4REZ/NzGngc8BBOusp7M/MY9e7\ntoXGXtZnT+uyn3XNdz9dD0GSBAzONQRJUp8ZCJIkwECQJBUGgiQJMBAkSYWBIEkCDARJUmEgSJIA\n+P947DT278P+7gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4cbbbb5550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_xscale(\"log\")\n",
    "ax.set_xlim(0.01,100)\n",
    "ax.scatter(C_params,scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 桁を抑えたところで0.01-10まで探索してみる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "C_params2=np.linspace(0.001,10,21)\n",
    "scores2=list()\n",
    "for C in C_params2:\n",
    "    clf.C=C\n",
    "    tmp_score=cv.cross_val_score(clf,data_shuffle,target_shuffle,cv=5,n_jobs=-1)\n",
    "    scores2.append(np.mean(tmp_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f4cbb8ceda0>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEDCAYAAAA849PJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFKRJREFUeJzt3X+s3fV93/HncUzakFzw3Xaazbi+LKC8AVdxwiaPNYpa\nnLa4aYsLf6y4WkpiSpAKFeqq6Gaepmx/TDNXqgoW+wcHELWWuRpL5vyRrRBhEbVSUxg2A+y8a9XU\nsTFl7gy2GYg69tkf3+8lN1eG+z33Hvt7vv48H5KFv9/v55zzOufi7+v7635PbzAYIEkqz7K2A0iS\n2mEBSFKhLABJKpQFIEmFsgAkqVAWgCQVanmTQRGxAbifqjAezsz75i1fATwCXAW8DWzOzH31snuB\n366Hbs/MbSPKLklaggX3ACJiGfAgcBOwBtgUEdfMG7YF2JOZa4HbgW31Y9cAdwD/FPgk8KsR8bHR\nxZckLVaTQ0DrgAOZeSgzTwM7gY3zxlwHPAWQmQlcGRF94Frge5n5TmaeAb4L3Dqy9JKkRWtSAFcA\nh+dMH6nnzfU89Yo9ItYBq4FVwIvAZyJiMiIuBT4H/PRSQ0uSlm5UJ4G3ApMR8RxwN7AHOJOZ3wfu\nA54Evj07f0SvKUlagiYngV+h2qKftaqe967MPAVsnp2OiJeBg/WyR4FH6/n/gR/fmzinwWAw6PV6\nDaJJkmpDrzSbFMAzwNURMQW8CtwGbJo7ICIuB97KzNMRcSfwdGa+WS/rZ+axiFgN3ALcsNAL9no9\njh07NeRbGQ/9/kRns4P522b+dnU5f78/MfRjFiyAzDwTEfcAT/Cjy0D3R8RdwCAzH6I62ftYRJwF\nXqK68mfWf4uIvwecBn4nM08OnVKSNHK9Mb0d9KDLLdzV7GD+tpm/XV3O3+9PDH0IyN8ElqRCWQCS\nVCgLQJIKZQFIUqEsAEkqlAUgSYWyACSpUBaAJBXKApCkQlkAklQoC0CSCmUBSFKhLABJKpQFIEmF\nsgAkqVAWgCQVygKQpEJZAJJUKAtAkgplAUhSoSwASSqUBSBJhbIAJKlQFoAkFcoCkKRCWQCSVCgL\nQJIKZQFIUqGWNxkUERuA+6kK4+HMvG/e8hXAI8BVwNvA5szcVy/7PeAO4CzwAvDFzPy7kb0DSdKi\nLLgHEBHLgAeBm4A1wKaIuGbesC3AnsxcC9wObKsfuxL4XeD6zPwEVeHcNrr4kqTFanIIaB1wIDMP\nZeZpYCewcd6Y64CnADIzgSsjol8v+wDw4YhYDlwKHB1JcknSkjQpgCuAw3Omj9Tz5noeuBUgItYB\nq4FVmXkU+APgB8ArwBuZ+Z2lhpYkLd2oTgJvBSYj4jngbmAPcKY+N7ARmAJWAh+JiN8c0WtKkpag\nyUngV6i26Getque9KzNPAZtnpyPiIHAQ2AAczMzj9fxvAD8LfH2hF+33JxpEG09dzg7mb5v529X1\n/MNoUgDPAFdHxBTwKtVJ3E1zB0TE5cBbmXk6Iu4EvpuZb0bED4AbIuIngXeAz9bPt6Bjx04N8TbG\nR78/0dnsYP62mb9dXc6/mOJa8BBQZp4B7gGeAF4Cdmbm/oi4KyK+VA+7FngxIvZTXS10b/3YvwAe\npzok9DzQAx4aOqUkaeR6g8Gg7QznMuhyC3c1O5i/beZvV5fz9/sTvWEf428CS1KhLABJKpQFIEmF\nsgAkqVAWgCQVygKQpEJZAJJUKAtAkgplAUhSoSwASSqUBSBJhbIAJKlQFoAkFcoCkKRCWQCSVCgL\nQJIKZQFIUqEsAEkqlAUgSYVa3nYANXP8+BtMT+/m0KHLmJo6wczMeiYnV4z8+Y8enWTlyuMjf35J\n48cC6Ijp6d3s2vV5oMfevQNgB9u333Jenh9G//ySxo+HgDri0KHLqFbOAL16ujvPL2n8WAAdMTV1\ngmrLHGDA1NTJTj2/pPHjIaCOmJlZD+yozwGcZGbmxvPy/NU5gNdH/vySxk9vMBgsPOrCGxw7dqrt\nDIvS70/Q1exg/raZv11dzt/vT/QWHvXjPAQkSYWyACSpUBaAJBXKApCkQjW6CigiNgD3UxXGw5l5\n37zlK4BHgKuAt4HNmbkvIj4O/DHV9YU94GPAv83MbaN7C5KkxVhwDyAilgEPAjcBa4BNEXHNvGFb\ngD2ZuRa4HdgGkJl/mZmfyszrgX8C/D/gmyPML0lapCaHgNYBBzLzUGaeBnYCG+eNuQ54CiAzE7gy\nIvrzxvwC8FeZeXiJmSVJI9CkAK4A5q60j9Tz5noeuBUgItYBq4FV88b8BvBfFhdTkjRqo/pN4K3A\nAxHxHPACsAc4M7swIi4Bbga+0vQJ+/2JEUW78LqcHczfNvO3q+v5h9GkAF6h2qKftaqe967MPAVs\nnp2OiJeBg3OG/DLwvzLzWNNgHf5tvM5mB/O3zfzt6nL+xRRXkwJ4Brg6IqaAV4HbgE1zB0TE5cBb\nmXk6Iu4Ens7MN+cM2YSHfyRprCx4DiAzzwD3AE8ALwE7M3N/RNwVEV+qh10LvBgR+6muFrp39vER\ncSnVCeBvjDq8JGnxvBnciHV5FxLM3zbzt6vL+b0ZnCSpMQtAkgplAUhSoSwASSqUBSBJhfI7gS9S\nx4+/wfT07vo7hE8wM7OeyckVncmz2PHVdxofP2/PPy75zzc//3Y//wvFArhITU/vZteuzwM99u4d\nADvYvv2WzuRZyvjq7uPn7/nHIf/55uff7ud/oXgI6CJ16NBlVP8zA/Tq6fYMm8fx/rzGefzFwgK4\nSE1NnaDakgEYMDV1ss04Q+dxvD+vcR5/sfAQ0EVqZmY9sKM+BnqSmZkbO5VnseOrY7ivn7fnH5f8\n55uff7uf/4XirSBGrMu/Sg7mb5v529Xl/N4KQpLUmAUgSYWyACSpUBaAJBXKApCkQlkAklQoC0CS\nCmUBSFKhLABJKpQFIEmFsgAkqVAWgCQVygKQpEJZAJJUKAtAkgplAUhSoSwASSpUo6+EjIgNwP1U\nhfFwZt43b/kK4BHgKuBtYHNm7quXXQ58DfgZ4Gy97HsjeweSpEVZcA8gIpYBDwI3AWuATRFxzbxh\nW4A9mbkWuB3YNmfZA8C3M/NaYC2wfxTBJUlL02QPYB1wIDMPAUTETmAj8P05Y64D/iNAZmZEXBkR\nfeAd4DOZ+YV62Q+Bk6OL313Hj7/B9PTu+kurTzAzs57JyRVtx2qs6/m7btjP35+XzqVJAVwBHJ4z\nfYSqFOZ6HrgV+LOIWAesBlZRHfL524h4lGrr/1ng3sx8e6nBu256eje7dn0e6LF37wDYwfbtt7Qd\nq7Gu5++6YT9/f146l0bnABrYCjwQEc8BLwB7gDPAJcD1wN2Z+WxE3A98BfjqQk/Y70+MKNqF1yT7\n0aOTQK+e6nH06OTYvOcS8o+z8/H5X8ifVwmf/8WiSQG8QrVFP2tVPe9dmXkK2Dw7HREvAweBDwOH\nM/PZetHjwHSTYMeOnWoybOz0+xONsq9ceRwYUP2jHLBy5etj8Z5LyT+uztfnf6F+XqV8/uNoMcXV\npACeAa6OiCngVeA2YNPcAfWVPm9l5umIuBN4OjPfBN6MiMMR8fHM/Evgs8C+oVNehGZm1gM76mOy\nJ5mZubHtSEPpev6uG/bz9+elc+kNBoMFB9WXgT7Ajy4D3RoRdwGDzHwoIm4AHqM65v8ScEdmnqgf\nu5bqMtBLqPYKvji77H0MutzCXc0O5m+b+dvV5fz9/kRv4VE/rlEBtMACaIn522X+dnU5/2IKwN8E\nlqRCWQCSVCgLQJIKZQFIUqEsAEkqlAUgSYWyACSpUBaAJBXKApCkQlkAklQoC0CSCmUBSFKhLABJ\nKpQFIEmFsgAkqVAWgCQVygKQpEJZAJJUKAtAkgplAUhSoSwASSqUBSBJhbIAJKlQFoAkFcoCkKRC\nWQCSVCgLQJIKZQFIUqGWNxkUERuA+6kK4+HMvG/e8hXAI8BVwNvA5szcVy/7a+AEcBY4nZnrRhVe\nkrR4C+4BRMQy4EHgJmANsCkirpk3bAuwJzPXArcD2+YsOwv8fGZ+ypW/JI2PJoeA1gEHMvNQZp4G\ndgIb5425DngKIDMTuDIi+vWyXsPXkSRdQE1WzFcAh+dMH6nnzfU8cCtARKwDVgOr6mUD4MmIeCYi\n7lxaXEnSqDQ6B9DAVuCBiHgOeAHYA5ypl306M1+t9wiejIj9mfmnCz1hvz8xomgXXpezg/nbZv52\ndT3/MJoUwCtUW/SzVtXz3pWZp4DNs9MR8TJwsF72av3fYxHxTapDSgsWwLFjpxpEGz/9/kRns4P5\n22b+dnU5/2KKq8khoGeAqyNiKiI+CNwGfGvugIi4PCIuqf9+J/B0Zr4ZEZdGxEfq+R8Gfgl4ceiU\nkqSRW3APIDPPRMQ9wBP86DLQ/RFxFzDIzIeAa4HHIuIs8BJwR/3wjwLfjIhB/Vr/OTOfOB9vRJI0\nnN5gMGg7w7kMurwb1tXsYP62mb9dXc7f70/0hn2Ml2dKUqEsAEkqlAUgSYWyACSpUBaAJBXKApCk\nQo3qVhDFO378Daand3P06CQrVx5nZmY9k5Mr2o4lSe/JAhiR6end7Nr1eaqbnw6AHWzffkvLqSTp\nvXkIaEQOHbqMauUP0KunJWl8WQAjMjV1gmrLH2DA1NTJNuNI0oI8BDQiMzPrgR31OYDXmZm5se1I\nkvS+LIARmZxcwfbtt3T6XiKSyuIhIEkqlAUgSYWyACSpUBaAJBXKApCkQlkAklQoC0CSCmUBSFKh\nLABJKpQFIEmFsgAkqVAWgCQVygKQpEJZAJJUKAtAkgplAUhSoRp9IUxEbADupyqMhzPzvnnLVwCP\nAFcBbwObM3PfnOXLgGeBI5l584iyS5KWYME9gHrl/SBwE7AG2BQR18wbtgXYk5lrgduBbfOW3wvs\nQ5I0NpocAloHHMjMQ5l5GtgJbJw35jrgKYDMTODKiOgDRMQq4HPA10aWWpK0ZE0K4Arg8JzpI/W8\nuZ4HbgWIiHXAamBVvewPgS8DgyUllSSN1KhOAm8FJiPiOeBuYA9wJiJ+BXgtM/cCvfqPJGkM9AaD\n998wj4gbgH+XmRvq6a8Ag/knguc95iDwCapzA/8S+CHwIWAC+EZm/tYCudxbkKThDL2B3aQAPgAk\n8FngVeAvgE2ZuX/OmMuBtzLzdETcCXw6M78w73l+Dvj9hlcBDY4dOzXUGxkX/f4EXc0O5m+b+dvV\n5fz9/sTQBbDgIaDMPAPcAzwBvATszMz9EXFXRHypHnYt8GJE7Ke6WujeYYNIki6sBfcAWuIeQEvM\n3y7zt6vL+c/LHoAk6eJkAUhSoSwASSqUBSBJhbIAJKlQFoAkFcoCkKRCWQCSVCgLQJIKZQFIUqEs\nAEkqlAUgSYWyACSpUBaAJBXKApCkQlkAklQoC0CSCmUBSFKhLABJKpQFIEmFsgAkqVAWgCQVygKQ\npEJZAJJUKAtAkgplAUhSoSwASSqUBSBJhVreZFBEbADupyqMhzPzvnnLVwCPAFcBbwObM3NfRPwE\n8F3gg/VrPZ6Z/36E+SVJi7TgHkBELAMeBG4C1gCbIuKaecO2AHsycy1wO7ANIDPfAW7MzE8BnwR+\nOSLWjTC/JGmRmhwCWgccyMxDmXka2AlsnDfmOuApgMxM4MqI6NfTb9VjfoJqL2AwiuCSpKVpUgBX\nAIfnTB+p5831PHArQL2FvxpYVU8vi4g9wN8AT2bmM0sNLUlaulGdBN4KTEbEc8DdwB7gDEBmnq0P\nAa0C/llEXDei15QkLUGTk8CvUG3Rz1pVz3tXZp4CNs9OR8TLwMF5Y05GxG5gA7Bvgdfs9fsTDaKN\npy5nB/O3zfzt6nr+YTTZA3gGuDoipiLig8BtwLfmDoiIyyPikvrvdwJPZ+abEfEPIuLyev6HgF8E\nvj/SdyBJWpQFCyAzzwD3AE8ALwE7M3N/RNwVEV+qh10LvBgR+6muFrq3nv+PgN0RsRf4HvAnmfnt\nUb8JSdLweoOBF+VIUon8TWBJKpQFIEmFsgAkqVCN7gV0oUXEDPBrwDvAXwFfzMyT7aZa2EL3TBpn\nEbEK+CPgo8BZYHtmbms31XDq25Y8CxzJzJvbzjOM+mq5rwE/Q/X5b87M77WbqrmI+D3gDqrsL1D9\nm/27dlO9t4h4GPhV4LXM/EQ9bxL4Y2AK+GvgX2TmidZCvo/3yD/0enNc9wCeANZk5ieBA8C/bjnP\nghreM2mc/RD4V5m5BvjnwN0dyw/V1WcL/Y7JuHoA+HZmXgusBfa3nKexiFgJ/C5wfb0yWk51ufg4\ne5Tq3+pcXwG+k5lBdWubcV7vnCv/0OvNsSyAzPxOZp6tJ/+c+rYSY67JPZPGVmb+TWburf/+JtUK\naP4tP8ZWvQfzOaqt6E6JiMuAz2TmowCZ+cMu7PHO8wHgwxGxHLgUONpynveVmX8KvD5v9kbgsfrv\njwG/fkFDDeFc+Rez3hzLAphnM/A/2g7RQJN7JnVCRFxJdffWzhyCAP4Q+DLdvNngPwb+NiIejYjn\nIuKh+hcnOyEzjwJ/APyA6i4Bb2Tmd9pNtSg/lZmvQbVBBPxUy3mWotF6s7UCiIgnI+J/z/nzQv3f\nX5sz5t8ApzPz623lLE1EfAR4HLi33hMYexHxK1THQvcCvfpPlywHrgf+U2ZeD7xFdTiiE+rvA9lI\ndex8JfCRiPjNdlONRBc3JoZab7Z2Ejgzf/H9lkfEF6h26ddfkEBLt+A9k8Zdvfv+OLAjM3e1nWcI\nnwZujojPAR8CJiLijzLzt1rO1dQR4HBmPltPPw5Mt5hnWL8AHMzM4wAR8Q3gZ4Gubbi9FhEfzczX\nIuIfAv+n7UDDGna9OZaHgOqrab4M3Fx/qUwXLHjPpA54BNiXmQ+0HWQYmbklM1dn5seoPvenOrTy\npz7scDgiPl7P+izdOpn9A+CGiPjJiOhR5e/CSez5e4vfAr5Q//12YNw3gn4s/2LWm2N5K4iIOED1\nNZL/t57155n5Oy1GaqT+ATzAjy4D3dpypMYi4tNUX9/5AtWu7wDYkpn/s9VgQ4qInwN+v4OXga6l\nOoF9CdWddL84rpcgnktEfJWqfE9T3Q7+t+uLIcZSRHwd+Hng7wOvAV8F/jvwX4GfBg5RXQb6RlsZ\n38975N/CkOvNsSwASdL5N5aHgCRJ558FIEmFsgAkqVAWgCQVygKQpEJZAJJUKAtAkgplAUhSof4/\nMN+0eo5UD3MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4cbb949e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(C_params2,scores2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 細かく探索して終了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f4cbb7d90b8>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEDCAYAAAAvNJM9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF91JREFUeJzt3X+sX/V93/HnJYAqMsc41QVqjC8DyhtwBYRlHuuWKSRZ\nMVEWE2sjOFIGmLjWEmeo2ySIpQgpSlXbUhj2SFZhDHXZiIkIk83UpZQSEbVTGS0G8cN5l8Xg+Ecx\nTvllJyx17e/+OOfiLzcf+x5/7/f6e77O8yFZ3HPO53zv65jvuS+fH99zRzqdDpIkTXTSoANIktrJ\ngpAkFVkQkqQiC0KSVGRBSJKKLAhJUtHJkw2IiPXAp4A9mXnpEcasBa4BfgrcmJnP1PMXAHdSFdH6\nzFxVz58FPAiMAa8A12XmW1PeGklS3zQ5grgPuPpICyPiGuD8zPx1YBnw+/X8k4C76nXnAYsj4qJ6\ntduAxzIzgMeBr/S8BZKkaTFpQWTmnwFvHGXIQuAP67FPAjMj4kxgPvBSZm7PzAPAxnrs+Dob6q83\nANf2Fl+SNF36cQ3ibGBH1/TOet6R5gOcmZl7ADLzVeCMPuSQJPXRdFykHulhHZ/3IUktM+lF6gZ2\nAed0Tc+p550KzC3MB3g1Is7MzD0RcRbwWpNv1Ol0OiMjvfSPJP1S6+kHZ9OCGDnKN9gMfAl4MCKu\nBN6sf/D/BLggIsaAvwGuBxZ3rXMjsAq4AdjUKMTICHv37msYuX1GR2eYf0CGOTuYf9BOhPy9aHKb\n6wPAR4FfjYgfA7dTHR10MvPuzPyjiPhkRPxfqttcbwLIzIMRsRx4lMO3uW6tX3YV8J2IWAJsB67r\nKb0kadqMDNnjvjvD3uLmH4xhzg7mH7QTIH9Pp5j8JLUkqciCkCQVWRCSpCILQpJUZEFIkoosCElS\nkQUhSSqyICRJRRaEJKnIgpAkFVkQkqQiC0KSVGRBSJKKLAhJUpEFIUkqsiAkSUUWhCSpyIKQJBVZ\nEJKkIgtCklRkQUiSiiwISVKRBSFJKjq5yaCIWADcSVUo6zNz1YTlpwP3AucD7wBLMvPFetktwBfq\nofdk5pp6/u3AUuC1etmKzPze1DZHktQvkx5BRMRJwF3A1cA8YHFEXDRh2ApgS2ZeBtwArK3XnQfc\nDHwYuBz4VESc17XeHZl5Rf3HcpCkFmlyimk+8FJmbs/MA8BGYOGEMZcAjwNkZgLnRsQocDHwZGb+\nPDMPAk8Ai7rWG5nqBkiSpkeTgjgb2NE1vbOe1+1Z6h/8ETEfmAvMAZ4HPhIRsyLiNOCTwDld6y2P\niGci4p6ImNnjNkiSpkGjaxANrATWRMTTwHPAFuBgZv4wIlYBfwLsH59fr/Mt4GuZ2YmIrwN3UJ2O\nOqrR0Rl9ijwY5h+cYc4O5h+0Yc/fiyYFsYvqiGDcnHreuzJzH7BkfDoiXga21cvuA+6r5/8u9dFI\nZu7teol1wCNNAu/du6/JsFYaHZ1h/gEZ5uxg/kE7EfL3oskppqeACyJiLCJOBa4HNncPiIiZEXFK\n/fVS4InM3F9Pj9b/nQt8Bnignj6r6yUWUZ2OkiS1xKRHEJl5MCKWA49y+DbXrRGxDOhk5t1UF6M3\nRMQh4AXee6rouxHxQeAA8MXMfLuevzoiLgcOAa8Ay/q1UZKkqRvpdDqDznAsOsN+mGf+wRjm7GD+\nQTsB8vd0x6ifpJYkFVkQkqQiC0KSVGRBSJKKLAhJUpEFIUkqsiAkSUUWhCSpyIKQJBVZEJKkIgtC\nklRkQUiSiiwISVKRBSFJKrIgJElFFoQkqciCkCQVWRCSpCILQpJUZEFIkoosCElSkQUhSSqyICRJ\nRSc3GRQRC4A7qQplfWaumrD8dOBe4HzgHWBJZr5YL7sF+EI9dF1mrq3nzwIeBMaAV4DrMvOtqW6Q\nJKk/Jj2CiIiTgLuAq4F5wOKIuGjCsBXAlsy8DLgBGC+BecDNwIeBy4F/FRHn1evcBjyWmQE8Dnxl\n6psjSeqXJqeY5gMvZeb2zDwAbAQWThhzCdUPeTIzgXMjYhS4GHgyM3+emQeBJ4BF9ToLgQ311xuA\na6e0JZKkvmpSEGcDO7qmd9bzuj1L/YM/IuYDc4E5wPPARyJiVkScBnwSOKde58zM3AOQma8CZ/S6\nEZKk/mt0DaKBlcCaiHgaeA7YAhzMzB9GxCrgT4D94/OP8BqdJt9odHRGH+IOjvkHZ5izg/kHbdjz\n96JJQeyiOiIYN6ee967M3AcsGZ+OiJeBbfWy+4D76vm/y+GjkVcj4szM3BMRZwGvNQm8d+++JsNa\naXR0hvkHZJizg/kH7UTI34smp5ieAi6IiLGIOBW4HtjcPSAiZkbEKfXXS4EnMnN/PT1a/3cu8Bng\ngXq1zcCN9dc3AJt62gJJ0rSY9AgiMw9GxHLgUQ7f5ro1IpYBncy8m+pi9IaIOAS8QHXn0rjvRsQH\ngQPAFzPz7Xr+KuA7EbEE2A5c17etkiRN2Uin0+jUf1t0hv0wz/yDMczZwfyDdgLkH+llPT9JLUkq\nsiAkSUUWhCSpyIKQJBVZEJKkIgtCklRkQUiSiiwISVKRBSFJKrIgJElFFoQkqciCkCQVWRCSpCIL\nQpJUZEFIkor69TuppV9Kr7/+Jrfe+n22b/8AY2NvsXr1x5g16/S+j9+9exazZ78+ba/flvxqFwtC\nmoJbb/0+mzZ9HhjhmWc6wP2sW/eZaRkP0/v6bcivdvEUkzQF27d/gOqHH8BIPe34XserXSwIaQrG\nxt6i+pcxQIexsbePNtzxk4xXu3iKSZqC1as/Btxfn5N/m9Wrr5qW8dU5/Dem7fXbkl/tMtLpdCYf\n1R6dIf/F4cP+i8+HNv8wZwfzD9oJkH9k8lG/yFNMkqQiC0KSVNToGkRELADupCqU9Zm5asLy04F7\ngfOBd4Almflivex3gJuBQ8BzwE2Z+XcRcTuwFHitfpkVmfm9qW+SJKkfJj2CiIiTgLuAq4F5wOKI\nuGjCsBXAlsy8DLgBWFuvOxv4MnBFZl5KVUjXd613R2ZeUf+xHCSpRZqcYpoPvJSZ2zPzALARWDhh\nzCXA4wCZmcC5ETFaL3sf8P6IOBk4DdjdtV5PF04kSdOvSUGcDezomt5Zz+v2LLAIICLmA3OBOZm5\nG/gG8GNgF/BmZj7Wtd7yiHgmIu6JiJk9boMkaRr063MQK4E1EfE01XWGLcDB+trEQmAMeAt4KCI+\nl5kPAN8CvpaZnYj4OnAH1bWKoxodndGnyINh/sEZ5uxg/kEb9vy9aFIQu6iOCMbNqee9KzP3AUvG\npyNiG7ANWABsy8zX6/kPA78JPJCZe7teYh3wSJPAQ34vsvkHZJizg/kH7UTI34smp5ieAi6IiLGI\nOJXqIvPm7gERMTMiTqm/Xgr8IDP3U51aujIifiUiRoCPA1vrcWd1vcQi4PmetkCSNC0mPYLIzIMR\nsRx4lMO3uW6NiGVAJzPvBi4GNkTEIeAF6lNFmfl/IuIhqlNOB+r/3l2/9OqIuJzq9tdXgGV93TJJ\n0pT4qI3j6EQ4TB3W/MOcHcw/aCdAfh+1IUnqHwtCklRkQUiSiiwISVKRBSFJKrIgJElFFoQkqciC\nkCQVWRCSpCILQpJUZEFIkoosCElSkQUhSSqyICRJRRaEJKnIgpAkFVkQkqQiC0KSVGRBSJKKLAhJ\nUpEFIUkqsiAkSUUWhCSp6OQmgyJiAXAnVaGsz8xVE5afDtwLnA+8AyzJzBfrZb8D3AwcAp4DbsrM\nv4uIWcCDwBjwCnBdZr7Vj42SJE3dpEcQEXEScBdwNTAPWBwRF00YtgLYkpmXATcAa+t1ZwNfBq7I\nzEupCun6ep3bgMcyM4DHga9MfXMkSf3S5BTTfOClzNyemQeAjcDCCWMuofohT2YmcG5EjNbL3ge8\nPyJOBk4DdtXzFwIb6q83ANf2vBUt9/rrb7J06f9g/vxHWLr0Yd54481G43/rt/60VePbkL9t2dVf\nw/7eP9E0OcV0NrCja3onVWl0exZYBPx5RMwH5gJzMnNLRHwD+DHwM+DRzPzTep0zMnMPQGa+GhFn\nTGE7Wu3WW7/Ppk2fB0aADnA/69Z9ptH4Z55p1/hB529bdvXXsL/3TzSNrkE0sBJYExFPU11n2AIc\nrK9NLKS6zvAW8FBEfC4zHyi8RqfJNxodndGnyMfP7t2zqN5gACPs3j3rqNvh+COPb1OW460tOXrV\nJH/b/v+2+f1wPDQpiF1URwTj5nD4NBEAmbkPWDI+HRHbgG3AAmBbZr5ez38Y+E3gAWBPRJyZmXsi\n4izgtSaB9+7d12RYq8ye/TpV/1X/Cpk9+42jbofjjzy+TVmOp9HRGa3I0aum+dv2/7et74dj1Wup\nNSmIp4ALImIM+Buqi8yLuwdExEzgZ5l5ICKWAj/IzP0R8WPgyoj4FeDnwMfr1wPYDNwIrKK6sL2p\npy0YAqtXfwy4n927ZzF79husXn1Vo/Hbt3+AsbG3WzO+Dfnbll39Nezv/RPNSKcz+Zmd+jbXNRy+\nzXVlRCwDOpl5d0RcSXWh+RDwAnDz+C2rEXE7VakcoDr19IW6SD4IfAc4B9hOdZvrZFeAOsPY3uN+\nWf4V2EbDnB3MP2gnQP6RyUf9okYF0SIWxAANc/5hzg7mH7QTIH9PBeEnqSVJRRaEJKnIgpAkFVkQ\nkqQiC0KSVGRBSJKKLAhJUpEFIUkqsiAkSUUWhCSpyIKQJBVZEJKkIgtCklRkQUiSiiwISVKRBSFJ\nKrIgJElFFoQkqciCkCQVWRCSpCILQpJUZEFIkoosCElS0clNBkXEAuBOqkJZn5mrJiw/HbgXOB94\nB1iSmS9GxIXAg0AHGAHOA76amWsj4nZgKfBa/TIrMvN7fdgmSVIfTFoQEXEScBfwcWA38FREbMrM\nH3YNWwFsycxFERHAN4FPZOZfAx/qep2dwMNd692RmXf0Z1MkSf3U5BTTfOClzNyemQeAjcDCCWMu\nAR4HyMwEzo2I0QljPgH8KDN3ds0b6S22JGm6NSmIs4EdXdM763ndngUWAUTEfGAuMGfCmM8C354w\nb3lEPBMR90TEzMapJUnTrtE1iAZWAmsi4mngOWALcHB8YUScAnwauK1rnW8BX8vMTkR8HbgDuHmy\nbzQ6OqNPkQfD/IMzzNnB/IM27Pl70aQgdlEdEYybU897V2buA5aMT0fEy8C2riHXAH+VmXu71tnb\ntXwd8EiTwHv37msyrJVGR2eYf0CGOTuYf9BOhPy9aHKK6SnggogYi4hTgeuBzd0DImJmfZRARCwF\nnsjM/V1DFjPh9FJEnNU1uQh4vof8kqRpMukRRGYejIjlwKMcvs11a0QsAzqZeTdwMbAhIg4BL9B1\nqigiTqO6QP3bE156dURcDhwCXgGW9WF7JEl9MtLpdAad4Vh0hv0wz/yDMczZwfyDdgLk7+mOUT9J\nLUkqsiAkSUUWhCSpyIKQJBVZEJKkIgtCklRkQUiSiiwISVKRBSFJKrIgJElFFoQkqciCkCQVWRCS\npCILQpJUZEFIkoosCElSkQUhSSqyICRJRRaEJKnIgpAkFVkQkqQiC0KSVGRBSJKKTm4yKCIWAHdS\nFcr6zFw1YfnpwL3A+cA7wJLMfDEiLgQeBDrACHAe8NXMXBsRs+plY8ArwHWZ+VZftkqSNGWTHkFE\nxEnAXcDVwDxgcURcNGHYCmBLZl4G3ACsBcjMv87MD2XmFcA/An4KPFyvcxvwWGYG8DjwlT5sjySp\nT5qcYpoPvJSZ2zPzALARWDhhzCVUP+TJzATOjYjRCWM+AfwoM3fW0wuBDfXXG4Bre8gvSZomTQri\nbGBH1/TOel63Z4FFABExH5gLzJkw5rPAt7umz8jMPQCZ+SpwRvPYkqTp1ugaRAMrgTUR8TTwHLAF\nODi+MCJOAT5NdVrpSDpNvtHo6IwpxBw88w/OMGcH8w/asOfvRZOC2EV1RDBuTj3vXZm5D1gyPh0R\nLwPbuoZcA/xVZu7tmrcnIs7MzD0RcRbwWpPAe/fuazKslUZHZ5h/QIY5O5h/0E6E/L1ocorpKeCC\niBiLiFOB64HN3QMiYmZ9lEBELAWeyMz9XUMW897TS9SvcWP99Q3ApmOPL0maLpMWRGYeBJYDjwIv\nABszc2tELIuI366HXQw8HxFbqe52umV8/Yg4jeoC9cPvfWVWAf8yIhL4ONVpKklSS4x0Oo1O/bdF\nZ9gP88w/GMOcHcw/aCdA/pFe1vOT1JKkIgtCklRkQUiSiiwISVKRBSFJKrIgJElFFoQkqciCkCQV\nWRCSpCILQpJUZEFIkoosCElSkQUhSSqyICRJRRaEJKnIgpAkFVkQkqQiC0KSVGRBSJKKLAhJUpEF\nIUkqsiAkSUUnNxkUEQuAO6kKZX1mrpqw/HTgXuB84B1gSWa+WC+bCdwD/AZwqF72ZETcDiwFXqtf\nZkVmfm/qmyRJ6odJjyAi4iTgLuBqYB6wOCIumjBsBbAlMy8DbgDWdi1bA/xRZl4MXAZs7Vp2R2Ze\nUf+xHCSpRZocQcwHXsrM7QARsRFYCPywa8wlwO8BZGZGxLkRMQr8HPhIZt5YL/t74O2u9UamvAWS\npGnRpCDOBnZ0Te+kKo1uzwKLgD+PiPnAXGAO1Smln0TEfVRHD38J3JKZ79TrLY+Iz9fz/2NmvtXz\nlkiS+qpfF6lXArMi4mngS8AW4CBVAV0BfDMzrwB+BtxWr/Mt4LzMvBx4FbijT1kkSX3Q5AhiF9UR\nwbg59bx3ZeY+YMn4dES8DGwD3g/syMy/rBc9BNxar7O36yXWAY80yDIyOjqjwbD2Mv/gDHN2MP+g\nDXv+XjQ5gngKuCAixiLiVOB6YHP3gIiYGRGn1F8vBZ7IzP2ZuQfYEREX1kM/Dozf3XRW10ssAp6f\n2qZIkvpp0iOIzDwYEcuBRzl8m+vWiFgGdDLzbuBiYENEHAJeAG7ueol/D/z3ukC2ATfV81dHxOVU\n1yleAZb1aZskSX0w0ul0Bp1BktRCfpJaklRkQUiSiiwISVJRo2cxHW8Nnv30AeC/Ud1++z7gG5n5\nB8c7Z0lErAc+BezJzEuPMGYtcA3wU+DGzHzmOEY8qsnyR8TnqG9VBvYB/y4znzuOEY+qyd9/Pe4f\nA/8b+GxmPny88h1Nw/fOR4H/DJwC7M3Mq45fwqNr8N5p7X4LEBFzgD8EzqS6eWZdZq4tjGvd/tsk\ney/7buuOIBo+++lLwAv1h+yuAr4REW0pu/uoshdFxDXA+Zn561R3bv3+8QrW0FHzU92J9i/q5259\nneozLG0yWf7x99hK4I+PS6LmJnvvzAS+CXwqM38D+DfHK1hDk/3dt3m/Bfh74D9k5jzgnwJfmviz\np8X776TZ6WHfbV1B0PXsp8w8AIw/+6lbBxj/1MoM4G/r5zwNXGb+GfDGUYYspGp6MvNJYGZEnHk8\nsjUxWf7M/IuuR6L8BdWjWFqjwd8/wJepPrT52iTjjqsG2T8HfDczd9Xjf3JcgjXUIH9r91uAzHx1\n/GggM/dTPVh04vu7lftvk+y97LttLIjSs58mbshdwCURsZvqOVC3HKds/TBx+3bRsh+yx+ALwP8a\ndIhjERGzgWsz878yfA+LvBD4YER8PyKeqp9jNkyGZr+NiHOBy4EnJyxq/f57lOzdGu27bSyIJq6m\nerz4bOBDwDcj4h8MONMvlYi4iupDj7dONrZl7uS9mYepJMafbXYNsAD4akRcMNhIx2Qo9ts600NU\nDxbdP+g8x6JJ9mPZd9tYEJM++4lq4x4GyMwfAS8DE8+3tdUu4Jyu6dL2tVpEXArcDXw6Myc7ndM2\nHwY21s8L+9dUP6Q+PeBMTe0E/jgz/19m/i3wA6qnJA+L1u+39TWRh4D7M3NTYUhr998G2Y95321j\nQUz67CdgO/AJgPr834VUF2DaYoQj/8t0M/BvASLiSuDN+plVbXLE/BExF/gu8Pl6J2+jI+bPzPPq\nP/+Qamf6YmZOfH8N0tHeO5uAfx4R74uI04B/wnt/AVcbHC1/2/dbqH4z5ouZueYIy9u8/x41ey/7\nbisftVHf5rqGw7e5rux+9lNE/BrwB8Cv1av8XmZ+ezBp3ysiHgA+CvwqsAe4HTiVw8+tIiLuojpF\n8FPgpsx8ejBpf9Fk+SNiHdXDFbdT/SA4kJkTfz/IwDT5++8aey/wP1t0m2uT985/ovqX+EGqWxn/\ny2DS/qIG753W7rcAEfHPqI7KnqO6oN6h+m2ZY7R8/22SvZd9t5UFIUkavDaeYpIktYAFIUkqsiAk\nSUUWhCSpyIKQJBVZEJKkIgtCklRkQUiSiv4/pmvN4O/LWDwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4cbb9015f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "C_params3=np.linspace(1,2,21)\n",
    "scores3=list()\n",
    "for C in C_params3:\n",
    "    clf.C=C\n",
    "    tmp_score=cv.cross_val_score(clf,data,target,cv=5,n_jobs=-1)\n",
    "    scores3.append(np.mean(tmp_score))\n",
    "plt.scatter(C_params3,scores3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- ランダムシャッフリングして確認してみる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9600 (+/- 0.0249)\n"
     ]
    }
   ],
   "source": [
    "clf.C=1.6\n",
    "data_shuffle,target_shuffle=shuffle(data,target)\n",
    "score=cv.cross_val_score(clf,data_shuffle,target_shuffle,cv=5)\n",
    "print(\"Accuracy: {0:04.4f} (+/- {1:04.4f})\".format(score.mean(),score.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "あまりチューニングできてない..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEDCAYAAADeP8iwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFtdJREFUeJzt3X+MXeV95/H3EINS0ontVBeQMR4W2HwBrwL1VhabKtom\nZIuTTTHxHxRHyhpMXEuJsyi7qgCvKqQo1WJL8WKLRisbgxxUB7KUro3URpQSEbVSs6TYiB/Ot2wM\nxj+KmS5g7AYR1777xzmDr+fBnuOZa8/c6/dLGvme5zznzPPVwP3M85xzzwy0220kSep0zmQPQJI0\n9RgOkqSC4SBJKhgOkqSC4SBJKhgOkqTCtLE6RMRG4EvA/sz81An6rAO+APwzcGtmbq/bFwD3UYXQ\nxsxcVbfPBB4FhoDXgJsz88CEq5EkdUWTmcNDwA0n2hkRXwAuz8x/DSwH/mfdfg5wf33sXGBxRFxZ\nH3YX8FRmBvA0cPe4K5Akdd2Y4ZCZfwO8fZIuC4Hv131/CkyPiAuB+cArmbkrMw8Dj9R9R47ZVL/e\nBNw0vuFLkk6HblxzuBjY3bG9p247UTvAhZm5HyAz3wAu6MI4JEldcjouSA+M4xif4SFJU8iYF6Qb\n2Atc0rE9u247D5jzIe0Ab0TEhZm5PyIuAt5s8o3a7XZ7YGA82SNJZ7VTfuNsGg4DJzn5VuAbwKMR\ncR3wTv2m/0/AFRExBPwjcAuwuOOYW4FVwBJgS6NBDAwwPHyw4ZB7T6s1aH09qp9rA+vrda3W4Ckf\n0+RW1s3A7wC/ERGvA/dQzQrambk+M/8iIr4YEf+X6lbW2wAy80hErACe5NitrDvq064CfhgRS4Fd\nwM2nPHJJ0mkz0GOP7G73e7pbX2/q59rA+npdqzV4ystKfkJaklQwHCRJBcNBklQwHCRJBcNBklQw\nHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJ\nBcNBklQwHCRJBcNBklQwHCRJBcNBklSY1qRTRCwA7qMKk42ZuWrU/hnAg8DlwHvA0sx8ud53B/C1\nuusDmbm2br8HWAa8We9bmZk/mlg5kqRuGHPmEBHnAPcDNwBzgcURceWobiuBbZl5DbAEWFcfOxe4\nHfgt4FrgSxFxWcdxazJzXv1lMEjSFNFkWWk+8Epm7srMw8AjwMJRfa4GngbIzAQujYgWcBXw08x8\nPzOPAM8AizqOG5hoAZKk7msSDhcDuzu299RtnZ6nftOPiPnAHGA28CLwmYiYGRHnA18ELuk4bkVE\nbI+IByJi+jhrkCR1WaNrDg3cC6yNiOeAF4BtwJHM/HlErAL+Cjg00l4f8z3g25nZjojvAGuolqBO\nqtUa7NKQpybr6139XBtY39mmSTjspZoJjJhdt30gMw8CS0e2I+JVYGe97yHgobr9j6lnIZk53HGK\nDcATTQY8PHywSbee1GoNWl+P6ufawPp63XiCr8my0rPAFRExFBHnAbcAWzs7RMT0iDi3fr0MeCYz\nD9XbrfrfOcCXgc319kUdp1hEtQQlSZoCxpw5ZOaRiFgBPMmxW1l3RMRyoJ2Z66kuPG+KiKPASxy/\nPPRnEfEJ4DDw9cx8t25fHRHXAkeB14Dl3SpKkjQxA+12e7LHcCra/T71s77e1M+1gfX1ulZr8JTv\nDPUT0pKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoY\nDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkwrQmnSJiAXAf\nVZhszMxVo/bPAB4ELgfeA5Zm5sv1vjuAr9VdN2Tmurp9JvAoMAS8BtycmQcmWpAkaeLGnDlExDnA\n/cANwFxgcURcOarbSmBbZl4DLAFGAmAucDvwW8C1wO9FxGX1MXcBT2VmAE8Dd0+8HElSNzRZVpoP\nvJKZuzLzMPAIsHBUn6up3uDJzAQujYgWcBXw08x8PzOPAM8Ai+pjFgKb6tebgJsmVIkkqWuahMPF\nwO6O7T11W6fnqd/0I2I+MAeYDbwIfCYiZkbE+cAXgUvqYy7MzP0AmfkGcMF4i5AkdVejaw4N3Aus\njYjngBeAbcCRzPx5RKwC/go4NNJ+gnO0m3yjVmuwC8Oduqyvd/VzbWB9Z5sm4bCXaiYwYnbd9oHM\nPAgsHdmOiFeBnfW+h4CH6vY/5tgs5I2IuDAz90fERcCbTQY8PHywSbee1GoNWl+P6ufawPp63XiC\nr8my0rPAFRExFBHnAbcAWzs7RMT0iDi3fr0MeCYzD9XbrfrfOcCXgc31YVuBW+vXS4Atpzx6SdJp\nMebMITOPRMQK4EmO3cq6IyKWA+3MXE914XlTRBwFXqK6Q2nEn0XEJ4DDwNcz8926fRXww4hYCuwC\nbu5aVZKkCRlotxst9U8V7X6f+llfb+rn2sD6el2rNThwqsf4CWlJUsFwkCQVDAdJUsFwkCQVDAdJ\nUsFwkCQVDAdJUsFwkCQVDAdJUsFwkCQVDAdJUsFwkCQVDAdJUsFwkCQVDAdJUqFbf0NafeKtt97h\nzjt/zK5dH2do6ACrV3+OmTNnTPawJJ1hhoOOc+edP2bLlq8CA2zf3gYeZsOGL0/2sCSdYS4r6Ti7\ndn0cGPmjUQP1tqSzjeGg4wwNHQBG/nRsm6Ghd0/WXVKfcllJx1m9+nPAw/U1h3dZvfqzkz0kSZPA\ncNBxZs6c4TUGSS4rSZJKhoMkqdBoWSkiFgD3UYXJxsxcNWr/DOBB4HLgPWBpZr5c7/sWcDtwFHgB\nuC0zfxUR9wDLgDfr06zMzB9NvCRJ0kSNOXOIiHOA+4EbgLnA4oi4clS3lcC2zLwGWAKsq4+dBXwT\nmJeZn6IKo1s6jluTmfPqL4NBkqaIJstK84FXMnNXZh4GHgEWjupzNfA0QGYmcGlEtOp9HwE+FhHT\ngPOBfR3HDSBJmnKahMPFwO6O7T11W6fngUUAETEfmAPMzsx9wHeB14G9wDuZ+VTHcSsiYntEPBAR\n08dZgySpy7p1K+u9wNqIeI7qusI24Eh9LWIhMAQcAB6LiK9k5mbge8C3M7MdEd8B1lBdmzipVmuw\nS0Oemqyvd/VzbWB9Z5sm4bCXaiYwYnbd9oHMPAgsHdmOiJ3ATmABsDMz36rbHwc+DWzOzOGOU2wA\nnmgy4OHhg0269aRWa9D6elQ/1wbW1+vGE3xNlpWeBa6IiKGIOI/qgvLWzg4RMT0izq1fLwN+kpmH\nqJaTrouIj0bEAHA9sKPud1HHKRYBL57y6CVJp8WYM4fMPBIRK4AnOXYr646IWA60M3M9cBWwKSKO\nAi9RLw9l5v+JiMeolpkO1/+ur0+9OiKupbrF9TVgeVcrkySN20C73R6719TR7vepn/X1pn6uDayv\n17Vag6d8Z6ifkJYkFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAk\nFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAkFQwHSVLBcJAkFaY1\n6RQRC4D7qMJkY2auGrV/BvAgcDnwHrA0M1+u930LuB04CrwA3JaZv4qImcCjwBDwGnBzZh7oRlGS\npIkZc+YQEecA9wM3AHOBxRFx5ahuK4FtmXkNsARYVx87C/gmMC8zP0UVRrfUx9wFPJWZATwN3D3x\nciRJ3dBkWWk+8Epm7srMw8AjwMJRfa6meoMnMxO4NCJa9b6PAB+LiGnA+cDeun0hsKl+vQm4adxV\ndNlbb73DsmV/zu/+7l+zbNnjvP32O5M9JJ1mZ/PP3Nr/nPnznzjrah9Lk2Wli4HdHdt7qAKj0/PA\nIuBvI2I+MAeYnZnbIuK7wOvAL4EnM/Ov62MuyMz9AJn5RkRcMIE6uurOO3/Mli1fBQbYvr0NPMyG\nDV+e7GHpNDqbf+bWXtUOZ1ftY2l0zaGBe4G1EfEc1XWFbcCR+lrEQqrrCgeAxyLiK5m5+UPO0W7y\njVqtwS4N+cT27ZtJ9R8LwAD79s08I98Xzkx9k2mq1teNn/lUrW0sTWvv1fpOZjL/X5/qmoTDXqqZ\nwIjZHFsaAiAzDwJLR7YjYiewE1gA7MzMt+r2x4FPA5uB/RFxYWbuj4iLgDebDHh4+GCTbhMya9Zb\nVFlV/TYxa9bbZ+T7tlqDZ+T7TJapXN9Ef+ZTubaxNKm9l+s7mcn6f/1MG0/gNQmHZ4ErImII+Eeq\nC8qLOztExHTgl5l5OCKWAT/JzEMR8TpwXUR8FHgfuL4+H8BW4FZgFdVF7C2nPPrTZPXqzwEPs2vX\nxxkaepfVqz872UPSaXY2/8yt/WH27ZvJrFlvn1W1j2Wg3R57Nae+lXUtx25lvTcilgPtzFwfEddR\nXVQ+CrwE3D5yW2pE3EMVKIeplpu+VofIJ4AfApcAu6huZR3ralC7H1N9RL/+djain+vr59rA+npd\nqzU4MHav4zUKhynEcOhh/VxfP9cG1tfrxhMOfkJaklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNB\nklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQwHCRJBcNBklQw\nHCRJBcNBklQwHCRJBcNBklSY1qRTRCwA7qMKk42ZuWrU/hnAg8DlwHvA0sx8OSI+CTwKtIEB4DLg\njzJzXUTcAywD3qxPszIzf9SFmiRJEzRmOETEOcD9wPXAPuDZiNiSmT/v6LYS2JaZiyIigD8BPp+Z\n/wD8Zsd59gCPdxy3JjPXdKcUSVK3NFlWmg+8kpm7MvMw8AiwcFSfq4GnATIzgUsjojWqz+eBX2Tm\nno62gfENW5J0OjUJh4uB3R3be+q2Ts8DiwAiYj4wB5g9qs/vAz8Y1bYiIrZHxAMRMb3xqCVJp1Wj\naw4N3AusjYjngBeAbcCRkZ0RcS5wI3BXxzHfA76dme2I+A6wBrh9rG/Uag12achTk/X1rn6uDazv\nbNMkHPZSzQRGzK7bPpCZB4GlI9sR8Sqws6PLF4C/z8zhjmOGO/ZvAJ5oMuDh4YNNuvWkVmvQ+npU\nP9cG1tfrxhN8TZaVngWuiIihiDgPuAXY2tkhIqbXswMiYhnwTGYe6uiymFFLShFxUcfmIuDFUx69\nJOm0GHPmkJlHImIF8CTHbmXdERHLgXZmrgeuAjZFxFHgJTqWhyLifKqL0X8w6tSrI+Ja4CjwGrC8\nC/VIkrpgoN1uT/YYTkW736d+1teb+rk2sL5e12oNnvKdoX5CWpJUMBwkSQXDQZJUMBwkSQXDQZJU\nMBwkSQXDQZJUMBwkSQXDQZJUMBwkSQXDQZJUMBwkSQXDQZJUMBwkSQXDQZJUMBwkSQXDQZJUMBwk\nSQXDQZJUMBwkSQXDQZJUMBwkSQXDQZJUmNakU0QsAO6jCpONmblq1P4ZwIPA5cB7wNLMfDkiPgk8\nCrSBAeAy4I8yc11EzKz3DQGvATdn5oGuVCVJmpAxZw4RcQ5wP3ADMBdYHBFXjuq2EtiWmdcAS4B1\nAJn5D5n5m5k5D/i3wD8Dj9fH3AU8lZkBPA3c3YV6JEld0GRZaT7wSmbuyszDwCPAwlF9rqZ6gycz\nE7g0Ilqj+nwe+EVm7qm3FwKb6tebgJvGMX5J0mnQJBwuBnZ3bO+p2zo9DywCiIj5wBxg9qg+vw/8\noGP7gszcD5CZbwAXNB+2JOl0anTNoYF7gbUR8RzwArANODKyMyLOBW6kWko6kXaTb9RqDU5gmFOf\n9fWufq4NrO9s0yQc9lLNBEbMrts+kJkHgaUj2xHxKrCzo8sXgL/PzOGOtv0RcWFm7o+Ii4A3mwx4\nePhgk249qdUatL4e1c+1gfX1uvEEX5NlpWeBKyJiKCLOA24BtnZ2iIjp9eyAiFgGPJOZhzq6LOb4\nJSXqc9xav14CbDnl0UuSTosxwyEzjwArgCeBl4BHMnNHRCyPiD+ou10FvBgRO6juarpj5PiIOJ/q\nYvTjx5+ZVcB/iIgErqdampIkTQED7Xajpf6pot3vUz/r6039XBtYX69rtQYHTvUYPyEtSSoYDpKk\nguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEg\nSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkguEgSSoYDpKkwrQmnSJiAXAfVZhszMxVo/bPAB4ELgfe\nA5Zm5sv1vunAA8C/AY7W+34aEfcAy4A369OszMwfTbwkSdJEjTlziIhzgPuBG4C5wOKIuHJUt5XA\ntsy8BlgCrOvYtxb4i8y8CrgG2NGxb01mzqu/DAZJmiKazBzmA69k5i6AiHgEWAj8vKPP1cB/B8jM\njIhLI6IFvA98JjNvrff9C/Bux3EDE65AktR1TcLhYmB3x/YeqsDo9DywCPjbiJgPzAFmUy0j/VNE\nPEQ1a/gZcEdmvlcftyIivlq3/9fMPDDuSiRJXdOtC9L3AjMj4jngG8A24AhV+MwD/iQz5wG/BO6q\nj/kecFlmXgu8Aazp0lgkSRPUZOawl2omMGJ23faBzDwILB3ZjohXgZ3Ax4DdmfmzetdjwJ31McMd\np9gAPNFgLAOt1mCDbr3L+npXP9cG1ne2aTJzeBa4IiKGIuI84BZga2eHiJgeEefWr5cBz2Tmoczc\nD+yOiE/WXa8HRu5iuqjjFIuAFydWiiSpW8acOWTmkYhYATzJsVtZd0TEcqCdmeuBq4BNEXEUeAm4\nveMU/xn40zo8dgK31e2rI+JaqusSrwHLu1STJGmCBtrt9mSPQZI0xfgJaUlSwXCQJBUMB0lSodGz\nlSbbWM926mURMRv4PnAh1cX5DZm57uRH9Z76MSw/A/Zk5o2TPZ5uOtHzwyZ3VN0TEd+iusnkKPAC\ncFtm/mpyRzV+EbER+BKwPzM/VbfNBB4FhqhukLm5Fz+Ue4LaVgO/R/XEil9Q/fzePfFZKlN+5tDw\n2U697F+A/5KZc4F/B3yjz+obcQf1bcx96GTPD+tpETEL+CYwr36zmUZ1O3sve4jq/aTTXcBTmRnA\n08DdZ3xU3fFhtT0JzK0/cPwKDWub8uFAx7OdMvMwMPJsp76QmW9k5vb69SGqN5aLJ3dU3VXPjr5I\n9dt1X4mIj1M9P+whqJ4f1uS3sh7zEeBjETENOB/YN8njmZDM/Bvg7VHNC4FN9etNwE1ndFBd8mG1\nZeZTmXm03vw7qg8yj6kXwuHDnu3UV2+eIyLiUuBaoG+WJGr/A/hDoB/vm/5X1M8Pi4jnImJ9RPza\nZA+qWzJzH/Bd4HWqJyO8k5lPTe6oTosL6g/tkplvABdM8nhOl6XAXzbp2AvhcFaIiF+nerzIHfUM\noi9ExH+kWv/cTvUU3n57Eu/Jnh/W8+q/1bKQai1+FvDrEfGVyR3VGdF3v8hExH8DDmfm5ib9eyEc\nxny2U6+rp+uPAQ9n5pbJHk+X/TZwY0TsBH4AfDYivj/JY+qmPZTPD5s3iePpts8DOzPzrcw8AjwO\nfHqSx3Q67I+IC+GDR/u8OUb/nhIRt1It7TYO9l4IhzGf7dQHHgRezsy1kz2QbsvMlZk5JzMvo/rZ\nPZ2Z/2myx9UtJ3t+WJ94HbguIj4aEQNU9fXDBffRs9itwK316yVAL/+Sdlxt9d2efwjcmJnvNz5J\nLzw+oy5uLcduZb13kofUNRHx28BPqG4RbNdfffknUyPi31P93Y5+u5X1GqqL7R88P6wXb4M8kfpP\n+t4CHKZ6HP/X6ptDelJEbAZ+B/gNYD9wD/C/gf8FXALsorqV9Z3JGuN4naC2lcB5wP+ru/1dZn59\nrHP1RDhIks6sXlhWkiSdYYaDJKlgOEiSCoaDJKlgOEiSCoaDJKlgOEiSCoaDJKnw/wG7U6+9KgV5\naQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4cd00eefd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(min_C=(1,10,1),max_C=(1,10,1),step=(5,20,1))\n",
    "def cost_search(min_C=1,max_C=10,step=5):\n",
    "    params=np.linspace(min_C,max_C,step)\n",
    "    rslt=list()\n",
    "    for C in params:\n",
    "        clf.C=C\n",
    "        rslt.append(np.mean(cv.cross_val_score(clf,data,target,cv=5,n_jobs=-1)))\n",
    "    plt.scatter(params,rslt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# おまけ\n",
    "> cross_val_scoreを使わず実装するなら"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([  0,   1,   2,   5,   8,   9,  10,  11,  12,  13,  14,  15,  16,\n",
       "         17,  18,  19,  23,  24,  25,  27,  28,  29,  30,  31,  32,  33,\n",
       "         34,  35,  36,  37,  38,  40,  41,  42,  43,  44,  45,  46,  48,\n",
       "         49,  50,  53,  54,  55,  56,  57,  59,  60,  63,  64,  65,  66,\n",
       "         67,  68,  69,  70,  71,  73,  74,  75,  76,  77,  78,  79,  80,\n",
       "         81,  82,  83,  84,  85,  86,  87,  88,  91,  92,  93,  94,  95,\n",
       "         96,  97, 100, 101, 102, 103, 104, 105, 106, 107, 109, 110, 111,\n",
       "        115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 126, 127, 128,\n",
       "        129, 130, 131, 132, 134, 135, 136, 137, 139, 142, 143, 145, 146,\n",
       "        147, 148, 149]),\n",
       " array([  3,   4,   6,   7,  20,  21,  22,  26,  39,  47,  51,  52,  58,\n",
       "         61,  62,  72,  89,  90,  98,  99, 108, 112, 113, 114, 125, 133,\n",
       "        138, 140, 141, 144]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skf=cv.StratifiedKFold(target,n_folds=5,shuffle=True) # generaterを作成する\n",
    "for i in skf:pass\n",
    "i # 一個一個のgeneraterにはindexのリストが含まれている"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all:  [ 1.          0.9         0.96666667  1.          0.96666667]\n",
      "Accuracy: 0.9667 (+/- 0.0365)\n"
     ]
    }
   ],
   "source": [
    "score=list()\n",
    "for train_index,test_index in skf:\n",
    "    data_train=data[train_index]\n",
    "    data_test=data[test_index]\n",
    "    target_train=target[train_index]\n",
    "    target_test=target[test_index]\n",
    "    clf.fit(data_train,target_train)\n",
    "    rslt=clf.predict(data_test)\n",
    "    tmp_score=0\n",
    "    for i,j in enumerate(rslt):\n",
    "        if j==target_test[i]:tmp_score+=1\n",
    "    score.append(tmp_score/len(test_index))\n",
    "score=np.array(score)\n",
    "print(\"all: \",score)\n",
    "print(\"Accuracy: {0:04.4f} (+/- {1:04.4f})\".format(score.mean(),score.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# トレーニング/検証用データの分割\n",
    "\n",
    "> cv.train_test_split\n",
    "\n",
    "- マニュアルで二分割したい時"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "data_train,data_test,target_train,target_test=cv.train_test_split(\n",
    "                                                data,target,train_size=0.95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "142\n",
      "8\n",
      "142\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "print(len(data_train))\n",
    "print(len(data_test))\n",
    "print(len(target_train))\n",
    "print(len(target_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- train_sizeで0~1の範囲で分割する\n",
    "- 最初に出てくるものがtrain用データ\n",
    "- random_stateでseed固定も可能\n",
    "- 実は分割したい配列は１つや２つだけでなく、いくつでも渡せる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "clf.fit(data_train,target_train)\n",
    "target_pred=clf.predict(data_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>prediction</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fact</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0  1  2  3  4  5  6  7\n",
       "prediction  0  2  1  2  0  1  1  1\n",
       "fact        0  2  1  2  0  1  1  1"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(np.c_[target_pred,target_test],columns=[\"prediction\",\"fact\"]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "shuffleと同様に、random_stateを指定せずに分割すると、１行で実行しないとばらばらなseedでランダム分割されるので注意"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_train_NG,data_test_NG=cv.train_test_split(data,train_size=0.95)\n",
    "target_train_NG,target_test_NG=cv.train_test_split(target,train_size=0.95)#上とseedが違う可能性あり!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>prediction</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fact</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0  1  2  3  4  5  6  7\n",
       "prediction  0  1  0  2  2  0  0  2\n",
       "fact        1  1  0  2  0  0  2  2"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(data_train_NG,target_train_NG)\n",
    "target_pred_NG=clf.predict(data_test_NG)\n",
    "pd.DataFrame(np.c_[target_pred_NG,target_test_NG],columns=[\"prediction\",\"fact\"]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ここまでやってなんですが、CVを単独で使うケースは少ないです。\n",
    "次で紹介するGridSearchはGridSearchCVでまとめてやるほうが多い気がします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 終わり\n",
    "@y__sama"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}