05.Python深度學習入門-標準神經網路DNN做手寫辨識（MNIST）.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# 主題 01-1. 標準神經網路做手寫辨識\n\n我們終於要開始做生命中第一個神經網路...\n(修改自蔡炎龍老師的Deep Learning MOOC 教學)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 1. 初始準備\n\nKeras 可以用各種不同的深度學習套件當底層, 我們在此指定用 Tensorflow 以確保執行的一致性。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%env KERAS_BACKEND=tensorflow",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": "env: KERAS_BACKEND=tensorflow\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "再來是我們標準數據分析動作!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%matplotlib inline\nimport numpy as np\nimport matplotlib.pyplot as plt",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": "/home/nbuser/anaconda3_420/lib/python3.5/site-packages/matplotlib/font_manager.py:281: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n  'Matplotlib is building the font cache using fc-list. '\n",
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 2. 讀入 MNIST 數據庫\n\nMNIST 是有一堆 0-9 的手寫數字圖庫。有 6 萬筆訓練資料, 1 萬筆測試資料。它是 \"Modified\" 版的 NIST 數據庫, 原來的版本有更多資料。這個 Modified 的版本是由 LeCun, Cortes, 及 Burges 等人做的。可以參考這個數據庫的[原始網頁](http://yann.lecun.com/exdb/mnist/)。\n\nMNIST 可以說是 Deep Learning 最有名的範例, 它被 Deep Learning 大師 Hinton 稱為「機器學習的果蠅」。"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 2.1 由 Keras 讀入 MNIST"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Keras 很貼心的幫我們準備好 MNIST 數據庫, 我們可以這樣讀進來 (第一次要花點時間)。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from keras.datasets import mnist",
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": "/home/nbuser/anaconda3_420/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n  from ._conv import register_converters as _register_converters\nUsing TensorFlow backend.\n",
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "(x_train, y_train), (x_test, y_test) = mnist.load_data()",
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n11264000/11490434 [============================>.] - ETA: 0s",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "我們來看看訓練資料是不是 6 萬筆、測試資料是不是有 1 筆。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "len(x_train)",
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 5,
          "data": {
            "text/plain": "60000"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "len(x_test)",
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 6,
          "data": {
            "text/plain": "10000"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "特別要注意的是, 萬一在讀的過程中失敗, 你需要找到下載的部份數據集刪去, 然後在一個網路通𣈱的地方再下載一次。"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 2.2 數據庫的內容\n\n每筆輸入 (x) 就是一個手寫的 0-9 中一個數字的圖檔, 大小為 28x28。而輸出 (y) 當然就是「正確答案」。我們來看看編號 9487 的訓練資料。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "x_train[9487].shape",
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 7,
          "data": {
            "text/plain": "(28, 28)"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "因為是圖檔, 當然可以顯示出來!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "plt.imshow(x_train[9487], cmap='Greys')",
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 8,
          "data": {
            "text/plain": "<matplotlib.image.AxesImage at 0x7f9ce0d7c278>"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAADI1JREFUeJzt3W+oXPWdx/HPR5OCSasYctVgdW+36GJQNl2GsKAsLuUWswRjH1QasGS17O2DKFsssiJCfaAgy9puhU0lXUMTbdMWWtc8kN0GWXCDS3AMkthkd6tyt80mJDeoSQqBovnug3tSbuKdM5OZ82duvu8XhJk5vzP3fJjkkzMzv7nzc0QIQD6XtR0AQDsoP5AU5QeSovxAUpQfSIryA0lRfiApyg8kRfmBpJY0ebCVK1fG5ORkk4cEUpmZmdGJEyc8yL4jld/2XZK+J+lySf8cEU+X7T85OalutzvKIQGU6HQ6A+879NN+25dL+idJ6yStlrTR9uphfx6AZo3ymn+tpHci4r2I+L2kn0jaUE0sAHUbpfzXS/rtvNuHi23nsT1tu2u7Ozs7O8LhAFRplPIv9KbCJ34/OCK2RkQnIjoTExMjHA5AlUYp/2FJN8y7/VlJR0aLA6Apo5T/DUk32f6c7U9J+qqkXdXEAlC3oaf6IuIj2w9K+jfNTfVti4hfVZYMQK1GmuePiFckvVJRFgAN4uO9QFKUH0iK8gNJUX4gKcoPJEX5gaQoP5AU5QeSovxAUpQfSIryA0lRfiApyg8kRfmBpCg/kBTlB5Ki/EBSlB9IivIDSVF+ICnKDyTV6BLdaN6KFStKx0+ePFk6vnnz5tLxZ5999qIzYTxw5geSovxAUpQfSIryA0lRfiApyg8kRfmBpEaa57c9I+m0pI8lfRQRnSpC4eJ8+OGHPcciovS+tkvHt2zZUjrOPP/iVcWHfP4yIk5U8HMANIin/UBSo5Y/JP3S9pu2p6sIBKAZoz7tvz0ijti+RtJu2/8VEa/N36H4T2Fakm688cYRDwegKiOd+SPiSHF5XNJLktYusM/WiOhERGdiYmKUwwGo0NDlt73c9mfOXZf0JUlvVxUMQL1Gedp/raSXiqmiJZJ+HBH/WkkqALUbuvwR8Z6kP60wC3roN1f/3HPP9Rw7ffp01XHOs3v37tLxqampWo+P4THVByRF+YGkKD+QFOUHkqL8QFKUH0iKr+5eBE6dOlU6/vjjjzeU5JP27NlTOs5U3/jizA8kRfmBpCg/kBTlB5Ki/EBSlB9IivIDSTHPvwgsXbq0dHz16tU9xw4ePFh1HFwiOPMDSVF+ICnKDyRF+YGkKD+QFOUHkqL8QFLM8y8Cy5YtKx2///77e4498sgjVcc5z86dO0vHH3744Z5jV111VdVxcBE48wNJUX4gKcoPJEX5gaQoP5AU5QeSovxAUn3n+W1vk7Re0vGIuLXYtkLSTyVNSpqRdG9EfFBfTIyrd999t3T8zJkzPceY52/XIGf+H0q664Jtj0p6NSJukvRqcRvAItK3/BHxmqT3L9i8QdL24vp2SfdUnAtAzYZ9zX9tRByVpOLymuoiAWhC7W/42Z623bXdnZ2drftwAAY0bPmP2V4lScXl8V47RsTWiOhERGdiYmLIwwGo2rDl3yVpU3F9k6SXq4kDoCl9y297p6T/lPQntg/b/rqkpyVN2f61pKniNoBFpO88f0Rs7DH0xYqzYEgPPfRQz7Enn3yy9L4nT56sOg4WCT7hByRF+YGkKD+QFOUHkqL8QFKUH0iKr+6+BCxZ0vuv0XaDSbCYcOYHkqL8QFKUH0iK8gNJUX4gKcoPJEX5gaQoP5AU5QeSovxAUpQfSIryA0lRfiApyg8kRfmBpPh9/kvc+vXrS8dfeOGFkX7+2bNnS8f37t3bc2zDhg0jHRuj4cwPJEX5gaQoP5AU5QeSovxAUpQfSIryA0n1nee3vU3SeknHI+LWYtsTkv5G0myx22MR8UpdITG8Bx54oHT8xRdfHOnnX3ZZ+fnjmWee6Tk2NTVVet9ly5YNlQmDGeTM/0NJdy2w/bsRsab4Q/GBRaZv+SPiNUnvN5AFQINGec3/oO39trfZvrqyRAAaMWz5vy/p85LWSDoqqecLO9vTtru2u7Ozs712A9CwocofEcci4uOIOCvpB5LWluy7NSI6EdGZmJgYNieAig1Vftur5t38sqS3q4kDoCmDTPXtlHSnpJW2D0v6tqQ7ba+RFJJmJH2jxowAatC3/BGxcYHNz9eQBZeg119/vefYvn37Su97xx13VB0H8/AJPyApyg8kRfmBpCg/kBTlB5Ki/EBSlB9IivIDSVF+ICnKDyRF+YGkKD+QFOUHkqL8QFIs0X2Ju+6660rH+327Up1fvTY9PV06fvDgwdqODc78QFqUH0iK8gNJUX4gKcoPJEX5gaQoP5AU8/yXuJtvvrl0fN26daXjO3bsqDLOeVi+rV2c+YGkKD+QFOUHkqL8QFKUH0iK8gNJUX4gqb7z/LZvkLRD0nWSzkraGhHfs71C0k8lTUqakXRvRHxQX1TU4amnniodr3Oe/8yZM6XjBw4cKB2/7bbbqoyTziBn/o8kfSsibpH055I2214t6VFJr0bETZJeLW4DWCT6lj8ijkbEvuL6aUmHJF0vaYOk7cVu2yXdU1dIANW7qNf8ticlfUHSXknXRsRRae4/CEnXVB0OQH0GLr/tT0v6uaRvRsSpi7jftO2u7S6f5QbGx0Dlt71Uc8X/UUT8oth8zPaqYnyVpOML3TcitkZEJyI6/b4sEkBz+pbftiU9L+lQRHxn3tAuSZuK65skvVx9PAB1GeRXem+X9DVJB2y/VWx7TNLTkn5m++uSfiPpK/VERJ2WL19eOt5vOm3//v1DH/uKK64oHb/llluG/tnor2/5I2KPJPcY/mK1cQA0hU/4AUlRfiApyg8kRfmBpCg/kBTlB5Liq7uTu/LKK0vH77777tLxUeb5+1myhH+edeLMDyRF+YGkKD+QFOUHkqL8QFKUH0iK8gNJMZGKUvfdd1/p+AcflH9b+5YtW6qMgwpx5geSovxAUpQfSIryA0lRfiApyg8kRfmBpBwRjR2s0+lEt9tt7HhANp1OR91ut9dX7Z+HMz+QFOUHkqL8QFKUH0iK8gNJUX4gKcoPJNW3/LZvsP3vtg/Z/pXtvy22P2H7/2y/Vfz5q/rjAqjKIF/m8ZGkb0XEPtufkfSm7d3F2Hcj4h/qiwegLn3LHxFHJR0trp+2fUjS9XUHA1Cvi3rNb3tS0hck7S02PWh7v+1ttq/ucZ9p213b3dnZ2ZHCAqjOwOW3/WlJP5f0zYg4Jen7kj4vaY3mnhk8s9D9ImJrRHQiojMxMVFBZABVGKj8tpdqrvg/iohfSFJEHIuIjyPirKQfSFpbX0wAVRvk3X5Lel7SoYj4zrztq+bt9mVJb1cfD0BdBnm3/3ZJX5N0wPZbxbbHJG20vUZSSJqR9I1aEgKoxSDv9u+RtNDvB79SfRwATeETfkBSlB9IivIDSVF+ICnKDyRF+YGkKD+QFOUHkqL8QFKUH0iK8gNJUX4gKcoPJEX5gaQaXaLb9qyk/523aaWkE40FuDjjmm1cc0lkG1aV2f4oIgb6vrxGy/+Jg9vdiOi0FqDEuGYb11wS2YbVVjae9gNJUX4gqbbLv7Xl45cZ12zjmksi27Baydbqa34A7Wn7zA+gJa2U3/Zdtv/b9ju2H20jQy+2Z2wfKFYe7racZZvt47bfnrdthe3dtn9dXC64TFpL2cZi5eaSlaVbfezGbcXrxp/2275c0v9ImpJ0WNIbkjZGxMFGg/Rge0ZSJyJanxO2/ReSfidpR0TcWmz7e0nvR8TTxX+cV0fE341Jtick/a7tlZuLBWVWzV9ZWtI9kv5aLT52JbnuVQuPWxtn/rWS3omI9yLi95J+ImlDCznGXkS8Jun9CzZvkLS9uL5dc/94Gtcj21iIiKMRsa+4flrSuZWlW33sSnK1oo3yXy/pt/NuH9Z4Lfkdkn5p+03b022HWcC1xbLp55ZPv6blPBfqu3Jzky5YWXpsHrthVryuWhvlX2j1n3Gacrg9Iv5M0jpJm4untxjMQCs3N2WBlaXHwrArXletjfIflnTDvNuflXSkhRwLiogjxeVxSS9p/FYfPnZukdTi8njLef5gnFZuXmhlaY3BYzdOK163Uf43JN1k+3O2PyXpq5J2tZDjE2wvL96Ike3lkr6k8Vt9eJekTcX1TZJebjHLecZl5eZeK0ur5cdu3Fa8buVDPsVUxj9KulzStoh4qvEQC7D9x5o720tzi5j+uM1stndKulNzv/V1TNK3Jf2LpJ9JulHSbyR9JSIaf+OtR7Y7NffU9Q8rN597jd1wtjsk/YekA5LOFpsf09zr69Yeu5JcG9XC48Yn/ICk+IQfkBTlB5Ki/EBSlB9IivIDSVF+ICnKDyRF+YGk/h85G4rir6/+MAAAAABJRU5ErkJggg==\n",
            "text/plain": "<matplotlib.figure.Figure at 0x7f9ce0db10f0>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "我們人眼辨識就知道這是 1, 我們看答案是不是和我們想的一樣。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "y_train[9487]",
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 9,
          "data": {
            "text/plain": "1"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 2.3 輸入格式整理\n\n我們現在要用標準神經網路學學手寫辨識。原來的每筆數據是個 28x28 的矩陣 (array), 但標準神經網路只吃「平平的」, 也就是每次要 28x28=784 長的向量。因此我們要用 `reshape` 調校一下。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "x_train = x_train.reshape(60000, 784)\nx_test = x_test.reshape(10000, 784)",
      "execution_count": 10,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 2.4 輸出格式整理\n\n我們可能會想, 我們想學的函數是這樣的型式:\n\n$$\\hat{f} \\colon \\mathbb{R}^{784} \\to \\mathbb{R}$$\n\n其實這樣不太好! 為什麼呢? 比如說我們的輸入 x 是一張 0 的圖, 因為我們訓練的神經網路總會有點誤差, 所以可能會得到:\n\n$$\\hat{f}(x) = 0.5$$\n\n那這意思是有可能是 0, 也有可能是 1 嗎!!?? 可是 0 和 1 根本不像啊。換句話說分類的問題這樣做其實不合理!\n\n於是我們會做 \"1-hot enconding\", 也就是\n\n* 1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0]\n* 5 -> [0, 0, 0, 0, 0, 1, 0, 0, 0]\n\n等等。因為分類問題基本上都要做這件事, Keras 其實已幫我們準備好套件!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from keras.utils import np_utils",
      "execution_count": 11,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "y_train = np_utils.to_categorical(y_train,10)",
      "execution_count": 12,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "y_test = np_utils.to_categorical(y_test,10)",
      "execution_count": 13,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "我們來看看剛剛是 1 的 9487 號數據的答案。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "y_train[9487]",
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 14,
          "data": {
            "text/plain": "array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0.])"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "和我們想的一樣! 至此我們可以打造我們的神經網路了。"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 3. 打造第一個神經網路\n\n我們決定了我們的函數是\n\n$$\\hat{f} \\colon \\mathbb{R}^{784} \\to \\mathbb{R}^{10}$$\n\n這個樣子。而我們又說第一次要用標準神網路試試, 所以我們只需要再決定要幾個隱藏層、每層要幾個神經元, 用哪個激發函數就可以了。\n\n### 3.1 決定神經網路架構、讀入相關套件\n\n假如我們要這麼做:\n\n* 使用 <span style=\"color:red;\">2</span> 個 hidden layers\n* 每個 hidden layer 用 <span style=\"color:red;\">500</span> 個神經元\n* Activation Function 唯一指名 <span style=\"color:red;\">sigmoid</span>\n\n於是從 Keras 把相關套件讀進來。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from keras.models import Sequential\nfrom keras.layers import Dense, Activation\nfrom keras.optimizers import SGD",
      "execution_count": 15,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 3.2 建構我們的神經網路\n\n和以前做迴歸或機器學習一樣, 我們就打開個「函數學習機」。標準一層一層傳遞的神經網路叫 `Sequential`, 於是我們打開一個空的神經網路。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model = Sequential()",
      "execution_count": 16,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "我們每次用 `add` 去加一層, 從第一個隱藏層開始。而第一個隱藏層因為 Keras 當然猜不到輸入有幾個 features, 所以我們要告訴它。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.add(Dense(500, input_dim=784))\nmodel.add(Activation('sigmoid'))",
      "execution_count": 17,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "第二層 hidden layer 因為前面輸出是 500, 現在輸入是 500, 就不用再說了! 這裡的 500 只告訴 Keras, 我們第二層還是用 500!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.add(Dense(500))\nmodel.add(Activation('sigmoid'))",
      "execution_count": 18,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "輸出有 10 個數字, 所以輸出層的神經元是 10 個! 而如果我們的網路輸出是 \n\n$$(y_1, y_2, \\ldots, y_{10})$$\n\n我們還希望\n\n$$\\sum_{i=1}^{10} y_i = 1$$\n\n這可能嗎, 結果是很容易, 就用 `softmax` 當激發函數就可以!!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.add(Dense(10))\nmodel.add(Activation('softmax'))",
      "execution_count": 19,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "至此我們的第一個神經網路就建好了!"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 3.3 組裝\n\n和之前比較不一樣的是我們還要做 `compile` 才正式把我們的神經網路建好。你可以發現我們還需要做幾件事:\n\n* 決定使用的 loss function, 一般是 `mse`\n* 決定 optimizer, 我們用標準的 SGD\n* 設 learning rate\n\n為了一邊訓練一邊看到結果, 我們加設\n\n    metrics=['accuracy']\n    \n本行基本上和我們的神經網路功能沒有什麼關係。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.compile(loss='mse', optimizer=SGD(lr=0.1), metrics=['accuracy'])",
      "execution_count": 20,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 4. 檢視我們的神經網路"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "我們可以檢視我們神經網路的架構, 可以確認一下是不是和我們想像的一樣。"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 4.1 看 model 的 summary"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.summary()",
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": "_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\ndense_1 (Dense)              (None, 500)               392500    \n_________________________________________________________________\nactivation_1 (Activation)    (None, 500)               0         \n_________________________________________________________________\ndense_2 (Dense)              (None, 500)               250500    \n_________________________________________________________________\nactivation_2 (Activation)    (None, 500)               0         \n_________________________________________________________________\ndense_3 (Dense)              (None, 10)                5010      \n_________________________________________________________________\nactivation_3 (Activation)    (None, 10)                0         \n=================================================================\nTotal params: 648,010\nTrainable params: 648,010\nNon-trainable params: 0\n_________________________________________________________________\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 4.2 畫出結構圖\n\n要使用這個功能要安裝 `pydot` 及 `graphviz` 兩個套件, 請在終端機 (Anaconda Prompt) 安裝:\n\n    conda install pydot\n    conda install graphviz"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from keras.utils import plot_model\nplot_model(model, show_shapes=True, to_file='.\\data\\model01.png')",
      "execution_count": 22,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "![我的神經網路](model01.png)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 5. 訓練你的第一個神經網路\n\n恭喜! 我們完成了第一個神經網路。現在要訓練的時候, 你會發現不是像以前沒頭沒腦把訓練資料送進去就好。這裡我們還有兩件事要決定:\n\n* 一次要訓練幾筆資料 (`batch_size`), 我們就 100 筆調一次參數好了\n* 這 6 萬筆資料一共要訓練幾次 (`epochs`), 我們訓練個 20 次試試\n\n於是最精彩的就來了。你要有等待的心理準備..."
    },
    {
      "metadata": {
        "scrolled": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.fit(x_train, y_train, batch_size=100, epochs=20)",
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Epoch 1/20\n60000/60000 [==============================] - 11s - loss: 0.0834 - acc: 0.3414    \nEpoch 2/20\n60000/60000 [==============================] - 10s - loss: 0.0638 - acc: 0.6270    \nEpoch 3/20\n60000/60000 [==============================] - 11s - loss: 0.0459 - acc: 0.7645    \nEpoch 4/20\n60000/60000 [==============================] - 10s - loss: 0.0334 - acc: 0.8397    \nEpoch 5/20\n60000/60000 [==============================] - 10s - loss: 0.0257 - acc: 0.8739    \nEpoch 6/20\n60000/60000 [==============================] - 11s - loss: 0.0213 - acc: 0.8895    \nEpoch 7/20\n60000/60000 [==============================] - 11s - loss: 0.0186 - acc: 0.8991    \nEpoch 8/20\n60000/60000 [==============================] - 11s - loss: 0.0168 - acc: 0.9061    \nEpoch 9/20\n60000/60000 [==============================] - 11s - loss: 0.0154 - acc: 0.9119    \nEpoch 10/20\n60000/60000 [==============================] - 11s - loss: 0.0144 - acc: 0.9164    \nEpoch 11/20\n60000/60000 [==============================] - 11s - loss: 0.0135 - acc: 0.9210    \nEpoch 12/20\n60000/60000 [==============================] - 11s - loss: 0.0129 - acc: 0.9247    \nEpoch 13/20\n60000/60000 [==============================] - 11s - loss: 0.0122 - acc: 0.9278    \nEpoch 14/20\n60000/60000 [==============================] - 12s - loss: 0.0117 - acc: 0.9308    \nEpoch 15/20\n60000/60000 [==============================] - 11s - loss: 0.0112 - acc: 0.9338    \nEpoch 16/20\n60000/60000 [==============================] - 11s - loss: 0.0108 - acc: 0.9364    \nEpoch 17/20\n60000/60000 [==============================] - 11s - loss: 0.0105 - acc: 0.9382    \nEpoch 18/20\n60000/60000 [==============================] - 11s - loss: 0.0101 - acc: 0.9404    \nEpoch 19/20\n60000/60000 [==============================] - 12s - loss: 0.0098 - acc: 0.9429    \nEpoch 20/20\n60000/60000 [==============================] - 13s - loss: 0.0095 - acc: 0.9448    \n",
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "execution_count": 23,
          "data": {
            "text/plain": "<keras.callbacks.History at 0x7f9ce0d7c978>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 6. 試用我們的結果\n\n我們來用比較炫的方式來看看可愛的神經網路學習成果。對指令有問題可以參考我們之前的 MOOC 影片教學。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from ipywidgets import interact_manual",
      "execution_count": 24,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "我們 \"predict\" 放的是我們神經網路的學習結果。這裡用 `predict_classes` 會讓我們 Keras 選 10 個輸出機率最大的那類。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "predict = model.predict_classes(x_test)",
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": " 9984/10000 [============================>.] - ETA: 0s",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "不要忘了我們的 `x_test` 每筆資料已經換成 784 維的向量, 我們要整型回 28x28 的矩陣才能當成圖形顯示出來!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "def test(測試編號):\n    plt.imshow(x_test[測試編號].reshape(28,28), cmap=\"Greys\")\n    print(\"神經網路判斷為:\", predict[測試編號])",
      "execution_count": 26,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "interact_manual(test, 測試編號 = (0, 9999));",
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f7c85478d60d401fb1a6968c437b46cc",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": "interactive(children=(IntSlider(value=4999, description='測試編號', max=9999), Button(description='Run Interact', …"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "到底測試資料總的狀況如何呢? 我們可以給我們神經網路「考一下試」。"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "score = model.evaluate(x_test, y_test)",
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": " 9536/10000 [===========================>..] - ETA: 0s",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print('測試資料的 loss:', score[0])\nprint('測試資料正確率:', score[1])",
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": "測試資料的 loss: 0.010674761578021571\n測試資料正確率: 0.9332\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## 7. 訓練好的神經網路存起來!\n\n如果對訓練成果滿意, 我們當然不想每次都再訓練一次! 我們可以把神經網路的架構和訓練好的參數都存起來, 以供日後使用!"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "之前還沒裝 pyh5 要在終端機 (Anaconda Prompt) 下安裝:\n    \n    conda install h5py"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model_json = model.to_json()\nopen('./data/handwriting_model_architecture.json', 'w').write(model_json)\nmodel.save_weights('./data/handwriting_model_weights.h5')",
      "execution_count": 30,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### reference\n- 蔡炎龍老師的Deep Learning MOOC 教學"
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.4",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}