lab-04-1-Multi-variable-Linear-Regression--Regression-eager.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "qa55_des11wj",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 03 - Multi-variable Linear Regression\n",
    "\n",
    "<img width=\"200\" src=\"https://i.imgur.com/hbPVe1T.png\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1622,
     "status": "ok",
     "timestamp": 1549861753492,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "BbOQ25cb11wl",
    "outputId": "182e949f-9183-4095-84bb-3edcf3f768fa",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.12.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "tf.enable_eager_execution()\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eXdPAm1y11xP",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Multi-variable linear regression\n",
    "Predicting exam score - regression using three inputs (x1, x2, x3)\n",
    "\n",
    "x1 (quiz 1) | x2 (quiz 2) | x3 (mid 1) | Y (final)\n",
    "---- | ---- | ----| ----\n",
    "73 | 80 | 75 | 152\n",
    "93 | 88 | 93 | 185\n",
    "89 | 91 | 90 | 180\n",
    "96 | 98 | 100 | 196\n",
    "73 | 66 | 70 | 142\n",
    "\n",
    "\n",
    "Test Scores for General Psychology ( https://goo.gl/g2T8Kp )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZCc0H5qT11xU",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Matrix multiplication\n",
    "\n",
    "## dot product(=scalar product, 내적)\n",
    "<img src=\"https://www.mathsisfun.com/algebra/images/matrix-multiply-a.svg\" >\n",
    "\n",
    "\n",
    "https://www.mathsisfun.com/algebra/matrix-multiplying.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HOpVHPfX11xV",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Multi-feature regression\n",
    "\n",
    "### Hypothesis\n",
    "\n",
    "$$ H(x) = w x + b $$\n",
    "\n",
    "$$ H(x_1, x_2, x_3) = w_1 x_1 + w_2 x_2 + w_3 x_3 + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jiJiS73i11xW",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hypothesis using matrix\n",
    "\n",
    "$$ H(x_1, x_2, x_3) = \\underline{w_1 x_1 + w_2 x_2 + w_3 x_3} + b $$\n",
    "\n",
    "$$ w_1 x_1 + w_2 x_2 + w_3 x_3 $$ \n",
    "\n",
    "$$ \\begin{pmatrix} w_{ 1 } & w_{ 2 } & w_{ 3 } \\end{pmatrix}\\cdot \\begin{pmatrix} x_{ 1 } \\\\ x_{ 2 } \\\\ x_{ 3 } \\end{pmatrix} $$\n",
    "\n",
    "$$ WX $$ (W, X 는 matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Qyo_v2q211xX",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hypothesis without b\n",
    "\n",
    "$$ H(x_1, x_2, x_3) = w_1 x_1 + w_2 x_2 + w_3 x_3 + b$$\n",
    "\n",
    "$$ = b + w_1 x_1 + w_2 x_2 + w_3 x_3 $$\n",
    "\n",
    "$$ = \\begin{pmatrix} b & x_{ 1 } & x_{ 2 } & x_{ 3 } \\end{pmatrix}\\cdot \\begin{pmatrix} 1 \\\\ w_{ 1 } \\\\ w_{ 2 } \\\\ w_{ 3 } \\end{pmatrix} $$\n",
    "\n",
    "$$ = XW $$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "70JryWbi11xZ",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hypothesis using matrix \n",
    "\n",
    "### Many x instances\n",
    "\n",
    "$$ \\begin{pmatrix} x_{ 11 } & x_{ 12 } & x_{ 13 } \\\\ x_{ 21 } & x_{ 22 } & x_{ 23 } \\\\ x_{ 31 } & x_{ 32 } & x_{ 33 }\\\\ x_{ 41 } & x_{ 42 } & x_{ 43 }\\\\ x_{ 51 } & x_{ 52 } & x_{ 53 }\\end{pmatrix} \\cdot \\begin{pmatrix} w_{ 1 } \\\\ w_{ 2 } \\\\ w_{ 3 } \\end{pmatrix}=\\begin{pmatrix} x_{ 11 }w_{ 1 }+x_{ 12 }w_{ 2 }+x_{ 13 }w_{ 3 } \\\\ x_{ 21 }w_{ 1 }+x_{ 22 }w_{ 2 }+x_{ 23 }w_{ 3 }\\\\ x_{ 31 }w_{ 1 }+x_{ 32 }w_{ 2 }+x_{ 33 }w_{ 3 } \\\\ x_{ 41 }w_{ 1 }+x_{ 42 }w_{ 2 }+x_{ 43 }w_{ 3 } \\\\ x_{ 51 }w_{ 1 }+x_{ 52 }w_{ 2 }+x_{ 53 }w_{ 3 } \\end{pmatrix} $$\n",
    "\n",
    "$$ [5, 3] \\cdot [3, 1] = [5, 1] $$\n",
    "\n",
    "$$ H(X) = XW $$\n",
    "\n",
    "5는 데이터(instance)의 수, 3은 변수(feature)의 수, 1은 결과"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_PmX23KO11xc",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hypothesis using matrix (n output)\n",
    "\n",
    "$$ [n, 3] \\cdot [?, ?] = [n, 2] $$\n",
    "\n",
    "$$ H(X) = XW $$\n",
    "\n",
    "* n은 데이터(instance)의 개수, 2는 결과 값의 개수로 주어진다.\n",
    "* 이때, W [?, ?] ⇒ [3, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TdLqWOL811xd",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# WX vs XW\n",
    "\n",
    "### Theory (Lecture) :\n",
    " $$ H(x) = Wx + b  $$\n",
    "\n",
    "### TensorFlow (Implementation) :\n",
    "\n",
    "$$ H(X) = XW $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4hzOpk9w11xf",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Simple Example (2 variables)\n",
    "\n",
    "x1 | x2 | y\n",
    "---- | ---- | ----\n",
    "1  |  0  |  1\n",
    "0  |  2  |  2\n",
    "3  |  0  |  3\n",
    "0  |  4  |  4\n",
    "5  |  0  |  5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "b645aPucdmMz"
   },
   "outputs": [],
   "source": [
    "tf.set_random_seed(0)  # for reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2134,
     "status": "ok",
     "timestamp": 1549859056786,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "XdIC0_tqpmc1",
    "outputId": "441c62e7-dac2-48ee-affe-fcd6e346cce1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    0 | 335.280823 |    -4.0663 |     1.1220 |  -6.065215\n",
      "   50 |  76.037262 |    -0.8001 |     1.6209 |  -4.978779\n",
      "  100 |  18.959265 |     0.7151 |     1.8781 |  -4.429109\n",
      "  150 |   6.310240 |     1.4125 |     2.0104 |  -4.134423\n",
      "  200 |   3.445082 |     1.7284 |     2.0768 |  -3.961648\n",
      "  250 |   2.743659 |     1.8667 |     2.1075 |  -3.847750\n",
      "  300 |   2.525401 |     1.9225 |     2.1184 |  -3.762738\n",
      "  350 |   2.417754 |     1.9402 |     2.1181 |  -3.692262\n",
      "  400 |   2.337300 |     1.9403 |     2.1114 |  -3.629400\n",
      "  450 |   2.264998 |     1.9325 |     2.1008 |  -3.570778\n",
      "  500 |   2.196329 |     1.9213 |     2.0881 |  -3.514729\n",
      "  550 |   2.130126 |     1.9085 |     2.0741 |  -3.460409\n",
      "  600 |   2.066037 |     1.8953 |     2.0595 |  -3.407385\n",
      "  650 |   2.003917 |     1.8819 |     2.0444 |  -3.355424\n",
      "  700 |   1.943679 |     1.8686 |     2.0293 |  -3.304398\n",
      "  750 |   1.885258 |     1.8555 |     2.0141 |  -3.254230\n",
      "  800 |   1.828595 |     1.8425 |     1.9990 |  -3.204873\n",
      "  850 |   1.773636 |     1.8297 |     1.9841 |  -3.156293\n",
      "  900 |   1.720330 |     1.8171 |     1.9693 |  -3.108468\n",
      "  950 |   1.668625 |     1.8048 |     1.9547 |  -3.061379\n",
      " 1000 |   1.618474 |     1.7926 |     1.9403 |  -3.015011\n"
     ]
    }
   ],
   "source": [
    "x1_data = [1, 0, 3, 0, 5]\n",
    "x2_data = [0, 2, 0, 4, 0]\n",
    "y_data  = [1, 2, 3, 4, 5]\n",
    "\n",
    "W1 = tf.Variable(tf.random_uniform([1], -10.0, 10.0))\n",
    "W2 = tf.Variable(tf.random_uniform([1], -10.0, 10.0))\n",
    "b  = tf.Variable(tf.random_uniform([1], -10.0, 10.0))\n",
    "\n",
    "learning_rate = tf.Variable(0.001)\n",
    "\n",
    "for i in range(1000+1):\n",
    "    with tf.GradientTape() as tape:\n",
    "        hypothesis = W1 * x1_data + W2 * x2_data + b\n",
    "        cost = tf.reduce_mean(tf.square(hypothesis - y_data))\n",
    "    W1_grad, W2_grad, b_grad = tape.gradient(cost, [W1, W2, b])\n",
    "    W1.assign_sub(learning_rate * W1_grad)\n",
    "    W2.assign_sub(learning_rate * W2_grad)\n",
    "    b.assign_sub(learning_rate * b_grad)\n",
    "\n",
    "    if i % 50 == 0:\n",
    "        print(\"{:5} | {:10.6f} | {:10.4f} | {:10.4f} | {:10.6f}\".format(\n",
    "          i, cost.numpy(), W1.numpy()[0], W2.numpy()[0], b.numpy()[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uNHVQS-311xk",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Simple Example (2 variables with Matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3709,
     "status": "ok",
     "timestamp": 1549859058376,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "1SWH2f6vnjCO",
    "outputId": "e37b1216-8540-4aa1-ea9e-3a52761230c0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    0 |  36.403778 |    -0.6231 |    -0.3508 |  -0.961774\n",
      "   50 |   9.372900 |     0.2914 |     0.1682 |  -0.557764\n",
      "  100 |   2.639858 |     0.7060 |     0.4867 |  -0.347756\n",
      "  150 |   0.825069 |     0.8912 |     0.6846 |  -0.235665\n",
      "  200 |   0.284990 |     0.9721 |     0.8088 |  -0.174012\n",
      "  250 |   0.106844 |     1.0062 |     0.8873 |  -0.138953\n",
      "  300 |   0.042677 |     1.0195 |     0.9372 |  -0.118279\n",
      "  350 |   0.018044 |     1.0241 |     0.9690 |  -0.105598\n",
      "  400 |   0.008188 |     1.0250 |     0.9893 |  -0.097477\n",
      "  450 |   0.004138 |     1.0246 |     1.0022 |  -0.092026\n",
      "  500 |   0.002439 |     1.0239 |     1.0104 |  -0.088173\n",
      "  550 |   0.001710 |     1.0230 |     1.0156 |  -0.085299\n",
      "  600 |   0.001384 |     1.0223 |     1.0188 |  -0.083036\n",
      "  650 |   0.001227 |     1.0217 |     1.0207 |  -0.081161\n",
      "  700 |   0.001142 |     1.0212 |     1.0218 |  -0.079538\n",
      "  750 |   0.001088 |     1.0207 |     1.0224 |  -0.078080\n",
      "  800 |   0.001046 |     1.0203 |     1.0227 |  -0.076735\n",
      "  850 |   0.001011 |     1.0199 |     1.0227 |  -0.075468\n",
      "  900 |   0.000980 |     1.0196 |     1.0226 |  -0.074258\n",
      "  950 |   0.000949 |     1.0192 |     1.0225 |  -0.073089\n",
      " 1000 |   0.000921 |     1.0189 |     1.0222 |  -0.071954\n"
     ]
    }
   ],
   "source": [
    "x_data = [\n",
    "    [1., 0., 3., 0., 5.],\n",
    "    [0., 2., 0., 4., 0.]\n",
    "]\n",
    "y_data  = [1, 2, 3, 4, 5]\n",
    "\n",
    "W = tf.Variable(tf.random_uniform([1, 2], -1.0, 1.0))\n",
    "b = tf.Variable(tf.random_uniform([1], -1.0, 1.0))\n",
    "\n",
    "learning_rate = tf.Variable(0.001)\n",
    "\n",
    "for i in range(1000+1):\n",
    "    with tf.GradientTape() as tape:\n",
    "        hypothesis = tf.matmul(W, x_data) + b # (1, 2) * (2, 5) = (1, 5)\n",
    "        cost = tf.reduce_mean(tf.square(hypothesis - y_data))\n",
    "\n",
    "        W_grad, b_grad = tape.gradient(cost, [W, b])\n",
    "        W.assign_sub(learning_rate * W_grad)\n",
    "        b.assign_sub(learning_rate * b_grad)\n",
    "    \n",
    "    if i % 50 == 0:\n",
    "        print(\"{:5} | {:10.6f} | {:10.4f} | {:10.4f} | {:10.6f}\".format(\n",
    "            i, cost.numpy(), W.numpy()[0][0], W.numpy()[0][1], b.numpy()[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GUs7qbTR11xs",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hypothesis without b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3701,
     "status": "ok",
     "timestamp": 1549859058377,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "114nIzJq5EuC",
    "outputId": "7b4a0292-0128-4a94-dfac-725f1311ecae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    0 |  16.019751 |    -0.1985 |     0.3424 |    -0.6835\n",
      "   50 |   5.635924 |     0.0582 |     0.6809 |    -0.1215\n",
      "  100 |   2.141112 |     0.1997 |     0.8238 |     0.2356\n",
      "  150 |   0.862825 |     0.2786 |     0.8808 |     0.4641\n",
      "  200 |   0.367090 |     0.3227 |     0.9015 |     0.6112\n",
      "  250 |   0.167513 |     0.3468 |     0.9074 |     0.7064\n",
      "  300 |   0.085210 |     0.3593 |     0.9082 |     0.7684\n",
      "  350 |   0.050615 |     0.3649 |     0.9074 |     0.8090\n",
      "  400 |   0.035731 |     0.3663 |     0.9067 |     0.8359\n",
      "  450 |   0.029064 |     0.3651 |     0.9063 |     0.8539\n",
      "  500 |   0.025846 |     0.3624 |     0.9064 |     0.8661\n",
      "  550 |   0.024085 |     0.3587 |     0.9069 |     0.8746\n",
      "  600 |   0.022948 |     0.3544 |     0.9076 |     0.8807\n",
      "  650 |   0.022085 |     0.3497 |     0.9086 |     0.8852\n",
      "  700 |   0.021348 |     0.3449 |     0.9097 |     0.8887\n",
      "  750 |   0.020676 |     0.3400 |     0.9109 |     0.8916\n",
      "  800 |   0.020042 |     0.3350 |     0.9121 |     0.8940\n",
      "  850 |   0.019434 |     0.3301 |     0.9133 |     0.8960\n",
      "  900 |   0.018848 |     0.3252 |     0.9146 |     0.8979\n",
      "  950 |   0.018280 |     0.3203 |     0.9158 |     0.8997\n",
      " 1000 |   0.017730 |     0.3155 |     0.9171 |     0.9013\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# 앞의 코드에서 bias(b)를 행렬에 추가\n",
    "x_data = [\n",
    "    [1., 1., 1., 1., 1.], # bias(b)\n",
    "    [1., 0., 3., 0., 5.], \n",
    "    [0., 2., 0., 4., 0.]\n",
    "]\n",
    "y_data  = [1, 2, 3, 4, 5]\n",
    "\n",
    "W = tf.Variable(tf.random_uniform([1, 3], -1.0, 1.0)) # [1, 3]으로 변경하고, b 삭제\n",
    "\n",
    "learning_rate = 0.001\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "\n",
    "for i in range(1000+1):\n",
    "    with tf.GradientTape() as tape:\n",
    "        hypothesis = tf.matmul(W, x_data) # b가 없다\n",
    "        cost = tf.reduce_mean(tf.square(hypothesis - y_data))\n",
    "\n",
    "    grads = tape.gradient(cost, [W])\n",
    "    optimizer.apply_gradients(grads_and_vars=zip(grads,[W]))\n",
    "    if i % 50 == 0:\n",
    "        print(\"{:5} | {:10.6f} | {:10.4f} | {:10.4f} | {:10.4f}\".format(\n",
    "            i, cost.numpy(), W.numpy()[0][0], W.numpy()[0][1], W.numpy()[0][2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ivo2V3xK11xx"
   },
   "source": [
    "# Custom Gradient\n",
    "* tf.train.GradientDescentOptimizer(): optimizer\n",
    "* optimizer.apply_gradients(): update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6779,
     "status": "ok",
     "timestamp": 1549859061465,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "QkPqZbkDiqe7",
    "outputId": "89754887-b544-4908-9e2d-2d9ab0dabc40"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch | cost\n",
      "    0 |  21.726822\n",
      "   50 |   0.258689\n",
      "  100 |   0.176868\n",
      "  150 |   0.120926\n",
      "  200 |   0.082678\n",
      "  250 |   0.056528\n",
      "  300 |   0.038649\n",
      "  350 |   0.026424\n",
      "  400 |   0.018067\n",
      "  450 |   0.012352\n",
      "  500 |   0.008445\n",
      "  550 |   0.005774\n",
      "  600 |   0.003948\n",
      "  650 |   0.002699\n",
      "  700 |   0.001845\n",
      "  750 |   0.001262\n",
      "  800 |   0.000863\n",
      "  850 |   0.000590\n",
      "  900 |   0.000403\n",
      "  950 |   0.000276\n",
      " 1000 |   0.000189\n"
     ]
    }
   ],
   "source": [
    "# Multi-variable linear regression (1)\n",
    "\n",
    "X = tf.constant([[1., 2.], \n",
    "                 [3., 4.]])\n",
    "y = tf.constant([[1.5], [3.5]])\n",
    "\n",
    "W = tf.Variable(tf.random_normal([2, 1]))\n",
    "b = tf.Variable(tf.random_normal([1]))\n",
    "\n",
    "# Create an optimizer\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)\n",
    "\n",
    "n_epoch = 1000+1\n",
    "print(\"epoch | cost\")\n",
    "for i in range(n_epoch):\n",
    "    # Use tf.GradientTape() to record the gradient of the cost function\n",
    "    with tf.GradientTape() as tape:\n",
    "        y_pred = tf.matmul(X, W) + b\n",
    "        cost = tf.reduce_mean(tf.square(y_pred - y))\n",
    "\n",
    "    # calculates the gradients of the loss\n",
    "    grads = tape.gradient(cost, [W, b])\n",
    "    \n",
    "    # updates parameters (W and b)\n",
    "    optimizer.apply_gradients(grads_and_vars=zip(grads, [W, b]))\n",
    "    if i % 50 == 0:\n",
    "        print(\"{:5} | {:10.6f}\".format(i, cost.numpy()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kGyEI57HLRWl"
   },
   "source": [
    "# Predicting exam score\n",
    "regression using three inputs (x1, x2, x3)\n",
    "\n",
    "x1 (quiz 1) | x2 (quiz 2) | x3 (mid 1) | Y (final)\n",
    "---- | ---- | ----| ----\n",
    "73 | 80 | 75 | 152\n",
    "93 | 88 | 93 | 185\n",
    "89 | 91 | 90 | 180\n",
    "96 | 98 | 100 | 196\n",
    "73 | 66 | 70 | 142"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ny-s4htZ4Lzo"
   },
   "outputs": [],
   "source": [
    "tf.set_random_seed(0)  # for reproducibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F1bSir9448BC"
   },
   "source": [
    "\n",
    "```python\n",
    "x1 = [ 73.,  93.,  89.,  96.,  73.]\n",
    "x2 = [ 80.,  88.,  91.,  98.,  66.]\n",
    "x3 = [ 75.,  93.,  90., 100.,  70.]\n",
    "Y  = [152., 185., 180., 196., 142.]\n",
    "\n",
    "# weights\n",
    "w1 = tf.Variable(10.)\n",
    "w2 = tf.Variable(10.)\n",
    "w3 = tf.Variable(10.)\n",
    "b  = tf.Variable(10.)\n",
    "\n",
    "hypothesis = w1 * x1 +  w2 * x2 + w3 * x3 + b        \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2003,
     "status": "ok",
     "timestamp": 1549861013021,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "ICTAblBh-3FO",
    "outputId": "f364e89a-1df9-4635-a1ae-7a4b10a56659"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    0 | 5793889.5000\n",
      "   50 |   64291.1484\n",
      "  100 |     715.2902\n",
      "  150 |       9.8462\n",
      "  200 |       2.0152\n",
      "  250 |       1.9252\n",
      "  300 |       1.9210\n",
      "  350 |       1.9177\n",
      "  400 |       1.9145\n",
      "  450 |       1.9114\n",
      "  500 |       1.9081\n",
      "  550 |       1.9050\n",
      "  600 |       1.9018\n",
      "  650 |       1.8986\n",
      "  700 |       1.8955\n",
      "  750 |       1.8923\n",
      "  800 |       1.8892\n",
      "  850 |       1.8861\n",
      "  900 |       1.8829\n",
      "  950 |       1.8798\n",
      " 1000 |       1.8767\n"
     ]
    }
   ],
   "source": [
    "# data and label\n",
    "x1 = [ 73.,  93.,  89.,  96.,  73.]\n",
    "x2 = [ 80.,  88.,  91.,  98.,  66.]\n",
    "x3 = [ 75.,  93.,  90., 100.,  70.]\n",
    "Y  = [152., 185., 180., 196., 142.]\n",
    "\n",
    "# weights\n",
    "w1 = tf.Variable(10.)\n",
    "w2 = tf.Variable(10.)\n",
    "w3 = tf.Variable(10.)\n",
    "b  = tf.Variable(10.)\n",
    "\n",
    "learning_rate = 0.000001\n",
    "\n",
    "for i in range(1000+1):\n",
    "    # tf.GradientTape() to record the gradient of the cost function\n",
    "    with tf.GradientTape() as tape:\n",
    "        hypothesis = w1 * x1 +  w2 * x2 + w3 * x3 + b\n",
    "        cost = tf.reduce_mean(tf.square(hypothesis - Y))\n",
    "    # calculates the gradients of the cost\n",
    "    w1_grad, w2_grad, w3_grad, b_grad = tape.gradient(cost, [w1, w2, w3, b])\n",
    "    \n",
    "    # update w1,w2,w3 and b\n",
    "    w1.assign_sub(learning_rate * w1_grad)\n",
    "    w2.assign_sub(learning_rate * w2_grad)\n",
    "    w3.assign_sub(learning_rate * w3_grad)\n",
    "    b.assign_sub(learning_rate * b_grad)\n",
    "\n",
    "    if i % 50 == 0:\n",
    "      print(\"{:5} | {:12.4f}\".format(i, cost.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ptg1sl3w8uFt"
   },
   "source": [
    "## Multi-variable linear regression (1)\n",
    "*  random  초기화: tf.random_normal()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1992,
     "status": "ok",
     "timestamp": 1549861572665,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "LhczGchq_zzQ",
    "outputId": "37949eab-e23e-4ef3-997a-e0e02f63daa3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    0 |   11325.9121\n",
      "   50 |     135.3618\n",
      "  100 |      11.1817\n",
      "  150 |       9.7940\n",
      "  200 |       9.7687\n",
      "  250 |       9.7587\n",
      "  300 |       9.7489\n",
      "  350 |       9.7389\n",
      "  400 |       9.7292\n",
      "  450 |       9.7194\n",
      "  500 |       9.7096\n",
      "  550 |       9.6999\n",
      "  600 |       9.6903\n",
      "  650 |       9.6806\n",
      "  700 |       9.6709\n",
      "  750 |       9.6612\n",
      "  800 |       9.6517\n",
      "  850 |       9.6421\n",
      "  900 |       9.6325\n",
      "  950 |       9.6229\n",
      " 1000 |       9.6134\n"
     ]
    }
   ],
   "source": [
    "# data and label\n",
    "x1 = [ 73.,  93.,  89.,  96.,  73.]\n",
    "x2 = [ 80.,  88.,  91.,  98.,  66.]\n",
    "x3 = [ 75.,  93.,  90., 100.,  70.]\n",
    "Y  = [152., 185., 180., 196., 142.]\n",
    "\n",
    "# random weights\n",
    "w1 = tf.Variable(tf.random_normal([1]))\n",
    "w2 = tf.Variable(tf.random_normal([1]))\n",
    "w3 = tf.Variable(tf.random_normal([1]))\n",
    "b  = tf.Variable(tf.random_normal([1]))\n",
    "\n",
    "learning_rate = 0.000001\n",
    "\n",
    "for i in range(1000+1):\n",
    "    # tf.GradientTape() to record the gradient of the cost function\n",
    "    with tf.GradientTape() as tape:\n",
    "        hypothesis = w1 * x1 +  w2 * x2 + w3 * x3 + b\n",
    "        cost = tf.reduce_mean(tf.square(hypothesis - Y))\n",
    "    # calculates the gradients of the cost\n",
    "    w1_grad, w2_grad, w3_grad, b_grad = tape.gradient(cost, [w1, w2, w3, b])\n",
    "    \n",
    "    # update w1,w2,w3 and b\n",
    "    w1.assign_sub(learning_rate * w1_grad)\n",
    "    w2.assign_sub(learning_rate * w2_grad)\n",
    "    w3.assign_sub(learning_rate * w3_grad)\n",
    "    b.assign_sub(learning_rate * b_grad)\n",
    "\n",
    "    if i % 50 == 0:\n",
    "      print(\"{:5} | {:12.4f}\".format(i, cost.numpy()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7w0kEX9C82D8"
   },
   "source": [
    "## Multi-variable linear regression (2)\n",
    "\n",
    "* Matrix 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2082,
     "status": "ok",
     "timestamp": 1549861764291,
     "user": {
      "displayName": "Sean-June Lee",
      "photoUrl": "https://lh3.googleusercontent.com/-j96Pvw9LbEw/AAAAAAAAAAI/AAAAAAACQxw/aY8i4PoItwc/s64/photo.jpg",
      "userId": "12576390002043678893"
     },
     "user_tz": -540
    },
    "id": "mmNogkEuQe5K",
    "outputId": "025477a5-868f-413d-fb4c-678f9510a6be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch | cost\n",
      "    0 |  5455.5903\n",
      "  100 |    31.7443\n",
      "  200 |    30.9326\n",
      "  300 |    30.7894\n",
      "  400 |    30.6468\n",
      "  500 |    30.5055\n",
      "  600 |    30.3644\n",
      "  700 |    30.2242\n",
      "  800 |    30.0849\n",
      "  900 |    29.9463\n",
      " 1000 |    29.8081\n",
      " 1100 |    29.6710\n",
      " 1200 |    29.5348\n",
      " 1300 |    29.3989\n",
      " 1400 |    29.2641\n",
      " 1500 |    29.1299\n",
      " 1600 |    28.9961\n",
      " 1700 |    28.8634\n",
      " 1800 |    28.7313\n",
      " 1900 |    28.5997\n",
      " 2000 |    28.4689\n"
     ]
    }
   ],
   "source": [
    "data = np.array([\n",
    "    # X1,   X2,    X3,   y\n",
    "    [ 73.,  80.,  75., 152. ],\n",
    "    [ 93.,  88.,  93., 185. ],\n",
    "    [ 89.,  91.,  90., 180. ],\n",
    "    [ 96.,  98., 100., 196. ],\n",
    "    [ 73.,  66.,  70., 142. ]\n",
    "], dtype=np.float32)\n",
    "\n",
    "# slice data\n",
    "X = data[:, :-1]\n",
    "y = data[:, [-1]]\n",
    "\n",
    "W = tf.Variable(tf.random_normal([3, 1]))\n",
    "b = tf.Variable(tf.random_normal([1]))\n",
    "\n",
    "learning_rate = 0.000001\n",
    "\n",
    "# hypothesis, prediction function\n",
    "def predict(X):\n",
    "    return tf.matmul(X, W) + b\n",
    "\n",
    "print(\"epoch | cost\")\n",
    "\n",
    "n_epochs = 2000\n",
    "for i in range(n_epochs+1):\n",
    "    # tf.GradientTape() to record the gradient of the cost function\n",
    "    with tf.GradientTape() as tape:\n",
    "        cost = tf.reduce_mean((tf.square(predict(X) - y)))\n",
    "\n",
    "    # calculates the gradients of the loss\n",
    "    W_grad, b_grad = tape.gradient(cost, [W, b])\n",
    "\n",
    "    # updates parameters (W and b)\n",
    "    W.assign_sub(learning_rate * W_grad)\n",
    "    b.assign_sub(learning_rate * b_grad)\n",
    "    \n",
    "    if i % 100 == 0:\n",
    "        print(\"{:5} | {:10.4f}\".format(i, cost.numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mt9zvM2OaljB"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.368576 ],\n",
       "       [ 2.1047728],\n",
       "       [-1.4229954]], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eCTzJDjH2k4z"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.1783521], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cs-KBzAHCf2u"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=383121, shape=(5, 1), dtype=float32, numpy=\n",
       "array([[160.38487],\n",
       "       [178.98064],\n",
       "       [184.08965],\n",
       "       [194.17314],\n",
       "       [138.03304]], dtype=float32)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.matmul(X, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9FgoGhG8GxEW"
   },
   "source": [
    "## predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0ZcVkvquG0g5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[152.0, 185.0, 180.0, 196.0, 142.0]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y # labels, 실제값"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f_YW4xsrGZAE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[160.38487],\n",
       "       [178.98064],\n",
       "       [184.08965],\n",
       "       [194.17314],\n",
       "       [138.03304]], dtype=float32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(X).numpy() # prediction, 예측값"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QrXNF_NSHZGo"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[189.66275],\n",
       "       [186.46652]], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 새로운 데이터에 대한 예측\n",
    "\n",
    "predict([[ 89.,  95.,  92.],[ 84.,  92.,  85.]]).numpy() "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "lab-04-1-Multi-variable-Linear-Regression-eager.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}