Computer Vision (Python).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ![upcode](images/front.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1><center>Why Self-Driving Cars?</center></h1>\n",
    "<br>\n",
    "<center><img src=\"images/autonomous_car.png\" alt=\"Autonomous Car\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/blue_car.png\" alt=\"Blue Car\"></center>\n",
    "\n",
    "# Cars are a good thing\n",
    "\n",
    "<center><img src=\"images/chuck.gif\" alt=\"Cars are good\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Cars helps us move\n",
    "# 1 billion cars worldwide\n",
    "# Transportation\n",
    "# Shape the cities, roads, world..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/manila.png\" alt=\"Manila Traffic Jam\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3>\n",
    "<ul>\n",
    "  <li>Filipinos spend 16 days a year on traffic</li>\n",
    "  <ul>\n",
    "    <li>Loosing 2,663 sgd inconme</li>\n",
    "  </ul>\n",
    "  <li>Comuters in Singapore spend 30 mins in traffic</li>\n",
    "  <ul>\n",
    "    <li>19 looking for parking spot</li>\n",
    "  </ul>\n",
    "</ul>\n",
    "</h3>\n",
    "<br>\n",
    "<br>\n",
    "<h4><div align=\"right\">source: straitstimes.com</div></h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/bkk.jpg\" alt=\"Bkk Traffic Jam\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3>\n",
    "<ul>\n",
    "  <li>24,000 deaths anually in Thailand</li>\n",
    "  <ul>\n",
    "    <li>Loses 3 to 5 % of its GDP</li>\n",
    "  </ul>\n",
    "  <li>10,000 injured each year in SG</li>\n",
    "  <ul>\n",
    "    <li>160 people die</li>\n",
    "  </ul>\n",
    "</ul>\n",
    "</h3>\n",
    "<br>\n",
    "<br>\n",
    "<h4><div align=\"right\">sources: World Health Organization and police.gov.sg</div></h4>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Road fatality one of the top 10 global causes of deaths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1><center>But... How?</center></h1>\n",
    "<center><img src=\"images/wtf.png\" alt=\"Wtf?\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/selfdrivingcar.png\" alt=\"Self Driving Car\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1><center>Why Computer Vision?</center></h1>\n",
    "<br>\n",
    "<center><img src=\"images/computer_vision.gif\" alt=\"Computer Vision\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/cameras.png\" alt=\"Cameras\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3>\n",
    "<ul>\n",
    "  <li>Cisco Visual Networking Index:</li>\n",
    "  <ul>\n",
    "    <li>5 million years to watch one month - 2021</li>\n",
    "    <li>82% of global traffic</li>\n",
    "    <li>Video surveillance traffic increased 72%</li>\n",
    "    <li>Virtual and augmented reality traffic will increase 20 times</li>\n",
    "  </ul>\n",
    "</ul>\n",
    "</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/youtube.png\" alt=\"Youtube\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Computer vision engineer: \n",
    "<br>\n",
    "\n",
    "## IT job that will be most in demand in 2020 *\n",
    "## More than 190k sgd salary in the US **\n",
    "\n",
    "<br>\n",
    "<h4><div align=\"right\">*source: techproresearch.com</div></h4>\n",
    "<h4><div align=\"right\">**source: indeed.com</div></h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# First rule of presentations:\n",
    "# Don't demo!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h1><center>Let's demo!</center></h1>\n",
    "<center><img src=\"images/homer.gif\" alt=\"Crazy Homer\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center> <video controls src=\"data/singapore_drive.mp4\" /> </center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import os\n",
    "import numpy as np\n",
    "from moviepy.editor import VideoFileClip\n",
    "from collections import deque\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "code_folding": [
     0,
     2,
     10,
     12,
     14,
     16,
     23,
     33,
     35,
     59,
     71,
     81
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def convert_to_hls(image):\n",
    "    return cv2.cvtColor(image, cv2.COLOR_RGB2HLS)\n",
    "def select_white(image):\n",
    "    converted = convert_to_hls(image)\n",
    "    # white color mask\n",
    "    lower = np.uint8([  0, 110,   0])\n",
    "    upper = np.uint8([255, 180, 250])\n",
    "    white_mask = cv2.inRange(converted, lower, upper)\n",
    "    masked = cv2.bitwise_and(image, image, mask = white_mask)\n",
    "    return masked\n",
    "def convert_to_grayscale(image):\n",
    "    return cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n",
    "def smoothing(image, kernel_size=15):\n",
    "    return cv2.GaussianBlur(image, (kernel_size, kernel_size), 0)\n",
    "def detect_edges(image, low_threshold=10, high_threshold=100):\n",
    "    return cv2.Canny(image, low_threshold, high_threshold)\n",
    "def filter_region(image, vertices):\n",
    "    mask = np.zeros_like(image)\n",
    "    if len(mask.shape)==2:\n",
    "        cv2.fillPoly(mask, vertices, 255)\n",
    "    else:\n",
    "        cv2.fillPoly(mask, vertices, (255,)*mask.shape[2])\n",
    "    return cv2.bitwise_and(image, mask)\n",
    "def select_region(image):\n",
    "    rows, cols = image.shape[:2]\n",
    "    bottom_left  = [int(cols*0.2), int(rows*0.95)]\n",
    "    top_left     = [int(cols*0.5), int(rows*0.65)]\n",
    "    bottom_right = [int(cols*0.95), int(rows*0.95)]\n",
    "    top_right    = [int(cols*0.65), int(rows*0.65)] \n",
    "    \n",
    "    vertices = np.array([[bottom_left, top_left, top_right, bottom_right]], dtype=np.int32)\n",
    "    \n",
    "    return filter_region(image, vertices)\n",
    "def hough_lines(image):\n",
    "    return cv2.HoughLinesP(image, rho=1, theta=np.pi/180, threshold=5, minLineLength=5, maxLineGap=200)\n",
    "def average_slope_intercept(lines):\n",
    "    left_lines    = [] \n",
    "    left_weights  = [] \n",
    "    right_lines   = [] \n",
    "    right_weights = []\n",
    "    \n",
    "    for line in lines:\n",
    "        for x1, y1, x2, y2 in line:\n",
    "            if x2==x1:\n",
    "                continue\n",
    "            slope = (y2-y1)/(x2-x1)\n",
    "            intercept = y1 - slope*x1\n",
    "            length = np.sqrt((y2-y1)**2+(x2-x1)**2)\n",
    "            if slope < 0:\n",
    "                left_lines.append((slope, intercept))\n",
    "                left_weights.append((length))\n",
    "            else:\n",
    "                right_lines.append((slope, intercept))\n",
    "                right_weights.append((length))\n",
    "     \n",
    "    left_lane  = np.dot(left_weights,  left_lines) /np.sum(left_weights)  if len(left_weights) >0 else None\n",
    "    right_lane = np.dot(right_weights, right_lines)/np.sum(right_weights) if len(right_weights)>0 else None\n",
    "    \n",
    "    return left_lane, right_lane\n",
    "def calculate_line_points(y1, y2, line):\n",
    "    if line is None:\n",
    "        return None\n",
    "    \n",
    "    slope, intercept = line\n",
    "    \n",
    "    x1 = int((y1 - intercept)/slope)\n",
    "    x2 = int((y2 - intercept)/slope)\n",
    "    y1 = int(y1)\n",
    "    y2 = int(y2)\n",
    "    \n",
    "    return ((x1, y1), (x2, y2))\n",
    "def lane_lines(image, lines):\n",
    "    left_lane, right_lane = average_slope_intercept(lines)\n",
    "    \n",
    "    y1 = image.shape[0]\n",
    "    y2 = y1*0.7         \n",
    "\n",
    "    left_line  = calculate_line_points(y1, y2, left_lane)\n",
    "    right_line = calculate_line_points(y1, y2, right_lane)\n",
    "    \n",
    "    return left_line, right_line\n",
    "def draw_lane_lines(image, lines, color=[0, 0, 255], thickness=20):\n",
    "    \n",
    "    line_image = np.zeros_like(image)\n",
    "    for line in lines:\n",
    "        if line is not None:\n",
    "            cv2.line(line_image, *line,  color, thickness)\n",
    "    return cv2.addWeighted(image, 1.0, line_image, 0.9, 0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class LanesDetector:\n",
    "    def __init__(self):\n",
    "        self.left_lines  = deque(maxlen=50)\n",
    "        self.right_lines = deque(maxlen=50)\n",
    "        \n",
    "    def mean_line(self, line, lines):\n",
    "        if line is not None:\n",
    "            lines.append(line)\n",
    "\n",
    "        if len(lines)>0:\n",
    "            line = np.mean(lines, axis=0, dtype=np.int32)\n",
    "            line = tuple(map(tuple, line))\n",
    "        return line\n",
    "\n",
    "    def process(self, image):\n",
    "        white        = select_white(image)\n",
    "        gray         = convert_to_grayscale(white)\n",
    "        smooth_gray  = smoothing(gray)\n",
    "        edges        = detect_edges(smooth_gray)\n",
    "        regions      = select_region(edges)\n",
    "        lines        = hough_lines(regions)\n",
    "        left_line, right_line = lane_lines(image, lines)\n",
    "\n",
    "        left_line  = self.mean_line(left_line,  self.left_lines)\n",
    "        right_line = self.mean_line(right_line, self.right_lines)\n",
    "\n",
    "        return draw_lane_lines(image, (left_line, right_line))\n",
    "    \n",
    "def process_video(video_input, video_output):\n",
    "    detector = LanesDetector()\n",
    "\n",
    "    clip = VideoFileClip(os.path.join('data', video_input))\n",
    "    processed = clip.fl_image(detector.process)\n",
    "    processed.write_videofile(os.path.join('data', video_output), audio=False)\n",
    "\n",
    "%time process_video('singapore_drive.mp4', 'detected_lanes.mp4')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center> <video controls src=\"data/detected_lanes.mp4\" /> </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1><center>But... How?</center></h1>\n",
    "<center><img src=\"images/wtf.png\" alt=\"Wtf?\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1><center>Learn the ways of Computer Vision you must...</center></h1>\n",
    "\n",
    "<center><img src=\"images/yoda.png\" alt=\"yoda\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# The tool\n",
    "\n",
    "<center><img src=\"images/opencv.png\" alt=\"opencv\">\n",
    "\n",
    "<h3><a href=\"https://docs.opencv.org\">docs.opencv.org</a></h3></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import os\n",
    "import numpy as np\n",
    "from moviepy.editor import VideoFileClip\n",
    "from collections import deque\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# What's an image?\n",
    "\n",
    "<center><img src=\"images/philosophator.png\" alt=\"philosophator\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/matrix.png\" alt=\"matrix\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def show_image(image, cmap=None):\n",
    "    plt.figure(figsize=(11,11))\n",
    "    if len(image.shape)==2:\n",
    "        cmap = 'gray'\n",
    "    else:\n",
    "        image=cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "    plt.imshow(image, cmap)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.show()\n",
    "\n",
    "#Read the image with OpenCV\n",
    "image = cv2.imread('data/test_image.png')\n",
    "\n",
    "show_image(image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "image.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "image[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Masks:\n",
    "## Matrix with some pixel values to zero and others to non zero\n",
    "## Output of algorithms will be used as a mask\n",
    "## Used to select parts of the image\n",
    "\n",
    "<center><img src=\"images/mask.png\" alt=\"mask\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def select_white(image): \n",
    "    # white color mask\n",
    "    lower = np.uint8([200, 200, 200])\n",
    "    upper = np.uint8([255, 255, 255])\n",
    "    white_mask = cv2.inRange(image, lower, upper)\n",
    "   \n",
    "    masked = cv2.bitwise_and(image, image, mask = white_mask)\n",
    "    return masked\n",
    "\n",
    "show_image(select_white(image))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Images Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/codification.png\" alt=\"Color Representations\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def convert_to_hsv(image):\n",
    "    return cv2.cvtColor(image, cv2.COLOR_RGB2HSV)\n",
    "\n",
    "show_image(convert_to_hsv(image))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def convert_to_hls(image):\n",
    "    return cv2.cvtColor(image, cv2.COLOR_RGB2HLS)\n",
    "\n",
    "show_image(convert_to_hls(image))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def select_white(image):\n",
    "    converted = convert_to_hls(image)\n",
    "    # white color mask\n",
    "    lower = np.uint8([  0, 110,   0])\n",
    "    upper = np.uint8([255, 180, 250])\n",
    "    white_mask = cv2.inRange(converted, lower, upper)\n",
    "    masked = cv2.bitwise_and(image, image, mask = white_mask)\n",
    "    return masked\n",
    "\n",
    "white_selection = select_white(image)\n",
    "show_image(white_selection) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Canny Edges\n",
    "\n",
    "## A multi-stage algorithm:\n",
    "\n",
    "#### 1. Gaussian filter\n",
    "#### 2. Finds edge gradient and direction for each pixel\n",
    "\n",
    "<img src=\"images/gradient.png\" alt=\"Gradient\">\n",
    "\n",
    "#### 3. Pixels are checked for local maximum in its neighborhood in the direction of gradient.\n",
    "#### 4. Hysteresis Thresholding: Uses minVal and maxVal as threshold of intensity gradient to finally detect the edges. Those in between are selected based on their connectivity.\n",
    "\n",
    "<center><a href=\"https://docs.opencv.org/3.4.2/dd/d1a/group__imgproc__feature.html#ga04723e007ed888ddf11d9ba04e2232de\">cv2.Canny()</a></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Canny Edges\n",
    "\n",
    "### Good For:\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>Edge detection</li>\n",
    "  <li>Preprocessing of images prior to lines or shapes detection</li>\n",
    "</ul>\n",
    "</h4>\n",
    "\n",
    "### Major Drawbacks:\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>Edge detection is sensitive to noise in the image</li>\n",
    "  <li>General thresholding problems apply</li>\n",
    "</ul>\n",
    "</h4>\n",
    "    \n",
    "    \n",
    "\n",
    "### Important tips:\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>The smoothing directly affects results</li>\n",
    "  <li>Larger blurry kernels are more useful for detecting larger smother edges</li>\n",
    "</ul>\n",
    "</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Convert to grayscale\n",
    "\n",
    "<h3><center> Since Canny measures the gradient </center></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def convert_to_gray_scale(image):\n",
    "    return cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n",
    "\n",
    "gray_selection = convert_to_gray_scale(white_selection)\n",
    "show_image(gray_selection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Gaussian Blur\n",
    "## Remove Noise from Images\n",
    "## Blur images using Gussian filter\n",
    "\n",
    "<center><a href=\"https://docs.opencv.org/3.4.2/d4/d86/group__imgproc__filter.html#gaabe8c836e97159a9193fb0b11ac52cf1\">cv2.GaussianBlur()</a></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Gaussian Blur\n",
    "\n",
    "<center><img src=\"images/convolution.jpg\" alt=\"convolution\"> <img src=\"images/gaussian.png\" alt=\"gaussian\"> </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    " # Gaussian Blur\n",
    " \n",
    " ## Edges: intensity changes rapidly\n",
    " ## Making edges smother to reduce noise\n",
    " ## Kernel size (smaller values if effect is similar) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def smoothing(image, kernel_size=15):\n",
    "    return cv2.GaussianBlur(image, (kernel_size, kernel_size), 0)\n",
    "\n",
    "smooth_image =  smoothing(gray_selection)\n",
    "show_image(smooth_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Canny Edges (OpenCV)\n",
    "\n",
    "### Selection based on pixel gradient :\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>Higher than the upper threshold: Accepted</li>\n",
    "  <li>Below the lower threshold: Rejected</li>\n",
    "  <li>Between the two: Accepted if connected to a pixel that is above the upper threshold </li>\n",
    "</ul>\n",
    "</h4>\n",
    "\n",
    "### Important tips:\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>Recommended a upper:lower ratio between 2:1 and 3:1</li>\n",
    "  <li>Use trials and errors</li>\n",
    "</ul>\n",
    "</h4>\n",
    "\n",
    "<h3><center><a href=\"https://docs.opencv.org/3.4.2/da/d22/tutorial_py_canny.html\">Canny Edge Detection OpenCV Tutorial</a></center></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def detect_edges(image, low_threshold=10, high_threshold=100):\n",
    "    return cv2.Canny(image, low_threshold, high_threshold)\n",
    "\n",
    "edges_image =  detect_edges(smooth_image)\n",
    "show_image(edges_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Region Of Interest (ROI)\n",
    "\n",
    "<br>\n",
    "<h3><center>Exclude the rest of the image by applying a mask</center><h3>\n",
    "\n",
    "<center><img src=\"images/roi.png\" alt=\"roi\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0,
     8
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def filter_region(image, vertices):\n",
    "    mask = np.zeros_like(image)\n",
    "    if len(mask.shape)==2:\n",
    "        cv2.fillPoly(mask, vertices, 255)\n",
    "    else:\n",
    "        cv2.fillPoly(mask, vertices, (255,)*mask.shape[2]) # in case, the input image has a channel dimension        \n",
    "    return cv2.bitwise_and(image, mask)\n",
    "  \n",
    "def select_region(image):\n",
    "    rows, cols = image.shape[:2]\n",
    "    bottom_left  = [int(cols*0.2), int(rows*0.95)]\n",
    "    top_left     = [int(cols*0.5), int(rows*0.65)]\n",
    "    bottom_right = [int(cols*0.95), int(rows*0.95)]\n",
    "    top_right    = [int(cols*0.65), int(rows*0.65)] \n",
    "    vertices = np.array([[bottom_left, top_left, top_right, bottom_right]], dtype=np.int32)\n",
    "    #image = cv2.line(image, tuple(bottom_left), tuple(top_left), (255,0,0), thickness=5)\n",
    "    #image = cv2.line(image, tuple(bottom_right), tuple(top_right), (255,0,0), thickness=5)\n",
    "    return filter_region(image, vertices)\n",
    "\n",
    "roi_image = select_region(edges_image)\n",
    "show_image(roi_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#  Hough Transform\n",
    "\n",
    "## Detect any shape (mathematical form)\n",
    "## Uses a voting procedure (acumulator)\n",
    "\n",
    "\n",
    "<h3><center><a href=\"https://docs.opencv.org/3.4.2/d6/d10/tutorial_py_houghlines.html\">Hough Line Transform OpenCV Tutorial</a></center></h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#  Hough Transform: Line\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>rho: Distance resolution of the accumulator in pixels.</li>\n",
    "  <li>theta: Angle resolution of the accumulator in radians.</li>\n",
    "  <li>threshold: Accumulator threshold parameter. Only those lines are returned that get enough votes (> `threshold`)</li>\n",
    "  <li>minLineLength: Minimum line length. Line segments shorter than that are rejected</li>\n",
    "  <li>maxLineGap: Maximum allowed gap between points on the same line to link them</li>\n",
    "</ul>\n",
    "</h4>\n",
    "<h3><center><a href=\"https://docs.opencv.org/3.4.2/dd/d1a/group__imgproc__feature.html#ga8618180a5948286384e3b7ca02f6feeb\">cv2.HoughLinesP()</a></center></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def draw_lines(image, lines, color=[0, 255, 0], thickness=2, make_copy=True):\n",
    "    # the lines returned by cv2.HoughLinesP has the shape (-1, 1, 4)\n",
    "    if make_copy:\n",
    "        image = np.copy(image) # don't want to modify the original\n",
    "    for line in lines:\n",
    "        for x1,y1,x2,y2 in line:\n",
    "            cv2.line(image, (x1, y1), (x2, y2), color, thickness)\n",
    "    return image\n",
    "def hough_lines(image):\n",
    "    return cv2.HoughLinesP(image, rho=1, theta=np.pi/180, threshold=5, minLineLength=5, maxLineGap=200)\n",
    "\n",
    "lines = hough_lines(roi_image)\n",
    "image_with_lines = draw_lines(image, lines)\n",
    "show_image(image_with_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Average lines\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>Multiple lines are detected for each lane</li>\n",
    "  <li>Only partially recognized</li>\n",
    "  <li>Extrapolate line to cover the full lane</li>\n",
    "  <li>Two lanes:</li>\n",
    "  <ul>\n",
    "      <li>left with positive slope</li>\n",
    "      <li>right with negative slope</li>\n",
    "  </ul>\n",
    "</ul>\n",
    "</h4>\n",
    "\n",
    "### Note: image has the `y` reversed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def average_slope_intercept(lines):\n",
    "    left_lines    = [] # (slope, intercept)\n",
    "    left_weights  = [] # (length,)\n",
    "    right_lines   = [] # (slope, intercept)\n",
    "    right_weights = [] # (length,)\n",
    "    for line in lines:\n",
    "        for x1, y1, x2, y2 in line:\n",
    "            if x2==x1:\n",
    "                continue # ignore a vertical line\n",
    "            slope = (y2-y1)/(x2-x1)\n",
    "            intercept = y1 - slope*x1\n",
    "            length = np.sqrt((y2-y1)**2+(x2-x1)**2)\n",
    "            if slope < 0: # y is reversed in image\n",
    "                left_lines.append((slope, intercept))\n",
    "                left_weights.append((length))\n",
    "            else:\n",
    "                right_lines.append((slope, intercept))\n",
    "                right_weights.append((length))   \n",
    "    left_lane  = np.dot(left_weights,  left_lines) /np.sum(left_weights)  if len(left_weights) >0 else None\n",
    "    right_lane = np.dot(right_weights, right_lines)/np.sum(right_weights) if len(right_weights)>0 else None\n",
    "    \n",
    "    return left_lane, right_lane "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Drawing the lanes \n",
    "\n",
    "<h2> <center> Need to convert to pixel points for cv2.line() </center> </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def calculate_line_points(y1, y2, line):\n",
    "    if line is None:\n",
    "        return None\n",
    "    \n",
    "    slope, intercept = line\n",
    "    x1 = int((y1 - intercept)/slope)\n",
    "    x2 = int((y2 - intercept)/slope)\n",
    "    y1 = int(y1)\n",
    "    y2 = int(y2)\n",
    "    \n",
    "    return ((x1, y1), (x2, y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Drawing the lanes \n",
    "\n",
    "## Each line is a list of x1, y1, x2, y2\n",
    "\n",
    "<h4>\n",
    "<ul>\n",
    "  <li>Each line is a list of x1, y1, x2, y2</li>\n",
    "  <li>Use cv2.line() to draw the lines</li>\n",
    "  <li>Use cv2.addWeighted() to mix the images</li>\n",
    "</ul>\n",
    "</h4>\n",
    "\n",
    "<h3><center><a href=\"https://docs.opencv.org/3.4.1/d6/d6e/group__imgproc__draw.html#ga7078a9fae8c7e7d13d24dac2520ae4a2\">cv2.line()</a></center></h3>\n",
    "<h3><center><a href=\"https://docs.opencv.org/3.4.1/d2/de8/group__core__array.html#gafafb2513349db3bcff51f54ee5592a19\">cv2.addWeighted()</a></center></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0,
     10
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def lane_lines(image, lines):\n",
    "    left_lane, right_lane = average_slope_intercept(lines)\n",
    "    y1 = image.shape[0]\n",
    "    y2 = y1*0.7         \n",
    "    left_line  = calculate_line_points(y1, y2, left_lane)\n",
    "    right_line = calculate_line_points(y1, y2, right_lane)\n",
    "    return left_line, right_line \n",
    " \n",
    "def draw_lane_lines(image, lines, color=[0, 255, 0], thickness=20):\n",
    "    line_image = np.zeros_like(image)\n",
    "    for line in lines:\n",
    "        if line is not None:\n",
    "            cv2.line(line_image, *line,  color, thickness)\n",
    "            \n",
    "    return cv2.addWeighted(image, 1.0, line_image, 0.9, 0.0)\n",
    "             \n",
    "lanes_detected = draw_lane_lines(image, lane_lines(image, lines))\n",
    "    \n",
    "show_image(lanes_detected)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# C.V. Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     5
    ],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class LanesDetector:\n",
    "    def __init__(self):\n",
    "        self.left_lines  = deque(maxlen=50)\n",
    "        self.right_lines = deque(maxlen=50)\n",
    "        \n",
    "    def mean_line(self, line, lines):\n",
    "        if line is not None:\n",
    "            lines.append(line)\n",
    "\n",
    "        if len(lines)>0:\n",
    "            line = np.mean(lines, axis=0, dtype=np.int32)\n",
    "            line = tuple(map(tuple, line))\n",
    "        return line\n",
    "\n",
    "    def process(self, image):\n",
    "        white_yellow = select_white(image)\n",
    "        gray         = convert_to_gray_scale(white_yellow)\n",
    "        smooth_gray  = smoothing(gray)\n",
    "        edges        = detect_edges(smooth_gray)\n",
    "        regions      = select_region(edges)\n",
    "        lines        = hough_lines(regions)\n",
    "        left_line, right_line = lane_lines(image, lines)\n",
    "\n",
    "\n",
    "\n",
    "        left_line  = self.mean_line(left_line,  self.left_lines)\n",
    "        right_line = self.mean_line(right_line, self.right_lines)\n",
    "\n",
    "        return draw_lane_lines(image, (left_line, right_line))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Let's process the video!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def process_video(video_input, video_output):\n",
    "    detector = LanesDetector()\n",
    "\n",
    "    clip = VideoFileClip(os.path.join('data', video_input))\n",
    "    processed = clip.fl_image(detector.process)\n",
    "    processed.write_videofile(os.path.join('data', video_output), audio=False)\n",
    "    \n",
    "%time process_video('singapore_drive.mp4', 'our_brand_new_detected_lanes.mp4')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center> <video controls src=\"data/our_brand_new_detected_lanes.mp4\" /> </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src=\"images/traffic_space.jpg\" alt=\"CV\">\n",
    "\n",
    "<h2> 92% of space is unsused <h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 1.3 million people die in road crashes each year\n",
    "## An average 3,287 deaths a day\n",
    "## Leading cause of death among young people (15-29)\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<h4><div align=\"right\">source: Association for Safe International Road Travel</div></h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src=\"images/degree.png\" alt=\"degree\"></center>\n",
    "<h1><center>A Self Driving Cars Degree!</center></h1>\n",
    "<br>\n",
    "<center><img src=\"images/omg.gif\" alt=\"OMG!\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/selfdrivingcar.png\" alt=\"Self Driving Car\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Computer Vision: Sign up now!  \n",
    "<br>\n",
    "<center><img src=\"images/cv.jpg\" alt=\"CV\">\n",
    "<br>\n",
    "<h2> <a href=\"https://www.upcodeacademy.com/courses/11\">upcodeacademy.com/courses/11</a> </h2></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Thanks!\n",
    "<center><img src=\"images/cv.jpg\" alt=\"CV\">\n",
    "<br>\n",
    "<h2> <a href=\"https://www.upcodeacademy.com/courses/11\">upcodeacademy.com/courses/11</a> </h2></center>\n",
    "<br>\n",
    "<h3> <center> marco@upcodeacademy.com </center> </h3>\n",
    "<br><br>\n",
    "<center><img src=\"images/banner.png\" alt=\"upcode banner\"></center>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "livereveal": {
   "autolaunch": true
  },
  "rise": {
   "theme": "night"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}