advanced_lane_finding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Self-Driving Car Engineer Nanodegree\n",
    "\n",
    "## Advanced Lane Finding Project\n",
    "---\n",
    "\n",
    "The goals / steps of this project are the following:\n",
    "1. Compute the camera calibration matrix and distortion coefficients given a set of chessboard images. \n",
    "2. Apply a distortion correction to raw images.\n",
    "3. Use color transforms, gradients, etc., to create a thresholded binary image.\n",
    "4. Apply a perspective transform to rectify binary image (\"birds-eye view\").\n",
    "5. Detect lane pixels and fit to find the lane boundary.\n",
    "6. Determine the curvature of the lane and vehicle position with respect to center.\n",
    "7. Warp the detected lane boundaries back onto the original image.\n",
    "8. Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.\n",
    "\n",
    "## [Rubric](https://review.udacity.com/#!/rubrics/476/view) Points\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "from matplotlib.patches import Polygon\n",
    "import numpy as np\n",
    "import glob\n",
    "\n",
    "from moviepy.editor import VideoFileClip\n",
    "from IPython.display import HTML\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# list of test images' paths\n",
    "test_in_paths = glob.glob('test_images/*')\n",
    "\n",
    "\n",
    "def plot_result(imgs, names, rows=0, cols=0, save=False):  # 저장 기능 추가\n",
    "    '''\n",
    "    plot the result\n",
    "    '''\n",
    "    if (len(imgs)==0 or len(names)==0): \n",
    "        return -1\n",
    "    \n",
    "    f, ax = plt.subplots(rows, cols, figsize=(16,8))\n",
    "    f.tight_layout()\n",
    "    i = 0\n",
    "    if rows <= 1 :\n",
    "        for c in range(cols):\n",
    "            ax[c].imshow(imgs[i], cmap='gray')\n",
    "            ax[c].set_title('{}'.format(names[i]), fontsize=24)\n",
    "            ax[c].axis('off')\n",
    "            i += 1\n",
    "    else:\n",
    "        for r in range(rows):\n",
    "            for c in range(cols):\n",
    "                ax[r, c].imshow(imgs[i], cmap='gray')\n",
    "                ax[r, c].set_title('{}'.format(names[i]), fontsize=24)\n",
    "                ax[r, c].axis('off')\n",
    "                i += 1\n",
    "    plt.suptitle(p.split('/')[-1], fontsize=36)\n",
    "    plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)\n",
    "    if save:\n",
    "        print(f\"{p.split('/')[-1]}plot.jpg\")\n",
    "        plt.savefig(f\"plot/{p.split('/')[-1]}-plot.jpg\")\n",
    "    \n",
    "\n",
    "\n",
    "def camera_calibration(cal_path, debug=False):\n",
    "    '''\n",
    "    this routine performs camera calibration\n",
    "    it returns `mtx` and `dist` needed to\n",
    "    undistort images taken from this camera\n",
    "    '''\n",
    "    # list all calibration images paths\n",
    "    cal_images_names = glob.glob(cal_path)\n",
    "\n",
    "    # chessboard-specific parameters\n",
    "    nx = 10\n",
    "    ny = 7\n",
    "\n",
    "    # code below is based on classroom example\n",
    "    objpoints = [] # 3D points\n",
    "    imgpoints = [] # 2D points\n",
    "\n",
    "    # (x,y,z): (0,0,0), (1,0,0), etc\n",
    "    objp = np.zeros((nx * ny,3), np.float32)\n",
    "    objp[:,:2] = np.mgrid[0:nx,0:ny].T.reshape(-1,2) # x, y coordinates, z stays 0\n",
    "\n",
    "    for fname in cal_images_names:\n",
    "        # read in image\n",
    "        img = cv2.imread(fname)\n",
    "\n",
    "        # convert to grayscale\n",
    "        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "        # find chessboard corners\n",
    "        ret, corners = cv2.findChessboardCorners(gray, (nx, ny), None)\n",
    "\n",
    "        # in case chessboard was found successfully\n",
    "        if ret == True:\n",
    "            # image points will be different for each calibration image\n",
    "            imgpoints.append(corners)\n",
    "            # object points are the same for all calibration images\n",
    "            objpoints.append(objp)\n",
    "\n",
    "            # Draw and display the corners\n",
    "            cv2.drawChessboardCorners(img, (nx, ny), corners, ret)\n",
    "            if debug:\n",
    "                print(fname)\n",
    "                cv2.imwrite(f'{fname[:-4]}-r.jpg', img)  # 테스트로 추가\n",
    "                \n",
    "                plt.figure(figsize=(15,10))\n",
    "                plt.imshow(img)\n",
    "\n",
    "    # calibration parameters calculation\n",
    "    ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)\n",
    "\n",
    "    # will only use `mtx` and `dist` in this project, hence return\n",
    "    return mtx, dist\n",
    "\n",
    "def undistort_image(image, mtx, dist):\n",
    "    '''\n",
    "    returns an undistorted image (after camera calibration)\n",
    "    '''\n",
    "    dst = cv2.undistort(image, mtx, dist, None, mtx)\n",
    "\n",
    "    return dst\n",
    "\n",
    "def gaussian_blur(image, kernel=5):\n",
    "    '''\n",
    "    this routine applies blur to reduce noise in images\n",
    "    '''\n",
    "    blurred = cv2.GaussianBlur(image, (kernel,kernel), 0)\n",
    "    return blurred\n",
    "\n",
    "\n",
    "def get_thresholded_image(img):\n",
    "    \n",
    "    img = cv2.undistort(img, mtx, dist, None, mtx)\n",
    "    \n",
    "    # convert to gray scale\n",
    "    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n",
    "    \n",
    "    height, width = gray.shape\n",
    "    \n",
    "    # # apply gradient threshold on the horizontal gradient\n",
    "    # sx_binary = abs_sobel_thresh(gray, 'x', 10, 200)\n",
    "    # \n",
    "    # # apply gradient direction threshold so that only edges closer to vertical are detected.\n",
    "    # dir_binary = dir_threshold(gray, thresh=(np.pi/6, np.pi/2))\n",
    "    # \n",
    "    # # combine the gradient and direction thresholds.\n",
    "    # combined_condition = ((sx_binary == 1) & (dir_binary == 1))\n",
    "    # \n",
    "    # # R & G thresholds so that yellow lanes are detected well.\n",
    "    # color_threshold = 100  # 150\n",
    "    # R = img[:,:,0]\n",
    "    # G = img[:,:,1]\n",
    "    # color_combined = np.zeros_like(R)\n",
    "    # r_g_condition = (R > color_threshold) & (G > color_threshold)\n",
    "    # \n",
    "    # \n",
    "    # # color channel thresholds\n",
    "    # hls = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)\n",
    "    # S = hls[:,:,2]\n",
    "    # L = hls[:,:,1]\n",
    "    # \n",
    "    # # S channel performs well for detecting bright yellow and white lanes\n",
    "    # s_thresh = (100, 255)\n",
    "    # s_condition = (S > s_thresh[0]) & (S <= s_thresh[1])\n",
    "    # \n",
    "    # # We put a threshold on the L channel to avoid pixels which have shadows and as a result darker.\n",
    "    # l_thresh = (120, 255)\n",
    "    # l_condition = (L > l_thresh[0]) & (L <= l_thresh[1])\n",
    "    # \n",
    "    # # combine all the thresholds\n",
    "    # # A pixel should either be a yellowish or whiteish\n",
    "    # # And it should also have a gradient, as per our thresholds\n",
    "    # color_combined[(r_g_condition & l_condition) & (s_condition | combined_condition)] = 1\n",
    "    # apply gradient threshold on the horizontal gradient\n",
    "    sx_binary = abs_sobel_thresh(gray, 'x', 10, 200)\n",
    "    \n",
    "    # apply gradient direction threshold so that only edges closer to vertical are detected.\n",
    "    # dir_binary = dir_threshold(gray, thresh=(np.pi/6, np.pi/2))\n",
    "    dir_binary = dir_threshold(gray, thresh=(np.pi/6, np.pi/3))\n",
    "    \n",
    "    # combine the gradient and direction thresholds.\n",
    "    combined_condition = ((sx_binary == 1) & (dir_binary == 1))\n",
    "    \n",
    "    # R & G thresholds so that yellow lanes are detected well.\n",
    "    color_threshold = 100\n",
    "    R = img[:,:,0]\n",
    "    G = img[:,:,1]\n",
    "    color_combined = np.zeros_like(R)\n",
    "    r_g_condition = (R > color_threshold) & (G > color_threshold)\n",
    "    \n",
    "    # color channel thresholds\n",
    "    hls = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)\n",
    "    H = hls[:,:,0]  # Hue channel\n",
    "    S = hls[:,:,2]  # Saturation channel\n",
    "    L = hls[:,:,1]  # Lightness channel\n",
    "    \n",
    "    # Adjust the H channel for yellow\n",
    "    h_thresh = (15, 30)  # Hue values for yellowish colors\n",
    "    h_condition = (H > h_thresh[0]) & (H <= h_thresh[1])\n",
    "    \n",
    "    # S channel performs well for detecting bright yellow and white lanes\n",
    "    s_thresh = (90, 255)  # slightly adjust the lower bound for better yellow detection\n",
    "    s_condition = (S > s_thresh[0]) & (S <= s_thresh[1])\n",
    "    \n",
    "    # We put a threshold on the L channel to avoid pixels which have shadows and as a result darker.\n",
    "    l_thresh = (120, 255)\n",
    "    l_condition = (L > l_thresh[0]) & (L <= l_thresh[1])\n",
    "    \n",
    "    # combine all the thresholds\n",
    "    color_combined[((r_g_condition & l_condition) & s_condition) | (h_condition & s_condition) | combined_condition] = 1\n",
    "\n",
    "    \n",
    "    # apply the region of interest mask\n",
    "    mask = np.zeros_like(color_combined)\n",
    "    region_of_interest_vertices = np.array([[0,height-1], [width/2, int(0.5*height)], [width-1, height-1]], dtype=np.int32)\n",
    "    cv2.fillPoly(mask, [region_of_interest_vertices], 1)\n",
    "    thresholded = cv2.bitwise_and(color_combined, mask)\n",
    "    \n",
    "    return thresholded\n",
    " \n",
    "    \n",
    "def abs_sobel_thresh(gray, orient='x', thresh_min=0, thresh_max=255):\n",
    "    if orient == 'x':\n",
    "        sobel = cv2.Sobel(gray, cv2.CV_64F, 1, 0)\n",
    "    else:\n",
    "        sobel = cv2.Sobel(gray, cv2.CV_64F, 0, 1)\n",
    "    abs_sobel = np.absolute(sobel)\n",
    "    max_value = np.max(abs_sobel)\n",
    "    binary_output = np.uint8(255*abs_sobel/max_value)\n",
    "    threshold_mask = np.zeros_like(binary_output)\n",
    "    threshold_mask[(binary_output >= thresh_min) & (binary_output <= thresh_max)] = 1\n",
    "    return threshold_mask\n",
    "\n",
    "# def dir_threshold(gray, sobel_kernel=3, thresh=(0, np.pi/2)):\n",
    "def dir_threshold(gray, sobel_kernel=21, thresh=(np.pi/3, np.pi/2)):\n",
    "    # Take the gradient in x and y separately\n",
    "    sobel_x = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)\n",
    "    sobel_y = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)\n",
    "    # 3) Take the absolute value of the x and y gradients\n",
    "    abs_sobel_x = np.absolute(sobel_x)\n",
    "    abs_sobel_y = np.absolute(sobel_y)\n",
    "    # 4) Use np.arctan2(abs_sobely, abs_sobelx) to calculate the direction of the gradient\n",
    "    direction = np.arctan2(abs_sobel_y,abs_sobel_x)\n",
    "    direction = np.absolute(direction)\n",
    "    # 5) Create a binary mask where direction thresholds are met\n",
    "    mask = np.zeros_like(direction)\n",
    "    mask[(direction >= thresh[0]) & (direction <= thresh[1])] = 1\n",
    "    # 6) Return this mask as your binary_output image\n",
    "    return mask\n",
    "\n",
    "\n",
    "\n",
    "def warp(img, src_coordinates=None, dst_coordinates=None):\n",
    "\n",
    "    # Define calibration box in source (original) and destination (desired or warped) coordinates\n",
    "    img_size = (img.shape[1], img.shape[0])\n",
    "    \n",
    "    \n",
    "    if src_coordinates is None:\n",
    "        src_coordinates = np.float32(\n",
    "            [[280,  700],  # Bottom left\n",
    "             [595,  460],  # Top left\n",
    "             [725,  460],  # Top right\n",
    "             [1125, 700]]) # Bottom right\n",
    "        \n",
    "    if dst_coordinates is None:\n",
    "        dst_coordinates = np.float32(\n",
    "            [[250,  720],  # Bottom left\n",
    "             [250,    0],  # Top left\n",
    "             [1065,   0],  # Top right\n",
    "             [1065, 720]]) # Bottom right   \n",
    "\n",
    "    # Compute the perspective transfor, M\n",
    "    M = cv2.getPerspectiveTransform(src_coordinates, dst_coordinates)\n",
    "\n",
    "    \n",
    "    # Compute the inverse perspective transfor also by swapping the input parameters\n",
    "    Minv = cv2.getPerspectiveTransform(dst_coordinates, src_coordinates)\n",
    "    \n",
    "    # Create warped image - uses linear interpolation\n",
    "    warped = cv2.warpPerspective(img, M, img_size, flags=cv2.INTER_LINEAR)\n",
    "\n",
    "    return warped, M, Minv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Camera Calibration\n",
    "\n",
    "1. Have the camera matrix and distortion coefficients been computed correctly and checked on one of the calibration images as a test?\n",
    "The code for this step is contained in the first code cell of the IPython notebook (or in lines # through # of the file called some_file.py ).\n",
    "I start by preparing \"object points\", which will be the (x, y, z) coordinates of the chessboard corners in the world. Here I am assuming the chessboard is fixed on the (x, y) plane at z=0, such that the object points are the same for each calibration image. Thus, objp is just a replicated array of coordinates, and objpoints will be appended with a copy of it every time I successfully detect all chessboard corners in a test image.\n",
    "imgpoints will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.\n",
    "                \n",
    "  I then used the output objpoints and imgpoints to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera() function. I applied this distortion correction to the test image using the cv2.undistort() function and obtained this result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# camera calibration step\n",
    "\n",
    "mtx, dist = camera_calibration('./cali/cal*.jpg', debug=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Apply a distortion correction to raw images.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Raw Test image\n",
    "# p = test_in_paths[5]\n",
    "p = 'test-vid2-frame2.jpg'\n",
    "image = mpimg.imread(p)\n",
    "\n",
    "# camera calibration raw image\n",
    "ca_img = mpimg.imread('cali/cal1.jpg')\n",
    "undistorted_ca = undistort_image(ca_img, mtx, dist)\n",
    "\n",
    "# STEP 1: UNDISTORT (using camera calibration step matrix and dist)\n",
    "undistorted = undistort_image(image, mtx, dist)\n",
    "\n",
    "\n",
    "# Plot the result\n",
    "imgs = [ca_img, undistorted_ca, image, undistorted]\n",
    "names = ['Original' ,'Undistored','','']\n",
    "plot_result(imgs, names, 2, 2, save=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Use color transforms, gradients, etc., to create a thresholded binary image.\n",
    "### Let's have a look at different color space configurations of challenging frames\n",
    "Gradient & Color Thresholding\n",
    "\n",
    "The lanes lines in the challenge and harder challenge videos were extremely difficult to detect. They were either too bright or too dull. This prompted me to have R & G channel thresholding and L channel thresholding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n",
    "\n",
    "red = image[:,:,0]\n",
    "green = image[:,:,1]\n",
    "blue = image[:,:,2]\n",
    "    \n",
    "hls = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)\n",
    "h = hls[:,:,0]\n",
    "l = hls[:,:,1]\n",
    "s = hls[:,:,2]\n",
    "    \n",
    "# Plot the result\n",
    "\n",
    "imgs = [image, gray, red, h, green, l, blue, s]\n",
    "names = ['Original', 'Gray' ,'R', 'H', 'G', 'L', 'B', 'S']\n",
    "plot_result(imgs, names, 4, 2)\n",
    "\n",
    "cv2.imwrite('plot/1.jpg', image)\n",
    "cv2.imwrite('plot/2.jpg', gray)\n",
    "cv2.imwrite('plot/3.jpg', red)\n",
    "cv2.imwrite('plot/4.jpg', h)\n",
    "cv2.imwrite('plot/5.jpg', green)\n",
    "cv2.imwrite('plot/6.jpg', l)\n",
    "cv2.imwrite('plot/7.jpg', blue)\n",
    "cv2.imwrite('plot/8.jpg', s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# STEP 2: GAUSSIAN BLUR\n",
    "blurred = gaussian_blur(undistorted, kernel=3)\n",
    "# STEP 3: APPLY COLOR SPACE TRANSFORM AND SOBEL THRESHOLDING\n",
    "combined = get_thresholded_image(blurred)\n",
    "\n",
    "# Plot the result\n",
    "imgs = [image, undistorted, blurred, combined]\n",
    "names = ['Original' ,'Undistored', 'Blurred', 'Color/Sobel Threshold']\n",
    "plot_result(imgs, names, 2, 2)\n",
    "\n",
    "# image_bgr = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "# undistorted2 = cv2.cvtColor(undistorted, cv2.COLOR_RGB2BGR)\n",
    "# blurred2 = cv2.cvtColor(blurred, cv2.COLOR_RGB2BGR)\n",
    "# combined2 = cv2.cvtColor(combined, cv2.COLOR_RGB2BGR)\n",
    "# cv2.imwrite('plot/1.jpg', image_bgr)\n",
    "# cv2.imwrite('plot/2.jpg', undistorted2)\n",
    "# cv2.imwrite('plot/3.jpg', blurred2)\n",
    "# cv2.imwrite('plot/4.jpg', combined2 * 255)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Apply a perspective transform to rectify binary image (\"birds-eye view\").\n",
    "The code for my perspective transform is includes a function called warper(). The warper() function takes as inputs an image ( img ), as well as source ( src ) and destination ( dst ) points. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function\n",
    "\n",
    "\n",
    "\n",
    "# original code\n",
    "# src_coordinates = np.float32(\n",
    "#     [[280,  700],  # Bottom left\n",
    "#      [595,  460],  # Top left\n",
    "#      [725,  460],  # Top right\n",
    "#      [1125, 700]]) # Bottom right\n",
    "\n",
    "dst_coordinates = np.float32(\n",
    "    [[250,  720],  # Bottom left\n",
    "     [250,    0],  # Top left\n",
    "     [1065,   0],  # Top right\n",
    "     [1065, 720]]) # Bottom right   \n",
    "src_coordinates = np.float32(\n",
    "    [[335, 540],  # Bottom left\n",
    "     [612, 390],  # Top left\n",
    "     [691, 390],  # Top right\n",
    "     [968, 540]]) # Bottom right\n",
    "\n",
    "# src_coordinates = np.float32(\n",
    "#     [[440, 540],  # Bottom left\n",
    "#      [617, 390],  # Top left\n",
    "#      [680, 390],  # Top right\n",
    "#      [910, 540]]) # Bottom right\n",
    "\n",
    "coordinates = [src_coordinates,dst_coordinates]\n",
    "\n",
    "# STEP 4: WARP BINARY IMAGE INTO TOP-DOWN VIEW\n",
    "warped, M, Minv = warp(image, src_coordinates, dst_coordinates)\n",
    "# cv2.imwrite('debug.jpg', warped)\n",
    "# warped2 = warped\n",
    "\n",
    "# Plot the result\n",
    "imgs = [image, warped]\n",
    "names = ['Original' ,'Warped (Birds-eye View)']\n",
    "\n",
    "image_bgr = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "warped2 = cv2.cvtColor(warped, cv2.COLOR_RGB2BGR)\n",
    "\n",
    "xy = Polygon(coordinates[0]).get_xy()\n",
    "pts = np.array(xy, np.int32)\n",
    "pts = pts.reshape((-1, 1, 2))\n",
    "\n",
    "# cv2.polylines 함수로 다각형 그리기\n",
    "cv2.polylines(image_bgr, [pts], isClosed=True, color=(0, 0, 255), thickness=2)\n",
    "\n",
    "xy = Polygon(coordinates[1]).get_xy()\n",
    "pts = np.array(xy, np.int32)\n",
    "pts = pts.reshape((-1, 1, 2))\n",
    "\n",
    "# cv2.polylines 함수로 다각형 그리기\n",
    "cv2.polylines(warped2, [pts], isClosed=True, color=(0, 0, 255), thickness=2)\n",
    "\n",
    "cv2.imwrite('plot/1.jpg', image_bgr)\n",
    "cv2.imwrite('plot/2.jpg', warped2)\n",
    "\n",
    "f, ax = plt.subplots(1, 2, figsize=(16,8))\n",
    "f.tight_layout()\n",
    "i = 0\n",
    "for c in range(2):\n",
    "    ax[c].imshow(imgs[i], cmap='gray')\n",
    "    ax[c].plot(Polygon(coordinates[i]).get_xy()[:, 0], Polygon(coordinates[i]).get_xy()[:, 1], color='red')\n",
    "    # warped, M, Minv = warp(combined)\n",
    "    warped, M, Minv = warp(combined, src_coordinates, dst_coordinates)\n",
    "    ax[c].set_title('{}'.format(names[i]), fontsize=24)\n",
    "    ax[c].axis('off')\n",
    "    i += 1\n",
    "\n",
    "# cv2.imwrite('debug2.jpg', warped)\n",
    "plt.suptitle(p.split('/')[-1], fontsize=36)\n",
    "plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)\n",
    "\n",
    "warped2 = cv2.cvtColor(warped, cv2.COLOR_RGB2BGR)\n",
    "cv2.imwrite('plot/3.jpg', warped2 * 255)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Detect lane pixels and fit to find the lane boundary.\n",
    "### Identifying the lane pixels: Part 1\n",
    "### Histogram\n",
    "The peaks int the histogram tell us about the likely position of the lanes in the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_histogram_peaks(img):\n",
    "    histogram = np.sum(img[img.shape[0]//2:,:], axis=0)\n",
    "\n",
    "    # Peak in the first half indicates the likely position of the left lane\n",
    "    half_width = int(histogram.shape[0]//2)\n",
    "    leftx_base = np.argmax(histogram[:half_width])\n",
    "\n",
    "    # Peak in the second half indicates the likely position of the right lane\n",
    "    rightx_base = np.argmax(histogram[half_width:]) + half_width\n",
    "    \n",
    "    return histogram, leftx_base, rightx_base\n",
    "\n",
    "histogram, leftx_base, rightx_base = find_histogram_peaks(warped)\n",
    "\n",
    "print(leftx_base, rightx_base)\n",
    "plt.plot(histogram)\n",
    "\n",
    "# plt.savefig('plot/plt1.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 피크 개수를 세어 노이즈 감지"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.signal import find_peaks\n",
    "from scipy.signal import peak_prominences\n",
    "\n",
    "\n",
    "def detect_noisy(histogram, min_prominence=30, avg_value_threshold=20, zero_count_threshold=300, return_peaks=False):\n",
    "    # Detect peaks in the histogram\n",
    "    peaks, _ = find_peaks(histogram)\n",
    "    \n",
    "    # Calculate the prominence of the detected peaks\n",
    "    prominences = peak_prominences(histogram, peaks)[0]\n",
    "    \n",
    "    # Filter peaks based on prominence\n",
    "    filtered_peaks = peaks[prominences > min_prominence]\n",
    "    \n",
    "    is_noisy = False\n",
    "    \n",
    "    # If there are at least two peaks, check the average value criterion\n",
    "    if len(filtered_peaks) >= 2:\n",
    "        left_peak_idx = filtered_peaks[0]\n",
    "        right_peak_idx = filtered_peaks[-1]\n",
    "        \n",
    "        # Calculate the average value between the two peaks\n",
    "        avg_value = np.mean(histogram[left_peak_idx:right_peak_idx])\n",
    "        # print(avg_value)\n",
    "        \n",
    "        # Check if the average value exceeds the threshold\n",
    "        if avg_value > avg_value_threshold:\n",
    "            is_noisy = True\n",
    "\n",
    "        # Calculate the count of zero values between the two peaks\n",
    "        zero_count = np.sum(histogram[left_peak_idx:right_peak_idx] < 10)\n",
    "        # print(zero_count)\n",
    "        \n",
    "        # Check if the zero count exceeds the threshold\n",
    "        if zero_count > zero_count_threshold:\n",
    "            is_noisy = False  # It's not noisy if there are more than 300 zero values\n",
    "    if return_peaks:\n",
    "        return filtered_peaks, is_noisy\n",
    "    \n",
    "    return is_noisy\n",
    "\n",
    "\n",
    "histogram, _, _ = find_histogram_peaks(warped)\n",
    "peaks, is_noisy = detect_noisy(histogram, return_peaks=True)\n",
    "\n",
    "print(\"Is histogram noisy?\", is_noisy)\n",
    "plt.title(f'noisy: {is_noisy}')\n",
    "plt.plot(histogram)\n",
    "for peak in peaks:\n",
    "    plt.plot(peak, histogram[peak], \"ro\")  # Plot detected peaks as red circles\n",
    "plt.savefig('plot/plt2.jpg')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sliding Window Search\n",
    "I then perform a sliding window search, starting with the base likely positions of the 2 lanes, calculated from the histogram. I have used 10 windows of width 100 pixels.\n",
    "\n",
    "The x & y coordinates of non zeros pixels are found, a polynomial is fit for these coordinates and the lane lines are drawn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_lines(img, return_img=False):\n",
    "    # Take a histogram of the bottom half of the image\n",
    "    \n",
    "    histogram, leftx_base, rightx_base = find_histogram_peaks(img)\n",
    "\n",
    "    if return_img:\n",
    "        # Create an output image to draw on and  visualize the result\n",
    "        out_img = np.dstack((img, img, img))*255\n",
    "\n",
    "    # Choose the number of sliding windows\n",
    "    nwindows = 9\n",
    "    \n",
    "    # Set height of windows\n",
    "    window_height = int(img.shape[0]//nwindows)\n",
    "    \n",
    "    # Identify the x and y positions of all nonzero pixels in the image\n",
    "    nonzero = img.nonzero()\n",
    "    nonzerox = np.array(nonzero[1])   \n",
    "    nonzeroy = np.array(nonzero[0])\n",
    "   \n",
    "\n",
    "    # Current positions to be updated for each window\n",
    "    leftx_current = leftx_base\n",
    "    rightx_current = rightx_base\n",
    "    \n",
    "    # Set the width of the windows +/- margin\n",
    "    margin = 100\n",
    "\n",
    "    # Set minimum number of pixels found to recenter window\n",
    "    minpix = 50\n",
    "    \n",
    "    # Create empty lists to receive left and right lane pixel indices\n",
    "    left_lane_inds = []\n",
    "    right_lane_inds = []\n",
    "\n",
    "    # Step through the windows one by one\n",
    "    for window in range(nwindows):\n",
    "        # Identify window boundaries in x and y (and right and left)\n",
    "        win_y_low = img.shape[0] - (window + 1)*window_height\n",
    "        win_y_high = img.shape[0] - window*window_height\n",
    "        win_xleft_low = leftx_current - margin\n",
    "        win_xleft_high = leftx_current + margin\n",
    "        win_xright_low = rightx_current - margin\n",
    "        win_xright_high = rightx_current + margin\n",
    "        \n",
    "        if return_img:\n",
    "            # Draw the windows on the visualization image\n",
    "            cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high), (0,255,0), 3) \n",
    "            cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high), (0,255,0), 3) \n",
    "\n",
    "        # Identify the nonzero pixels in x and y within the window\n",
    "        good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & \n",
    "                          (nonzerox >= win_xleft_low) &  (nonzerox < win_xleft_high)).nonzero()[0]\n",
    "        good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & \n",
    "                           (nonzerox >= win_xright_low) &  (nonzerox < win_xright_high)).nonzero()[0]\n",
    "        # Append these indices to the lists\n",
    "        left_lane_inds.append(good_left_inds)\n",
    "        right_lane_inds.append(good_right_inds)\n",
    "        \n",
    "        # If you found > minpix pixels, recenter next window on their mean position\n",
    "        if len(good_left_inds) > minpix:\n",
    "            leftx_current = int(np.mean(nonzerox[good_left_inds]))\n",
    "        if len(good_right_inds) > minpix:        \n",
    "            rightx_current = int(np.mean(nonzerox[good_right_inds]))\n",
    "\n",
    "    # Concatenate the arrays of indices\n",
    "    left_lane_inds = np.concatenate(left_lane_inds)\n",
    "    right_lane_inds = np.concatenate(right_lane_inds)\n",
    "    \n",
    "    # Extract left and right line pixel positions\n",
    "    leftx = nonzerox[left_lane_inds]\n",
    "    lefty = nonzeroy[left_lane_inds] \n",
    "    rightx = nonzerox[right_lane_inds]\n",
    "    righty = nonzeroy[right_lane_inds] \n",
    "\n",
    "    # If line is not detected\n",
    "    if lefty.size == 0:\n",
    "        lefty = np.array([10, 10])\n",
    "    if leftx.size == 0:\n",
    "        leftx = np.array([10, 10])\n",
    "    if righty.size == 0:\n",
    "        righty = np.array([10, 10])\n",
    "    if rightx.size == 0:\n",
    "        rightx = np.array([10, 10])\n",
    "\n",
    "    # Fit a second order polynomial to each\n",
    "    left_fit = np.polyfit(lefty, leftx, 2)\n",
    "    right_fit = np.polyfit(righty, rightx, 2)\n",
    "    \n",
    "    ploty = np.linspace(0, img.shape[0] - 1, img.shape[0])\n",
    "    left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]\n",
    "    right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]\n",
    "###########################\n",
    "    # 여기부터는\n",
    "    # 탐지된 두 차선의 간격을 계산하고 이상값이 있다면\n",
    "    # 탐지를 건너 뛰도록 하는 부분이다\n",
    "    \n",
    "    # x값들의 차를 계산하여 두 점 사이의 거리들을 구함\n",
    "    differences = right_fitx - left_fitx\n",
    "    \n",
    "    # 계산한 두 점사이의 거리 평균 계산\n",
    "    average_distance = np.mean(differences)\n",
    "    # print('min', min(differences))\n",
    "    # print('max', max(differences))\n",
    "    # print('avg', average_distance)\n",
    "    \n",
    "    # 평균과 최소값, 평균과 최대값이 차를 구하고 절대값으로 변환\n",
    "    min_diff_from_avg = abs(min(differences) - average_distance)\n",
    "    max_diff_from_avg = abs(max(differences) - average_distance)\n",
    "\n",
    "    # min/max와 평균값의 차이 허용 임계값\n",
    "    diff_threshold = 300\n",
    "\n",
    "    # 평균값이 400과 600사이에 있지 않은가\n",
    "    criteria1 = not (400 <= average_distance <= 600)\n",
    "    # min/max와 평균값의 차이가 diff_threshold 이상인가\n",
    "    criteria2 = min_diff_from_avg > diff_threshold or max_diff_from_avg > diff_threshold\n",
    "\n",
    "    # if criteria1 or criteria2:\n",
    "    #     lefty = np.array([10, 10])\n",
    "    #     leftx = np.array([10, 10])\n",
    "    #     righty = np.array([10, 10])\n",
    "    #     rightx = np.array([10, 10])\n",
    "        \n",
    "    #      # Fit a second order polynomial to each\n",
    "    #     left_fit = np.polyfit(lefty, leftx, 2)\n",
    "    #     right_fit = np.polyfit(righty, rightx, 2)\n",
    "        \n",
    "    #     ploty = np.linspace(0, img.shape[0] - 1, img.shape[0])\n",
    "    #     left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]\n",
    "    #     right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]\n",
    "\n",
    "    # if criteria1:\n",
    "    #     print(\"average is not between 400 and 600\")\n",
    "    # if criteria2:\n",
    "    #     print(\"Line detection failed due to large differences from average.\")\n",
    "\n",
    "    #########################\n",
    "    if return_img:\n",
    "        out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]\n",
    "        out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]\n",
    "\n",
    "        # Draw left and right lines\n",
    "        for index in range(img.shape[0]):\n",
    "            cv2.circle(out_img, (int(left_fitx[index]), int(ploty[index])), 3, (255,255,0))\n",
    "            cv2.circle(out_img, (int(right_fitx[index]), int(ploty[index])), 3, (255,255,0))\n",
    "            \n",
    "        return (left_fit, right_fit), (left_fitx, ploty), (right_fitx, ploty), out_img\n",
    "\n",
    "    return (left_fit, right_fit), (left_fitx, ploty), (right_fitx, ploty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function\n",
    "lines_fit, left_points, right_points, out_img = detect_lines(warped, return_img=True)\n",
    "\n",
    "# Plot the result\n",
    "imgs = [warped, out_img]\n",
    "names = ['Warped image' ,'Lane lines detected']\n",
    "plot_result(imgs, names, 1, 2)\n",
    "\n",
    "warped2 = cv2.cvtColor(warped, cv2.COLOR_RGB2BGR)\n",
    "cv2.imwrite('plot/detect1.jpg', warped2 * 255)\n",
    "out_img2 = cv2.cvtColor(out_img, cv2.COLOR_RGB2BGR)\n",
    "cv2.imwrite('plot/detect2.jpg', out_img2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching around a previously detected line.\n",
    "Since consecutive frames are likely to have lane lines in roughly similar positions, in this section we search around a margin of 100 pixels of the previously detected lane lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_averaged_line(previous_lines, new_line):\n",
    "    '''\n",
    "        This function computes an averaged lane line by averaging over previous good frames.\n",
    "    '''\n",
    "    \n",
    "    # Number of frames to average over\n",
    "    num_frames = 12\n",
    "    \n",
    "    if new_line is None:\n",
    "        # No line was detected\n",
    "        \n",
    "        if len(previous_lines) == 0:\n",
    "            # If there are no previous lines, return None\n",
    "            return previous_lines, None\n",
    "        else:\n",
    "            # Else return the last line\n",
    "            return previous_lines, previous_lines[-1]\n",
    "    else:\n",
    "        if len(previous_lines) < num_frames:\n",
    "            # we need at least num_frames frames to average over\n",
    "            previous_lines.append(new_line)\n",
    "            return previous_lines, new_line\n",
    "        else:\n",
    "            # average over the last num_frames frames\n",
    "            previous_lines[0:num_frames-1] = previous_lines[1:]\n",
    "            previous_lines[num_frames-1] = new_line\n",
    "            new_line = np.zeros_like(new_line)\n",
    "            for i in range(num_frames):\n",
    "                new_line += previous_lines[i]\n",
    "            new_line /= num_frames\n",
    "            return previous_lines, new_line\n",
    "\n",
    "def detect_similar_lines(img, \n",
    "                         line_fits=None,\n",
    "                         past_good_left_lines=[],\n",
    "                         past_good_right_lines=[],\n",
    "                         running_mean_difference_between_lines = 0,\n",
    "                         return_img=False):\n",
    "    # if line_fits is None:\n",
    "    return detect_lines(img, return_img)\n",
    "    \n",
    "    \n",
    "    left_fit = line_fits[0]\n",
    "    right_fit = line_fits[1]\n",
    "    \n",
    "    nonzero = img.nonzero()\n",
    "    nonzeroy = np.array(nonzero[0])\n",
    "    nonzerox = np.array(nonzero[1])\n",
    "    margin = 100\n",
    "    left_lane_inds = ((nonzerox > (left_fit[0]*(nonzeroy**2) + left_fit[1]*nonzeroy + \n",
    "    left_fit[2] - margin)) & (nonzerox < (left_fit[0]*(nonzeroy**2) + \n",
    "    left_fit[1]*nonzeroy + left_fit[2] + margin))) \n",
    "\n",
    "    right_lane_inds = ((nonzerox > (right_fit[0]*(nonzeroy**2) + right_fit[1]*nonzeroy + \n",
    "    right_fit[2] - margin)) & (nonzerox < (right_fit[0]*(nonzeroy**2) + \n",
    "    right_fit[1]*nonzeroy + right_fit[2] + margin)))  \n",
    "\n",
    "    # Again, extract left and right line pixel positions\n",
    "    leftx = nonzerox[left_lane_inds]\n",
    "    lefty = nonzeroy[left_lane_inds] \n",
    "    rightx = nonzerox[right_lane_inds]\n",
    "    righty = nonzeroy[right_lane_inds]\n",
    "\n",
    "    # If any of the lines could not be found, \n",
    "    # perform a more exhaustive search\n",
    "    if (leftx.size == 0 or rightx.size == 0):\n",
    "        return detect_lines(img, return_img)\n",
    "    \n",
    "    # Fit a second order polynomial to each\n",
    "    left_fit = np.polyfit(lefty, leftx, 2)\n",
    "    right_fit = np.polyfit(righty, rightx, 2)\n",
    "    \n",
    "    # Generate x and y values for plotting\n",
    "    \n",
    "    # If no pixels were found return None\n",
    "    ploty = np.linspace(0, img.shape[0] - 1, img.shape[0] )\n",
    "    left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]\n",
    "    right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]\n",
    "    \n",
    "\n",
    "    # Smoothing\n",
    "    mean_difference = np.mean(right_fitx - left_fitx)\n",
    "        \n",
    "    if running_mean_difference_between_lines == 0:\n",
    "        running_mean_difference_between_lines = mean_difference\n",
    "        \n",
    "    if (mean_difference < 0.7*running_mean_difference_between_lines or mean_difference > 1.3*running_mean_difference_between_lines):\n",
    "        if len(past_good_left_lines) == 0 and len(past_good_right_lines) == 0:\n",
    "            return detect_lines(img, return_img)\n",
    "        else:\n",
    "            left_fitx = past_good_left_lines[-1]\n",
    "            right_fitx = past_good_right_lines[-1]\n",
    "    else:\n",
    "        past_good_left_lines, left_fitx = get_averaged_line(past_good_left_lines, left_fitx)\n",
    "        past_good_right_lines, right_fitx = get_averaged_line(past_good_right_lines, right_fitx)\n",
    "        mean_difference = np.mean(right_fitx - left_fitx)\n",
    "        running_mean_difference_between_lines = 0.9*running_mean_difference_between_lines + 0.1*mean_difference\n",
    "  \n",
    "        \n",
    "    if return_img:\n",
    "        # Create an image to draw on and an image to show the selection window\n",
    "        out_img = np.dstack((img, img, img))*255\n",
    "        window_img = np.zeros_like(out_img)\n",
    "\n",
    "        # Generate a polygon to illustrate the search window area\n",
    "        # And recast the x and y points into usable format for cv2.fillPoly()\n",
    "        left_line_window1 = np.array([np.transpose(np.vstack([left_fitx - margin, ploty]))])\n",
    "        left_line_window2 = np.array([np.flipud(np.transpose(np.vstack([left_fitx + margin, ploty])))])\n",
    "        left_line_pts = np.hstack((left_line_window1, left_line_window2))\n",
    "        right_line_window1 = np.array([np.transpose(np.vstack([right_fitx - margin, ploty]))])\n",
    "        right_line_window2 = np.array([np.flipud(np.transpose(np.vstack([right_fitx + margin, ploty])))])\n",
    "        right_line_pts = np.hstack((right_line_window1, right_line_window2))\n",
    "\n",
    "        # Draw the lane onto the warped blank image\n",
    "        cv2.fillPoly(window_img, np.int_([left_line_pts]), (0,255, 0))\n",
    "        cv2.fillPoly(window_img, np.int_([right_line_pts]), (0,255, 0))\n",
    "        out_img = cv2.addWeighted(out_img, 1, window_img, 0.3, 0)\n",
    "\n",
    "        # Color in left and right line pixels\n",
    "        out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]\n",
    "        out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]\n",
    "     \n",
    "        for index in range(img.shape[0]):\n",
    "            cv2.circle(out_img, (int(left_fitx[index]), int(ploty[index])), 3, (255,255,0))\n",
    "            cv2.circle(out_img, (int(right_fitx[index]), int(ploty[index])), 3, (255,255,0))\n",
    "    \n",
    "        return (left_fit, right_fit), (left_fitx, ploty), (right_fitx, ploty), out_img\n",
    "    \n",
    "    return (left_fit, right_fit), (left_fitx, ploty), (right_fitx, ploty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function\n",
    "# Notice I am passing the same image than before.\n",
    "# In a video stream, it should be passed the next frame.\n",
    "lines_fit, left_points, right_points, out_img = detect_similar_lines(warped, lines_fit, return_img=True)\n",
    "\n",
    "# Plot the result\n",
    "imgs = [warped, out_img]\n",
    "names = ['Warped image' ,'Lane lines detected']\n",
    "plot_result(imgs, names, 1, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 6. Determine the curvature of the lane and vehicle position with respect to center.\n",
    "### Computing the radius of curvature and center offset.\n",
    "The radius of curvature is computed according to the formula and method described in the classroom material. Since we perform the polynomial fit in pixels and whereas the curvature has to be calculated in real world meters, we have to use a pixel to meter transformation and recompute the fit again.\n",
    "\n",
    "The mean of the lane pixels closest to the car gives us the center of the lane. The center of the image gives us the position of the car. The difference between the 2 is the offset from the center."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# y: 720픽셀당 40미터\n",
    "\n",
    "def curvature_radius (leftx, rightx, img_shape, xm_per_pix=3.7/800, ym_per_pix = 25/720):\n",
    "    ploty = np.linspace(0, img_shape[0] - 1, img_shape[0])\n",
    "    \n",
    "    leftx = leftx[::-1]  # Reverse to match top-to-bottom in y\n",
    "    rightx = rightx[::-1]  # Reverse to match top-to-bottom in y\n",
    "    \n",
    "    # Fit a second order polynomial to pixel positions in each fake lane line\n",
    "    left_fit = np.polyfit(ploty, leftx, 2)\n",
    "    left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]\n",
    "    right_fit = np.polyfit(ploty, rightx, 2)\n",
    "    right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]\n",
    "\n",
    "    # Define conversions in x and y from pixels space to meters\n",
    "    ym_per_pix = 25/720 # meters per pixel in y dimension\n",
    "    xm_per_pix = 3.7/800 # meters per pixel in x dimension\n",
    "\n",
    "    # Fit new polynomials to x,y in world space\n",
    "    y_eval = np.max(ploty)\n",
    "    left_fit_cr = np.polyfit(ploty*ym_per_pix, leftx*xm_per_pix, 2)\n",
    "    right_fit_cr = np.polyfit(ploty*ym_per_pix, rightx*xm_per_pix, 2)\n",
    "    \n",
    "    # Calculate the new radii of curvature\n",
    "    left_curverad = ((1 + (2*left_fit_cr[0]*y_eval*ym_per_pix + left_fit_cr[1])**2)**1.5) / np.absolute(2*left_fit_cr[0])\n",
    "    right_curverad = ((1 + (2*right_fit_cr[0]*y_eval*ym_per_pix + right_fit_cr[1])**2)**1.5) / np.absolute(2*right_fit_cr[0])\n",
    "    \n",
    "    # Now our radius of curvature is in meters\n",
    "    return (left_curverad, right_curverad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# STEP 8: CALCULATE OFFSET DISTANCE AND ROAD CURVATURE\n",
    "\n",
    "# Run the function\n",
    "curvature_rads = curvature_radius(leftx=left_points[0], rightx=right_points[0], img_shape = image.shape)\n",
    "\n",
    "# Print the results\n",
    "print('Left line curvature:', curvature_rads[0], 'm')\n",
    "print('Right line curvature:', curvature_rads[1], 'm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def car_offset(leftx, rightx, img_shape, xm_per_pix=1/153.3):\n",
    "    ## Image mid horizontal position \n",
    "    mid_imgx = img_shape[1]//2\n",
    "        \n",
    "    ## Car position with respect to the lane\n",
    "    car_pos = (leftx[-1] + rightx[-1])/2\n",
    "    \n",
    "    ## Horizontal car offset \n",
    "    offsetx = (mid_imgx - car_pos) * xm_per_pix\n",
    "\n",
    "    return offsetx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function\n",
    "offsetx = car_offset(leftx=left_points[0], rightx=right_points[0], img_shape=image.shape)\n",
    "\n",
    "print ('Car offset from center:', offsetx, 'm.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Warp the detected lane boundaries back onto the original image.\n",
    "### Inverse Transform\n",
    "In this block of code we:\n",
    "\n",
    "Paint the lane area\n",
    "Perform an inverse perspective transform\n",
    "Combine the precessed image with the original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# STEP 7: UNWARP OVERLAY FROM TOP-DOWN VIEW BACK INTO CAMERA VIEW\n",
    "def draw_lane(img, warped_img, left_points, right_points, Minv):\n",
    "    # Create an image to draw the lines on\n",
    "    warp_zero = np.zeros_like(warped_img).astype(np.uint8)\n",
    "    color_warp = np.dstack((warp_zero, warp_zero, warp_zero))\n",
    "\n",
    "    # Recast the x and y points into usable format for cv2.fillPoly()\n",
    "    left_fitx = left_points[0]\n",
    "    right_fitx = right_points[0]\n",
    "    ploty = left_points[1]\n",
    "\n",
    "    pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])\n",
    "    pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])\n",
    "    pts = np.hstack((pts_left, pts_right))\n",
    "\n",
    "    #########################\n",
    "    # histogram, _, _ = find_histogram_peaks(warped_img)\n",
    "    \n",
    "    # # 히스토그램에 노이즈가 많은가\n",
    "    # is_noisy = detect_noisy(histogram)\n",
    "    \n",
    "    \n",
    "    # if is_noisy:\n",
    "    #     cv2.fillPoly(color_warp, np.int_([pts]), (255,255, 0))\n",
    "    # else:\n",
    "    #     cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))\n",
    "    #########################\n",
    "\n",
    "    # Draw the lane onto the warped blank image\n",
    "    cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))\n",
    "\n",
    "    cv2.imwrite('plot/img1.jpg', color_warp)\n",
    "\n",
    "    # Warp the blank back to original image space using inverse perspective matrix (Minv)\n",
    "    newwarp = cv2.warpPerspective(color_warp, Minv, (img.shape[1], img.shape[0])) \n",
    "\n",
    "    cv2.imwrite('plot/img2.jpg', newwarp)\n",
    "\n",
    "    # Combine the result with the original image\n",
    "    return cv2.addWeighted(img, 1, newwarp, 0.3, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function\n",
    "img_lane = draw_lane(image, warped, left_points, right_points, Minv)\n",
    "\n",
    "# Plot the result\n",
    "imgs = [image, img_lane]\n",
    "names = ['Original' ,'Lane detected']\n",
    "plot_result(imgs, names, 1, 2)\n",
    "\n",
    "img_lane2 = cv2.cvtColor(img_lane, cv2.COLOR_RGB2BGR)\n",
    "cv2.imwrite('plot/img3.jpg', img_lane2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_metrics(img, leftx, rightx, xm_per_pix=3.7/800, ym_per_pix = 25/720):    \n",
    "    # Calculate radius of curvature\n",
    "    # curvature_rads = curvature_radius(leftx=leftx, rightx=rightx, img_shape=img.shape,\n",
    "    #                                   xm_per_pix=xm_per_pix, ym_per_pix=ym_per_pix)\n",
    "    # Calculate car offset\n",
    "    offsetx = car_offset(leftx=leftx, rightx=rightx, img_shape=img.shape)\n",
    "\n",
    "    # Display lane curvature\n",
    "    out_img = img.copy()\n",
    "    # cv2.putText(out_img, 'Left lane line curvature: {:.2f} m'.format(curvature_rads[0]), \n",
    "    #             (60, 60), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (255,255,255), 5)\n",
    "    # cv2.putText(out_img, 'Right lane line curvature: {:.2f} m'.format(curvature_rads[1]), \n",
    "    #             (60, 110), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (255,255,255), 5)\n",
    "    \n",
    "    # Display car offset\n",
    "    if offsetx > 0:\n",
    "        cv2.putText(out_img, 'right: {:.2f} m'.format(abs(offsetx)), \n",
    "                (500, 60), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0,0,0), 5)\n",
    "    else:\n",
    "        cv2.putText(out_img, 'left: {:.2f} m'.format(abs(offsetx)), \n",
    "                (500, 60), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0,0,0), 5)\n",
    "    \n",
    "    \n",
    "    return out_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function\n",
    "out_img = add_metrics(img_lane, leftx=left_points[0], rightx=right_points[0])\n",
    "\n",
    "# Plot the result\n",
    "imgs = [image, out_img]\n",
    "names = ['Original' ,'Lane detected with metrics']\n",
    "plot_result(imgs, names, 1, 2)\n",
    "\n",
    "out_img2 = cv2.cvtColor(out_img, cv2.COLOR_RGB2BGR)\n",
    "cv2.imwrite('plot/img4.jpg', out_img2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Run pipeline in a video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_green(result_img):\n",
    "    \"\"\"Detect green regions from the result image.\"\"\"\n",
    "    # Convert image to HSV color space\n",
    "    hsv = cv2.cvtColor(result_img, cv2.COLOR_BGR2HSV)\n",
    "\n",
    "    # Define range for green color\n",
    "    lower_green = np.array([40, 40, 40])\n",
    "    upper_green = np.array([90, 255, 255])\n",
    "\n",
    "    # Create mask for green color\n",
    "    mask = cv2.inRange(hsv, lower_green, upper_green)\n",
    "\n",
    "    return mask\n",
    "\n",
    "def detect_red(result_img):\n",
    "    \"\"\"Detect red regions from the result image.\"\"\"\n",
    "    # Convert image to HSV color space\n",
    "    hsv = cv2.cvtColor(result_img, cv2.COLOR_BGR2HSV)\n",
    "\n",
    "    # Define range for red color\n",
    "    # Red in HSV can be in two ranges due to wrapping around 0\n",
    "    lower_red1 = np.array([0, 100, 100])\n",
    "    upper_red1 = np.array([10, 255, 255])\n",
    "    \n",
    "    lower_red2 = np.array([160, 100, 100])\n",
    "    upper_red2 = np.array([180, 255, 255])\n",
    "\n",
    "    mask1 = cv2.inRange(hsv, lower_red1, upper_red1)\n",
    "    mask2 = cv2.inRange(hsv, lower_red2, upper_red2)\n",
    "\n",
    "    # Combine the two masks to get a complete mask for red\n",
    "    mask = cv2.bitwise_or(mask1, mask2)\n",
    "\n",
    "    return mask\n",
    "\n",
    "\n",
    "def mask_green_box(original_img, result_img):\n",
    "    \"\"\"Detect green region from result image and mask it in the original image.\"\"\"\n",
    "    green_mask = detect_green(result_img)\n",
    "    \n",
    "    # Find the contour of the green area\n",
    "    contours, _ = cv2.findContours(green_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n",
    "\n",
    "    if not contours:\n",
    "        return original_img\n",
    "    \n",
    "    # Assuming the largest contour corresponds to the lane, get its bounding box\n",
    "    largest_contour = max(contours, key=cv2.contourArea)\n",
    "    x, y, w, h = cv2.boundingRect(largest_contour)\n",
    "\n",
    "    # Create a black mask with the size of the original image\n",
    "    mask = np.zeros_like(green_mask)\n",
    "    \n",
    "    # Draw the bounding box as a white rectangle on the black mask\n",
    "    mask[y:y+h, x:x+w] = 255\n",
    "\n",
    "    # Extract the box region's corresponding area from the original image\n",
    "    box_region = cv2.bitwise_and(original_img, original_img, mask=mask)\n",
    "\n",
    "    return box_region\n",
    "\n",
    "def mask_red_box(original_img, result_img):\n",
    "    \"\"\"Detect red region from result image and mask it in the original image.\"\"\"\n",
    "    red_mask = detect_red(result_img)\n",
    "    \n",
    "    # Find the contour of the red area\n",
    "    contours, _ = cv2.findContours(red_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n",
    "\n",
    "    if not contours:\n",
    "        return original_img\n",
    "    \n",
    "    # Assuming the largest contour corresponds to the lane, get its bounding box\n",
    "    largest_contour = max(contours, key=cv2.contourArea)\n",
    "    x, y, w, h = cv2.boundingRect(largest_contour)\n",
    "\n",
    "    # Create a black mask with the size of the original image\n",
    "    mask = np.zeros_like(red_mask)\n",
    "    \n",
    "    # Draw the bounding box as a white rectangle on the black mask\n",
    "    mask[y:y+h, x:x+w] = 255\n",
    "\n",
    "    # Extract the box region's corresponding area from the original image\n",
    "    box_region = cv2.bitwise_and(original_img, original_img, mask=mask)\n",
    "\n",
    "    return box_region\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Pipeline:\n",
    "    def __init__(self, cal_path):\n",
    "        # Make a list of calibration images\n",
    "\n",
    "        # Calibrate camera\n",
    "        self.mtx, self.dist = camera_calibration(cal_path)\n",
    "        \n",
    "        # Reinitialize \n",
    "        self.lines_fit = None\n",
    "        self.past_good_left_lines = []\n",
    "        self.past_good_right_lines = []\n",
    "        self.running_mean_difference_between_lines = 0\n",
    "\n",
    "    def __call__(self, img):\n",
    "        \n",
    "        # STEP 1: UNDISTORT (using camera calibration step matrix and dist)\n",
    "        undistorted = undistort_image(img, self.mtx, self.dist)\n",
    "        \n",
    "        # STEP 2: GAUSSIAN BLUR\n",
    "        blurred = gaussian_blur(undistorted, kernel=3)\n",
    "\n",
    "        # STEP 3: APPLY COLOR SPACE TRANSFORM AND SOBEL THRESHOLDING\n",
    "        combined = get_thresholded_image(blurred)\n",
    "        \n",
    "        # Apply a perspective transform to rectify binary image (\"birds-eye view\")\n",
    "        # src_coordinates = np.float32(\n",
    "        #     [[280,  700],  # Bottom left\n",
    "        #      [595,  460],  # Top left\n",
    "        #      [725,  460],  # Top right\n",
    "        #      [1125, 700]]) # Bottom right\n",
    "    #     src_coordinates = np.float32(\n",
    "    # [[385, 540],  # Bottom left\n",
    "    #  [618, 390],  # Top left\n",
    "    #  [685, 390],  # Top right\n",
    "    #  [918, 540]]) # Bottom right\n",
    "        src_coordinates = np.float32(\n",
    "        [[335, 540],  # Bottom left\n",
    "         [612, 390],  # Top left\n",
    "         [691, 390],  # Top right\n",
    "         [968, 540]]) # Bottom right\n",
    "\n",
    "        dst_coordinates = np.float32(\n",
    "            [[250,  720],  # Bottom left\n",
    "             [250,    0],  # Top left\n",
    "             [1065,   0],  # Top right\n",
    "             [1065, 720]]) # Bottom right   \n",
    "\n",
    "        # STEP 4: WARP BINARY IMAGE INTO TOP-DOWN VIEW\n",
    "        warped, M, Minv = warp(combined, src_coordinates, dst_coordinates)\n",
    "        \n",
    "        \n",
    "        # STEP 5: Detect lane pixels and fit to find the lane boundary\n",
    "        self.lines_fit, left_points, right_points, out_img = detect_similar_lines(warped, \n",
    "                                                                                  self.lines_fit, \n",
    "                                                                                  self.past_good_left_lines,\n",
    "                                                                                  self.past_good_right_lines,\n",
    "                                                                                  self.running_mean_difference_between_lines,\n",
    "                                                                                  return_img=True,)\n",
    "\n",
    "        # STEP 6: Warp the detected lane boundaries back onto the original image.\n",
    "        img_lane = draw_lane(img, warped, left_points, right_points, Minv)\n",
    "            \n",
    "        # STEP 7: Add metrics to the output img\n",
    "        out_img = add_metrics(img_lane, leftx=left_points[0], rightx=right_points[0])\n",
    "            \n",
    "        return out_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "from keras.models import load_model\n",
    "\n",
    "\n",
    "model = load_model('full_CNN_model.h5')\n",
    "\n",
    "\n",
    "class Pipeline2():\n",
    "    def __init__(self):\n",
    "        self.recent_fit = []\n",
    "        self.avg_fit = []\n",
    "\n",
    "    def process(self, image):\n",
    "        \"\"\" Takes in a road image, crops it 200px above the bottom and 300px from both sides, re-sizes for the model,\n",
    "        predicts the lane to be drawn from the model in RED color,\n",
    "        recreates an RGB image of a lane and merges with the\n",
    "        original road image. A red line is drawn at 200px above the bottom.\n",
    "        \"\"\"\n",
    "        \n",
    "        # Crop the image to 200px above the bottom and 300px from both sides\n",
    "        cropped_image = image[:-50, 300:-300]\n",
    "        \n",
    "        # Convert cropped image to PIL for resizing\n",
    "        cropped_image_pil = Image.fromarray(np.uint8(cropped_image))\n",
    "        cropped_image_resized = cropped_image_pil.resize((160, 80))\n",
    "        small_img = np.array(cropped_image_resized)\n",
    "        small_img = small_img[None,:,:,:]\n",
    "        \n",
    "        # Make prediction with neural network (un-normalize value by multiplying by 255)\n",
    "        prediction = model.predict(small_img)[0] * 255\n",
    "        \n",
    "        # Add lane prediction to list for averaging\n",
    "        self.recent_fit.append(prediction)\n",
    "        # Only using last five for average\n",
    "        if len(self.recent_fit) > 5:\n",
    "            self.recent_fit = self.recent_fit[1:]\n",
    "        \n",
    "        # Calculate average detection\n",
    "        self.avg_fit = np.mean(np.array([i for i in self.recent_fit]), axis=0)\n",
    "        \n",
    "        # Generate fake G & B color dimensions, stack with R\n",
    "        blanks = np.zeros_like(self.avg_fit).astype(np.uint8)\n",
    "        lane_drawn = np.dstack((blanks, blanks, self.avg_fit))\n",
    "        \n",
    "        # Re-size to match the cropped image size\n",
    "        lane_image_pil = Image.fromarray(np.uint8(lane_drawn))\n",
    "        lane_image_resized = lane_image_pil.resize(cropped_image.shape[1::-1])  # Get width and height from cropped image\n",
    "        lane_image = np.array(lane_image_resized)\n",
    "        \n",
    "        # Merge the predicted lane with the cropped image\n",
    "        combined_cropped = cv2.addWeighted(cropped_image, 1, lane_image, 1, 0)\n",
    "        \n",
    "        # Replace pixels in the region of interest with combined_cropped pixels\n",
    "        result = image.copy()\n",
    "        height_of_cropped = combined_cropped.shape[0]\n",
    "        result[-height_of_cropped-50:-50, 300:-300] = combined_cropped\n",
    "        \n",
    "        # Draw a red line 200px above the bottom of the result image\n",
    "        cv2.line(result, (300, result.shape[0]-50), (result.shape[1]-300, result.shape[0]-50), (0, 0, 255), 2)\n",
    "        \n",
    "        return result\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnsemblePipeline():\n",
    "    def __init__(self, cal_path):\n",
    "        self.pipeline1 = Pipeline(cal_path)\n",
    "        self.pipeline2 = Pipeline2()\n",
    "\n",
    "    def __call__(self, image):\n",
    "        result_img = self.pipeline2.process(image)\n",
    "        focused_img = mask_red_box(image, result_img)\n",
    "        second_result = self.pipeline1(focused_img)\n",
    "\n",
    "        # ensemble_result = np.maximum(img1, img2)\n",
    "        return second_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_video2(input_video, output_video):\n",
    "\n",
    "    # Process video frames with our 'process_image' function\n",
    "    process_image = EnsemblePipeline('./camera_cal/*.jpg')\n",
    "    \n",
    "    ## You may uncomment the following line for a subclip of the first 5 seconds\n",
    "    #clip1 = VideoFileClip(input_video).subclip(0,5)\n",
    "    clip1 = VideoFileClip(input_video)\n",
    "    white_clip = clip1.fl_image(process_image)\n",
    "    %time white_clip.write_videofile(output_video, audio=False)\n",
    "    \n",
    "    print(\"Apply video pipeline: SUCCESS!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_image = 'test-vid2-frame.jpg'\n",
    "img = mpimg.imread(test_image)\n",
    "\n",
    "pipe = Pipeline2()\n",
    "pipe2 = Pipeline('./camera_cal/*.jpg')\n",
    "result_img = pipe.process(img)\n",
    "focused_img = mask_red_box(img, result_img)\n",
    "cv2.imwrite('keras_mask_result1.jpg', result_img)\n",
    "cv2.imwrite('keras_mask_result2.jpg', focused_img)\n",
    "img1 = pipe2(focused_img)\n",
    "cv2.imwrite('keras_mask_result3.jpg', img1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_video = 'test-vid.mp4'\n",
    "output_video = 'test-vid-output3.mp4'\n",
    "\n",
    "apply_video2(input_video, output_video)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Result\n",
    "We apply the pipeline to a test image. The original image and the processed image are shown side by side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_image = 'test-vid2-frame.jpg'\n",
    "# test_image = 'test_images/test2.jpg'\n",
    "img = mpimg.imread(test_image)\n",
    "\n",
    "# Process video frames with our 'process_image' function\n",
    "process_image = Pipeline('./camera_cal/*.jpg')\n",
    "\n",
    "# Apply pipeline\n",
    "processed = process_image(img)\n",
    "\n",
    "# Plot the 2 images\n",
    "f, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 9))\n",
    "f.tight_layout()\n",
    "ax1.imshow(img)\n",
    "ax1.set_title('Original Image', fontsize=50)\n",
    "ax2.imshow(processed, cmap='gray')\n",
    "ax2.set_title('Processed Image', fontsize=50)\n",
    "plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_video(input_video, output_video):\n",
    "\n",
    "    # Process video frames with our 'process_image' function\n",
    "    process_image = Pipeline('./camera_cal/*.jpg')\n",
    "    \n",
    "    ## You may uncomment the following line for a subclip of the first 5 seconds\n",
    "    #clip1 = VideoFileClip(input_video).subclip(0,5)\n",
    "    clip1 = VideoFileClip(input_video)\n",
    "    white_clip = clip1.fl_image(process_image)\n",
    "    %time white_clip.write_videofile(output_video, audio=False)\n",
    "    \n",
    "    print(\"Apply video pipeline: SUCCESS!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_video = 'test-vid.mp4'\n",
    "output_video = 'test-vid-output.mp4'\n",
    "\n",
    "apply_video(input_video, output_video)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"360\" controls>\n",
    "  <source src=\"{0}\">\n",
    "</video>\n",
    "\"\"\".format(output_video))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-11T09:20:34.333161800Z",
     "start_time": "2023-09-11T09:20:34.317108300Z"
    }
   },
   "source": [
    "\n",
    "# Discussion\n",
    "## Issues and Challenges\n",
    "---\n",
    "### Figuring out bad frames\n",
    "There will be some frames where no lanes will be detected or the lanes might not make sense. We determine the bad frames if any of the following conditions are met:\n",
    "\n",
    "1. No pixels were detected using the sliding window search or search around the previously detected line.\n",
    "2. The average gap between the lanes is less than 0.7 times pr greater than 1.3 times the globally maintained moving average of the lane gap.\n",
    "\n",
    "### Averaging lanes\n",
    "The lane for each frame is a simple average of 12 previously computed lanes. This is done in the get_averaged_line method in the code block below.\n",
    "\n",
    "### What to do if a bad frame is detected?\n",
    "1. Perform a sliding window search again (this is done in the detect_lines function)\n",
    "2. If this still results in a bad frame then we fall back to the previous well detected frame.\n",
    "\n",
    "### Final Pipeline\n",
    "We combine all the code described in the code block above, plus the averaging and fallback techniques described in this block. The final code is in the Pipeline class."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}