Lecture 3 - Applied AI Course (17 July, 2017).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class 3 -- Applied AI Course (17 July, 2017)\n",
    "\n",
    "### Outline of the class\n",
    "\n",
    "#### Recap of the first and second class\n",
    "1. Scatter Plots (1D, 2D, ...)\n",
    "2. Pairplots, Box plots, Q-Q plots\n",
    "3. Gaussian Distribution\n",
    "4. Probability Density Function (PDFs) and Cummulative Distribution Function (CDFs)\n",
    "5. Mean, Variance, Standard Deviation\n",
    "6. Discussion for over 40 minutes on KPI (Key Performance Indicator) by considering an example of what should be the measure of response time for a search query at Amazon (any other company, for that matter). Is it the mean of response times, median, variance, standard deviation, difference between maximum response time and the mean, percentiles or any other metric. \n",
    "\n",
    "#### Introduced the terms Parametric and Non-parametric statistics\n",
    "* We should not consider Mean, Variance and Standard Deviation as our KPI because these parameter can be easily affected by Outliers/Noise\n",
    "* So we should consider some parameters which are robust to noise, such are Median and Percentile. These are called Non-parametric statistics. We shall learn in detail about what these non-parametric stats are and the related concepts in further classes.\n",
    "* We have median in Non-parametric stats which is like mean in parametric, we have IQR and MAD in non-parametric like we have variance in parametric.\n",
    "\n",
    "#### Inter Quartile Range (IQR)\n",
    "* The distance from max to the $75^{th}$ percentile and the $25^{th}$ are given by $1.5*IQR$ \n",
    "* Inter Quartile Deviation is defined as the amount of spread in the middle 50% of the dataset. In other words, it is the distance between first quartile and the third quartile.\n",
    "* Meaning that, in a box plot, the distance between 75th percentile and the 25th percentile is defined as IQR. \n",
    "    $IQR = X_{75\\%ile}-X_{25\\%ile}$\n",
    "    \n",
    "\n",
    "1. **Calculating IQR**\n",
    "    1. Sort the given data in non-decreasing order\n",
    "    2. Find the median. If the number of data points is odd, the median is the middle data point. If the number of data points is even, the median is the average of the middle two data points. \n",
    "    3. Find the first quartile $Q_1$ (i.e., 25th percentile). The first quartile is the median of the data points to the left of the median in the sorted list.\n",
    "    4. Find the third quartile $Q_3$ (i.e., 75th percentile). The third quartile is the median of the data points to the right of the median in the sorted list.\n",
    "    5. Calculate $IQR = Q_3 - Q_1$.\n",
    "\n",
    "2. **Example:** Find the Inter Quartile Deviation for the data 4, 3, 6, 4, 1, 3, 3, 6, 4\n",
    "    1. Sort the data: 1, 3, 3, 3, 4, 4, 4, 6, 6\n",
    "    2. Find the median. There are 9 data points, so the median is the middle data point. $\\therefore $ 4 is the median. So, the list is 1, 3, 3, 3, <font color='red'>4</font>, 4, 4, 6, 6\n",
    "    \n",
    "    3. Find $Q_1$, which is the median of the data to the left of the median. There are 4 data points to the left of the median: 1, 3, 3, 3. The median of these 4 points is $Q_1=\\frac{3+3}{2}=3$. So the first quartile or $25^{th}$ percentile is 3.\n",
    "    4. Find $Q_3$, which is the median of the data to the right of the median. There are 4 data points: 4, 4, 6, 6. The median of these 4 points is $Q_3=\\frac{4+6}{2}=5$. So the third quartile or $75^{th}$ percentile is 5.\n",
    "    5. Inter Quartile Deviation (IQR) for the given data is $Q_3-Q_1=5-3=2$\n",
    "3. **Example 2:** Find the outliers, if any, for the following data set: 10.2,  14.1,  14.4.   14.4,  14.4,  14.5,  14.5,  14.6,  14.7,   14.7,  14.7,  14.9,  15.1,  15.9,   16.4\n",
    "    1. To find out if there are any outliers, we first have to find the IQR. There are fifteen data points, so the median will be at the eighth position: 14.6\n",
    "    2. There are 7 data points to the either side of the median. The two halves are 10.2,  14.1,  14.4.   14.4,  14.4,  14.5,  14.5 and 14.7,   14.7,  14.7,  14.9,  15.1,  15.9,   16.4. So, $Q_1=14.4$ and $Q_3=14.9$.\n",
    "    3. $IQR=Q_3-Q_1=14.9-14.4=0.5$\n",
    "    4. The outliers will be \n",
    "        1. any points below $Q_1-1.5\\times IQR=14.4-1.5\\times 0.5=13.65$ and \n",
    "        2. any points above $Q_3+1.5\\times IQR=14.9+1.5\\times 0.5=15.65$\n",
    "    5. So the outliers are 10.2, 15.9 and 16.4\n",
    "\n",
    "3. Inter Quartile Deviation tells us about how spread out the data is (more spread data tends to have large IQR), but it is insensitive to outliers (particularly compared to standard deviation).\n",
    "\n",
    "\n",
    "#### Median Absolute Deviation (MAD)\n",
    "The median absolute deviation (MAD) is a robust measure of how spread out the data is. Variance and standard deviation are also measures of spread, but they are more affected by extremely high or extreamly low values and non normality, generally called outliers. IQR is similar to MAD and is simpler statistics, but it's less robust. \n",
    "\n",
    "$$MAD=b\\times median\\{| x_i - X_{median} |\\}, \\forall i\\in n$$ \n",
    "\n",
    "where n is the number of datapoints and $b=1.4826$ if the data follows normal distribution or $b$ is the $75^{th}$ percentile or $Q_3$. <a href=\"http://www.sciencedirect.com/science/article/pii/S0022103113000668\">(Leys, et al. 2013)</a> (I think you need to look at the Q-Q plot or perform some normality testing to see if your data follows normal distribution or not.)\n",
    "1. **Calculating MAD:**\n",
    "    1. Calculate median, $X_{median}$.\n",
    "    2. Calculate how far away each data point is from the median using positive distances. These are called absolute deviations.\n",
    "    3. Find the median of these deviations, which gives us the MAD. $$MAD = b\\times median\\{|x_1-X_{median}|, |x_2-X_{median}|,..., |x_n-X_{median}|\\}$$  \n",
    "    4. The outliers will be:\n",
    "        1. any points below $X_{median}-2\\times MAD$ and \n",
    "        2. any points above $X_{median}+2\\times MAD$\n",
    "    5. We can use either 3 or 2.5 instead of the multipication factor 2 while computing the outliers. It depends on how conservative you are. More the value, more the conservative. 3 if you are more conservative and 2 if you are less conservative. \n",
    "        \n",
    "2. **Example 1:** Erica enjoys posting pictures of her cat online. Here's how many \"likes\" the past 666 pictures each received: 10, 15, 15, 17, 18, 21 (For this example, ignore the multiplication factor $b$)\n",
    "    1. Median of the given datapoints is $\\frac{15+17}{2}=16$.\n",
    "    2. Absolute deviations (distances) from each datapoint to the median:\n",
    "        1. $|10-16|=6$\n",
    "        2. $|15-16|=1$\n",
    "        3. $|15-16|=1$\n",
    "        4. $|17-16|=1$\n",
    "        5. $|18-16|=2$\n",
    "        6. $|21-16|=5$\n",
    "    3. Median of all the absolute deviations: $MAD=Median\\{6,1,1,1,2,5\\} = Median\\{1,1,1,2,5,6\\} = \\frac{1+2}{2}=1.5$\n",
    "    4. Finding outliers:\n",
    "        1. any point below $X_{median}-2\\times MAD=16-2\\times 1.5=13$\n",
    "        2. any point above $X_{median}+2\\times MAD=16+2\\times 1.5=19$\n",
    "    5. So, the outliers are 10 and 21. Remember, the outliers may vary based on the multiplication factor. \n",
    "    \n",
    "3. **Example 2:** Your friend Jay has taken a weekly botany quiz for 9 weeks and consistently received scores between 83 and 86. Her mean score is 84.6 with a very narrow spread. Jay's twin brother, Out, has the identical scores for the first 8 weeks. For some reason, maybe illness, the result of Out's ninth quiz is very low: a 45. This score throws off Out's performance report and produces a spread which misrepresents the actual consistency of his quiz scores. Find the median absolute deviation for Out's scores: 83, 83, 84, 85, 85, 86, 86, 83 and 45 and also find the outliers if any, using 2 as multiplication factor. \n",
    "    1. Median of given scores, $X_{median} = 84$.\n",
    "    2. Absloute deviations from each score to the median:{39, 1, 1, 1, 0, 1, 1, 2, 2}\n",
    "    3. MAD = Median of absolute deviations {0, 1, 1, 1, 1, 1, 2, 2, 39} is 1.\n",
    "    4. Finding outliers:\n",
    "        1. any point below $X_{median}-2\\times MAD=84-2\\times1=82$\n",
    "        2. any point above $X_{median}+2\\times MAD=84+2\\times1=86$\n",
    "    5. So only one outlier 45 is detected, hence this score should not be considered in Out's performance report :) \n",
    "    \n",
    "*If your data follows Gaussian distribution, then standard deviation is usually the best choice for assessing spread. However, if your data isn't normal, then MAD is one statistic you can use instead.*\n",
    "\n",
    "#### Breakdown point\n",
    "\n",
    "One way of looking at breakdown point is the number of outliers or percentage of outliers that will result in an error in the estimator like median, IQR and MAD. We define breakdown points especially for non-parametric statistics or robust statistics. \n",
    "* If we have n data points and we let a minority of them $floor(\\frac{n-1}{2})$ go to infinity leaving the rest fixed, then the median stays with the majority. The median changes, but does not become arbitrarily bad. The finite sample breakdown point is $floor(\\frac{n-1}{2n})$. The asymptotic breakdown point is one-half, 50%. \n",
    "\n",
    "Informally, if $\\geq 50\\%$ of the data is corrupted (or containing outliers) then median can be corrupted. So, breakdown point for median is 50%."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}