lesson-01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Connect Intensive - Machine Learning Nanodegree\n",
    "# Lesson 01: An intro to Statistical Analysis using `pandas`\n",
    "\n",
    "## Objectives\n",
    "  - Practice running python from within a [Jupyter Notebook](http://jupyter.org/) (FKA IPython Notebook).\n",
    "  - Become familiar with importing useful modules and packages, *e.g.* `pandas`, `numpy`, `matplotlib.pyplot`.\n",
    "  - Learn about the [`pandas` data structures](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#), including the `Series` and `DataFrame` objects.\n",
    "  - Create a `DataFrame` object from data in a comma-separated variable (csv) file using [`pandas.read_csv`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)\n",
    "  - [Index and select data](http://pandas.pydata.org/pandas-docs/stable/indexing.html) from `Series` and `DataFrame` objects using `loc` and `iloc`\n",
    "  - Compute descriptive statistics on a `Series` or `DataFrame`, including the [`mean`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.mean.html), the [`median`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.median.html), and the [`min`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.min.html) & [`max`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.max.html)\n",
    "  - Explore a public data set found on [Kaggle](https://www.kaggle.com/)\n",
    "  - Conduct some exploratory data analysis, and visualize trends in data using `matplotlib`\n",
    "  \n",
    "## Prerequisites\n",
    "  - You should be able to run Lesson 00: Hello Jupyter Notebook!\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Welcome!\n",
    "Hello astute young machine learnist, I'm glad you're here to learn more about statistical analysis! One of the major skills you can bring to the table as a machine learnist is the ability to explore and understand a data set. The library **`pandas`** is a Python package developed by Wes McKinney that machine learnists use to quickly and efficiently navigate data sets. From [the pandas documentation](http://pandas.pydata.org/pandas-docs/stable/index.html):\n",
    "\n",
    "> \"**`pandas`** is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive.\"\n",
    "\n",
    "Fun fact: the name \"`pandas`\" derives from **pan**el **da**ta, a term for multi-dimensional data sets! [(source)](http://www.dlr.de/sc/Portaldata/15/Resources/dokumente/pyhpc2011/submissions/pyhpc2011_submission_9.pdf)\n",
    "\n",
    "## There's plenty to learn!\n",
    "Depending on your background, a lot of the information and code in this Jupyter Notebook may be new to you. But don't worry, lots of documentation and references have been hyperlinked for you. Feel free to click on any of the hyperlinks to learn more -- the links should open within a new tab, so you won't lose your progress in the Jupyter Notebook. If you're still feeling uncertain about a topic, often the best place to start is a quick web search. For example, if Python is throwing me an error I've never seen before, I'll often copy and paste the error right into a Google search and see what comes up!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First things first: Import statements\n",
    "In Jupyter Notebooks, one of the first things we want to do is import useful modules and packages that will provide more functionality to our code. For example, we will want to import `pandas` so that we can manipulate data using the `DataFrame` object. We also want to have our Jupyter Notebook create figures and plots directly within the notebook. To do this, we use the [magic function](http://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magics-explained) `%matplotlib inline`. For more information on plotting within the IPython kernel, [check this out](http://ipython.readthedocs.io/en/stable/interactive/plotting.html)!\n",
    "\n",
    "**Run** the cell below (**click** on the cell to highlight it, then press **shift + enter** or **shift + return** to run it) to import modules and libraries for this Jupyter Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "try:\n",
    "    import numpy as np\n",
    "    print(\"Successfully imported numpy! (Version {})\".format(np.version.version))\n",
    "except ImportError:\n",
    "    print(\"Could not import numpy!\")\n",
    "\n",
    "    \n",
    "try:\n",
    "    import warnings\n",
    "    with warnings.catch_warnings():\n",
    "        warnings.simplefilter(\"ignore\");\n",
    "        import matplotlib\n",
    "        import matplotlib.pyplot as plt\n",
    "    plt.style.use('ggplot')\n",
    "    print(\"Successfully imported matplotlib.pyplot! (Version {})\".format(matplotlib.__version__))\n",
    "except ImportError:\n",
    "    print(\"Could not import matplotlib.pyplot!\")\n",
    "\n",
    "    \n",
    "try:\n",
    "    import pandas as pd\n",
    "    print(\"Successfully imported pandas! (Version {})\".format(pd.__version__))\n",
    "    pd.options.display.max_rows = 10\n",
    "except ImportError:\n",
    "    print(\"Could not import pandas!\")\n",
    "\n",
    "    \n",
    "try:\n",
    "    from IPython.display import display\n",
    "    print(\"Successfully imported display from IPython.display!\")\n",
    "except ImportError:\n",
    "    print(\"Could not import display from IPython.display\")\n",
    "\n",
    "    \n",
    "try:\n",
    "    from zipfile import ZipFile\n",
    "    print(\"Successfully imported ZipFile from zipfile!\")\n",
    "except ImportError:\n",
    "    print(\"Could not import ZipFile from zipfile\")\n",
    "    \n",
    "try:\n",
    "    import os.path\n",
    "    print(\"Successfully imported os.path!\")\n",
    "except ImportError:\n",
    "    print(\"Could not import os.path\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting from a ZIP archive\n",
    "Let's create our first DataFrame using pandas! You can learn more about the DataFrame object from [the pandas documentation on DataFrames](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html). The data set that we will play around with comes from [Kaggle datasets](https://www.kaggle.com/datasets). The user Eric Grinstein crawled the [ign game review page]() for video game review data from the last 20 years, and put the [video game data on Kaggle](https://www.kaggle.com/egrinstein/20-years-of-games). The csv file `'ign.csv'` containing the data is zipped within the archive `'20-years-of-games.zip'`.\n",
    "\n",
    "**Run** the cell below to extract the csv file from the ZIP archive using [the zipfile module](https://docs.python.org/2/library/zipfile.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The dataset is compressed within the zip file in this directory called '20-years-of-games.zip'\n",
    "zip_file_name = '20-years-of-games.zip'\n",
    "\n",
    "# Create a ZipFile object using the zip file name\n",
    "zf = ZipFile(file=zip_file_name)\n",
    "\n",
    "# Within the zip file is a comma-separated values (csv) file containing the games data named 'ign.csv'\n",
    "in_file_name = 'ign.csv'\n",
    "\n",
    "# Extract the games data into the current directory from the zip file \n",
    "zf.extract(member=in_file_name)\n",
    "\n",
    "# Close the ZipFile object -- we won't need it any more\n",
    "zf.close()\n",
    "\n",
    "# Print a success message if the csv file was extracted from the zip file\n",
    "if os.path.isfile(in_file_name):\n",
    "    print(\"The file {} has been extracted!\".format(in_file_name))\n",
    "else:\n",
    "    print(\"Could not extract the file {}\".format(in_file_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our first `DataFrame`!\n",
    "Now that we've extracted the file `'ign.csv'` from the ZIP archive containing the video game review data, we can create a `DataFrame` object from the csv file. The first line of the csv file contains the **feature** or **attribute** names, while each subsequent line in the file describes one **instance** or **input** of the data.\n",
    "\n",
    "**Run** the cell below to use `pd.read_csv` to read the csv into a `DataFrame` object that we will call `df`. Then, the first 5 lines of the `DataFrame` will be displayed using `df.head(5)`. For more information on these methods, the documentation for [`read_csv`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) and [`head`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.head.html) is a good place to start!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(in_file_name)\n",
    "display(df.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing non-useful features\n",
    "Our `DataFrame` has a lot of information for each game: a text descriptor for the game's rating called `'score_phrase'`, the `'title'` of the game, the `'url'` of the game review, the `'platform'` or console, the `'score'` from the review, *etc*. However, the first column of the `DataFrame`, the column labeled `'Unnamed: 0'` appears to simply repeat the [index of the instance](http://pandas.pydata.org/pandas-docs/stable/indexing.html). We don't really need to keep `Unnamed: 0`, because we can get the same information using the code `df.index.values`. So let's delete the first column:\n",
    "\n",
    "**Run** the cell below to delete the column `'Unnamed: 0'` from the `DataFrame` object `df`. Then, display the first few lines of the data frame to see what it looks like without `'Unnamed: 0'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if 'Unnamed: 0' in df.columns:\n",
    "    del df['Unnamed: 0']\n",
    "    print(\"Deleted the column 'Unnamed: 0' from df!\")\n",
    "else:\n",
    "    print(\"The column 'Unnamed: 0' has already been deleted!\")\n",
    "display(df.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing `DataFrame` and `Series` objects\n",
    "On the leftmost edge of the `DataFrame`, we can see the index. Each row (instance, input) in the `DataFrame` has an index. To access a specific row based on the index, we can use `loc` or `iloc`. Label-based indexing is done with `loc`, while integer-position based indexing is done with `iloc`. For example, looking above, we can see that the first row in the `DataFrame` contains the game LittleBigPlanet for PS Vita. Let's get the first row (index 0) using `loc`!\n",
    "\n",
    "**Run** the cell below to get the first row of the `DataFrame` using `df.loc[0]`. What does the result look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.loc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result doesn't look like a DataFrame! That's because one-dimensional objects in `pandas` are `Series` objects. `Series` objects are displayed as columns, with the indices shown on the left and the values shown on the right. Below the `Series` object, we see the name of the `Series` object and the `dtype` or data type of the `Series` object. The `dtype` of a `Series` object is chosen to accomodate all data within the `Series`.\n",
    "\n",
    "What if we don't want the entire first row of the DataFrame, but just the `score_phrase`?\n",
    "\n",
    "**Run** the cell below to see one way to get the `score_phrase` from the first row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.loc[0,'score_phrase']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`'Amazing'`, right? There are a lot of ways we can get that `score_phrase` from the first row of the `DataFrame`.\n",
    "\n",
    "**Run** the cells below to see many other ways to get the same `score_phrase` from the first row of the `DataFrame`. Do you understand how each line works?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.iloc[0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.loc[0].loc['score_phrase']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['score_phrase'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.iloc[0].iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['score_phrase'].loc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get multiple rows from the `DataFrame` by doing `numpy`-like slicing: `df.iloc[lower:upper]` will take a slice of the `DataFrame` object from the lower bound `lower` up to (but not including) the upper bound `upper`. Be careful! We get different results by slicing the `DataFrame` with `loc` and with `iloc`.\n",
    "\n",
    "When slicing a `DataFrame` using `iloc` (the *integer-based* position indexing) the lower bound is included, while the upper bound is excluded.\n",
    "\n",
    "**Run** the cell below to get the first three rows of the `DataFrame` using `df.iloc[0:3]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.iloc[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lower bound (0) is *included*, while the upper bound (3) is *excluded*.\n",
    "\n",
    "However, when slicing a `DataFrame` using `loc` (the *label-based* indexing), the starting and upper bounds are **both** included!\n",
    "\n",
    "**Run** the cell below to see what happens when we call `df.loc[0:3]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.loc[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the lower bound (0) and the upper bound (3) are *included* with *label-based* indexing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises - Part 1\n",
    "Now it's your turn to play with `pandas` to answer questions using the ign game review data set. If you're not sure how to do something, feel free to ask questions, read up the `pandas` documentation, or refer to the code examples above! For this section, the `pandas` documentation on [indexing and slicing](http://pandas.pydata.org/pandas-docs/stable/indexing.html) might be most useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1:\n",
    "What is the `'title'` of the game in the row with index 144?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2:\n",
    "What is the `'platform'` of the game in the row with index 2353?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3:\n",
    "What do rows 500 up to (but not including) row 505 look like? Were any of those games `'editors_choice'`? Were any of those games `'Awful'`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4:\n",
    "What does `df.iloc[0:100:10]` return? How would you describe integer-based slicing when there are three numbers in the square brackets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5:\n",
    "What does `df.iloc[:8]` return? How would you describe integer-based slicing when no lower bound is specified?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6:\n",
    "What does `df.iloc[-3:]` return? How would you describe integer-based slicing with a negative lower bound and no upper bound?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How much data do we have?\n",
    "So we have our `DataFrame` object and we're able to slice it using `loc` and `iloc`, but we might be wondering exactly how much data is in the dataset. We can call  `len(df)` method to count the number of rows in the `DataFrame` object.\n",
    "\n",
    "**Run** the cell below to count the number of rows in the `DataFrame` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"There are {} rows in the DataFrame\".format(len(df)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[The method `df.count()`](http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.count.html) gives you even more information than `len(df)`! For each feature (column) in the `DataFrame` object, `df.count()` will count up the number of non-NA/null values for that feature. This is useful to get an idea of how many missing values are in your `DataFrame`, if any.\n",
    "\n",
    "**Run** the cell below to count the number of non-NA/null values for each feature using `df.count()`. Is there a feature with missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "display(df.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More about the `Series` and `DataFrame` objects\n",
    "**Note:** This section assumes that you are familiar with basic data structures, in particular Python dictionaries. If you need to brush up on this aspect of Python, check out the [Python tutorial on data structures](https://docs.python.org/3/tutorial/datastructures.html)!\n",
    "\n",
    "The `DataFrame` object is a two-dimensional labeled data structure. As we saw above, each row is labeled with an index and each column is labeled with a feature name. Recall that if we take a single row from the DataFrame, it becomes a `Series` object. Similarly, if we take a single column of the `DataFrame` object, we would get a `Series` object. Thus, we can think of the `DataFrame` as a Python dictionary. The key-value pairs for this dictionary are the feature names and the `Series` objects. You can retrieve a single `Series` object from a `DataFrame` the same way you would retrieve a value from a dictionary: `df['title']` would retrieve the column `'title'` from `df` as a `Series` object. Let's try it!\n",
    "\n",
    "**Run** the cell below to print the first ten lines of the `Series` object containing the all of the game titles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "display(df['title'].head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We also see the `Name` and `dtype` (data type) of the `Series` object. It looks like `title` has a data type of `object` -- we'll learn more about that in a bit.\n",
    "\n",
    "Suppose we don't just want the `title` from `df`. What if we want both the `title` and `score`? We can access more than one feature of a `DataFrame` object with a list of keys rather than a single key:\n",
    "\n",
    "**Run** the cell below to print the first ten lines of just the `'title'` and `'score'` features of the `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "display(df[['title','score']].head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We accessed two features from the `DataFrame` object, and got a new `DataFrame` object as a result! That's a neat feature of `pandas`: one-dimensional labeled data structures become `Series` objects, while two-dimensional labeled data structures remain as `DataFrame` objects!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Data\n",
    "A `DataFrame` often contains `Series` where the variable takes on one of a small, discrete set of values. Some examples include:\n",
    "\n",
    "  - day of the week {Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday}\n",
    "  - a person's blood type {A+, A-, B+, B-, AB+, AB-, O+, O-}\n",
    "  - a movie's rating {G, PG, PG-13, R, NC-17}\n",
    "  - number of stars for a review {zero, one, two, three, four}\n",
    "  - *Can you think of other similar examples?*\n",
    "\n",
    "\n",
    "The `'categorical'` data type allows pandas to efficiently store a `Series` object that encodes data like in the examples above. Some examples of using the `'categorical'` data type are:\n",
    "\n",
    "  - it uses less memory than the `'object'` data type, as long as the number of categories is *much less than* the length of the `Series`\n",
    "  - it can be sorted on *logical* order (e.g. zero, one, two, three, four) rather than *lexical* or alphabetical order (e.g. four, one, three, two, zero)\n",
    "  - other statistical methods are well-suited to categorical variables.\n",
    "  \n",
    "The [pandas documentation on categorical data](http://pandas.pydata.org/pandas-docs/stable/categorical.html#) has a lot more information, so feel free to explore more!\n",
    "\n",
    "**Run** the cell below to compare memory usage when a `Series` object has relatively few vs. relatively many categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Comparison of memory usage when the number of categories is SMALL:\n",
    "print('df[\\'genre\\'] has {} distinct categories among {} values'.format(\n",
    "        len(df['genre'].astype('category').cat.categories), len(df['genre'])))\n",
    "print('df[\\'genre\\'] takes up {:6d} bytes of memory as an \\'object\\''.format(df['genre'].astype('object').nbytes))\n",
    "print('df[\\'genre\\'] takes up {:6d} bytes of memory as a  \\'category\\'\\n'.format(df['genre'].astype('category').nbytes))\n",
    "\n",
    "# Comparison of memory usage when the number of categories is LARGE:\n",
    "print('df[\\'url\\'] has {} distinct categories among {} values'.format(\n",
    "        len(df['url'].astype('category').cat.categories), len(df['url'])))\n",
    "print('df[\\'url\\'] takes up {:6d} bytes of memory as an \\'object\\''.format(df['url'].astype('object').nbytes))\n",
    "print('df[\\'url\\'] takes up {:6d} bytes of memory as a  \\'category\\'\\n'.format(df['url'].astype('category').nbytes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it looks like we only get the memory advantage of the `'category'` dtype over the `'object'` dtype for features with a relatively small amount of distinct categories. From our data set, it looks like the features `'score_phrase'`, `'platform'`, `'genre'`, and `'editors_choice'` are good candidates to convert from the `'object'` dtype to the `'category'` dtype.\n",
    "\n",
    "**Run** the cell below to perform these dtype conversions, and then check to see if they converted correctly by printing `df.dtypes`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['genre'] = df['genre'].astype('category')\n",
    "df['platform'] = df['platform'].astype('category')\n",
    "df['score_phrase'] = df['score_phrase'].astype('category')\n",
    "df['editors_choice'] = df['editors_choice'].astype('category')\n",
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Descriptive statistics on a `Series` object\n",
    "\n",
    "Now that we know a bit more about the `Series` and `DataFrame` objects in `pandas`, let's start to explore the utility to keeping our data within these structures. The main quantitative feature in our data set is the `'score'` column. Let's do some descriptive statistics on that feature!\n",
    "\n",
    "### Computing the mean\n",
    "\n",
    "The mean or average is one measure of central tendency for a data set. The mean of a set of values is the sum of all values, divided by the number of values. We can compute the mean of the `Series` object `df['score']` using [the `sum()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sum.html) to add up all the scores, and then dividing by the length of the `Series` object\n",
    "\n",
    "**Run** the cell below to store the `Series` object `df['score']` into the variable `scores`, and then compute the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "scores = df['score']\n",
    "print(\"The mean of the scores is {:.2f}\".format(scores.sum() / len(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Computing the mean was pretty easy, but there's an even simpler way! We can use [the `mean()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.mean.html) to compute the same thing!\n",
    "\n",
    "**Run** the cell below to compute the mean of `scores` using the `mean()` method. Do we find the same result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The mean of the scores is {:.2f}\".format(scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Computing the median\n",
    "The median is another measure of central tendency for a data set. When the data is arranged from smallest to largest, the median is the value right in the middle of the ordered data set. With an *odd* number of values in the data set $(2n+1)$, one value will be right in the middle, with $n$ values above it and $n$ values below it. With an *even* number of values in the data set $(2n)$, *two* values will be in the middle, with $n-1$ values above them and $n-1$ values below them: in this case, the median is actually the average of the two values right in the middle.\n",
    "\n",
    "**Run** the cell below to compute the median by sorting `scores` in ascending order with the method [sort_values()](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sort_values.html#pandas.Series.sort_values) and then finding the value(s) right in the middle of the data set with [the iloc indexer](http://pandas.pydata.org/pandas-docs/version/0.17.1/generated/pandas.DataFrame.iloc.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "scores = scores.sort_values(ascending=True)\n",
    "n = len(scores)\n",
    "if n % 2 == 1:\n",
    "    median = scores.iloc[(n-1)//2]\n",
    "else:\n",
    "    median = (scores.iloc[n//2 - 1] + scores.iloc[n//2]) / 2.0\n",
    "print(\"The median of the scores is {:.2f}\".format(median))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was a little more work than computing the mean, but we can be even more efficient! We can use [the `median()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.median.html) to compute the same thing!\n",
    "\n",
    "**Run** the cell below to compute the median of scores using the `median()` method. Do we find the same result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The median of the scores is {:.2f}\".format(scores.median()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the maximum and minimum values\n",
    "The maximum and the minimum values within a data set give us an idea of the range for that feature. Because we sorted `scores` in an eariler cell, the smallest (minimum) value within `scores` is at the start of the `Series` object, while the largest (maximum) value within `scores` is at the end of the `Series` object. Recall that the indexing convention in Python starts with 0 as the first index, and we can use the index -1 to retrieve the last item.\n",
    "\n",
    "**Run** the cell below to print the minimum and maximum values from the sorted `scores` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The minimum and maximum scores are {:.1f} and {:.1f}, respectively\".format(scores.iloc[0],scores.iloc[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find the minimum and maximum values using [the `min()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.min.html) and [the `max()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.max.html), which don't require the `Series` object to be sorted.\n",
    "\n",
    "**Run** the cell below to print the minimum and maximum values using the `min()` and `max()` methods. Do we find the same results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The minimum and maximum scores are {:.1f} and {:.1f}, respectively\".format(scores.min(),scores.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting instances\n",
    "We have lots of video game data, but we might not want to average all the scores together. We may want to select certain instances of the `DataFrame` before processing the data. To select certain instances, we can use [Boolean indexing](http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing).\n",
    "\n",
    "Before we get into Boolean indexing, we should see what happens when we compute a logical expression using a `Series` object. For example, let's see what happens when we check whether the `'editors_choice'` feature is equal to `'Y'`.\n",
    "\n",
    "**Run** the cell below to compute the logical expression `df['editors_choice'] == 'Y'`. What is the result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['editors_choice'] == 'Y'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The logical expression above returns a `Series` object of dtype `bool`, with a value of `True` at the indices where `df['editors_choice']` is equal to `'Y'`, and `False` at the indices where `df['editors_choice']` is equal to `'N'`. \n",
    "\n",
    "**Run** the cell below to see what happens when we index a `DataFrame` object using this `Series` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df[df['editors_choice'] == 'Y']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see what Boolean Indexing does -- it returns a DataFrame keeping only the rows where the logical expression was `True`. So we are able to easily select instances using a logical expression. We can use other methods to return `Series` objects of dtype `bool`. For example, we will use [the method `Series.str.contains()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html) below to return all games that contain the string `'Mario'` in their title.\n",
    "\n",
    "**Run** the cell below to select only the instances where the title contains the string `'Mario'` and display the first few `'Mario'` games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mario_df = df[df['title'].str.contains('Mario')]\n",
    "display(mario_df.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've selected the instances where the `title` feature contains the `'Mario'` string, we can check to see if Mario games get higher scores from ign than the entire population of reviewed games.\n",
    "\n",
    "**Run** the block of code below to display the mean ign review score for Mario games and compare it to the overall mean. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The mean score is {:.2f} for Mario games, compared to the mean score {:.2f} for all reviewed games.\".format(\n",
    "    mario_df['score'].mean(), df['score'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also check the average score for games released in the last year... are we living in a golden age of gaming?\n",
    "\n",
    "**Run** the block of code below to display the mean ign review score for games with `release_year` of 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_2016 = df[df['release_year']==2016]\n",
    "print(\"The mean score of the {} games released in 2016 is {:.2f}.\".format(len(df_2016),df_2016['score'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use [the `isin()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.isin.html) to see if the value of a feature is equal to any of the values within a list. This is useful if we want to pick items that fall within a category that is not necessarily numerical or lexical. For example, if we want to consider ign's reviews of mobile gaming, we might want to select only the instances from `df` where the `'platform'` feature is either `'Android'`, `'iPhone'`, or `'Windows Phone'`.\n",
    "\n",
    "**Run** the block of code below to display the mean score for mobile games, where the `'platform'` is either `'Android'`, `'iPhone'`, or `'Windows Phone'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mobile_df = df[df['platform'].isin(['Android', 'iPhone', 'Windows Phone'])]\n",
    "print(\"The mean score of {} mobile games is {:.2f}.\".format(len(mobile_df),mobile_df['score'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Describing and grouping\n",
    "Before you try to solve some data problems on your own, there are a few other `pandas` techniques that will help you with your initial data exploration. The first is [the `describe()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html), that provides a descriptive summary of all numerical features within a `DataFrame` object.\n",
    "\n",
    "**Run** the cell below to see what happens when you call `df.describe()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "display(df.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that `describe()` returns the count, the mean, the standard deviation, the minimum, the 25th, 50th, and 75th percentiles, and the maximum for features with numeric dtypes. Pretty handy!\n",
    "\n",
    "The last thing I want to cover is [the `groupby()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html). We can combine the `groupby()` method with descriptive statistics methods, such as `min()`, `max()`, `mean()`, and `median()` to apply the function to each groups. For example, let's use `groupby()` to see the mean for each `platform`.\n",
    "\n",
    "**Run** the code below to try your first `groupby()` command!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.groupby('platform')['score'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty awesome, huh! With one line of code, we are able to compute the mean review score for each platform. Even more awesome is that we can use that result to pick out the results for specific platforms, and we can also plot the results really efficiently by just putting `.plot()` at the end!\n",
    "\n",
    "**Run** the cell below to use `groupby()` to plot the results for only a few specific consoles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "consoles = ['Game Boy','Game Boy Color','Game Boy Advance']\n",
    "ax = df.groupby('platform')['score'].mean()[consoles].plot(kind=\"bar\")\n",
    "xlabel = ax.set_xlabel(\"Console\")\n",
    "ylabel = ax.set_ylabel(\"Average Score (out of 10)\")\n",
    "ylim = ax.set_ylim([0,10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope you're now appreciating how powerful `pandas` is as a tool for exploratory data analysis. Let's do one more advanced example of plotting using `groupby()` where we group by two features:\n",
    "\n",
    "**Run** the cell below to generate a plot of the average release score for games from each year, for games with and without the editor's choice label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ax = df\\\n",
    "    [df['release_year']>=2000]\\\n",
    "    .groupby(['editors_choice','release_year'])\\\n",
    "    ['score']\\\n",
    "    .mean()\\\n",
    "    .unstack('editors_choice')\\\n",
    "    .plot(title=\"IGN review scores from 2000 to present\")\n",
    "xlabel = ax.set_xlabel(\"Release Year\")\n",
    "ylabel = ax.set_ylabel(\"Average Score (out of 10)\")\n",
    "ylim = ax.set_ylim([0,10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a lot going on in the cell above. Note that the first seven lines are a single command broken over several lines with slashes for easier comprehension. Let's break it all down:\n",
    "  - In line 1, we start with the data frame `df`. Eventually this long line of code will return a `matplotlib.axes.AxesSubplot` object, so we store the result into `ax`.\n",
    "  - In line 2, we use boolean indexing to select only the instances with games released in the year 2000 or later.\n",
    "  - In line 3, we use the `groupby()` method to group the data first by `'editors_choice'` and then by `'release_year'`.\n",
    "  - In line 4, we keep only the `'score'` feature of the data frame.\n",
    "  - In line 5, we compute our desired descriptive statistic: the mean(). Note that because we've grouped by `'editors_choice'`, and then by `'release_year'`, each feature has two indices. The outermost index (level 0) is `'editors_choice'` and the innermost index (level -1) is `'release_year'`. So we will have the mean of editors-choice games and the mean of non-editors-choice games for each year.\n",
    "  - In line 6, [the method unstack('editors_choice')](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.unstack.html) pivots the `'editors_choice'` index, and each possible value of `'editors_choice'` becomes a column.\n",
    "  - In line 7, we make a plot with one line for each column. The x-axis is the only remaining index, which is `'release_year'`\n",
    "  - The last three lines use the `ax` object to set the labels and y-axis limits.\n",
    "\n",
    "More helpful hints on visualization can be found [in the pandas documentation](http://pandas.pydata.org/pandas-docs/stable/visualization.html#)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, you may **run** the three cells below to make more sense of what the `unstack()` method is actually doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Here is the DataFrame unstacking index 0 (editors_choice)\n",
    "df\\\n",
    "    [df['release_year']>=2000]\\\n",
    "    .groupby(['editors_choice','release_year'])\\\n",
    "    ['score']\\\n",
    "    .mean()\\\n",
    "    .unstack(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Here is the DataFrame unstacking index 1 (release_year)\n",
    "df\\\n",
    "    [df['release_year']>=2000]\\\n",
    "    .groupby(['editors_choice','release_year'])\\\n",
    "    ['score']\\\n",
    "    .mean()\\\n",
    "    .unstack(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Here is the Series without unstacking either index. Note that each instance (row) has two indices.\n",
    "df\\\n",
    "    [df['release_year']>=2000]\\\n",
    "    .groupby(['editors_choice','release_year'])\\\n",
    "    ['score']\\\n",
    "    .mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises - Part 2\n",
    "Now it's your turn to play with `pandas` to answer questions using the ign game review data set. If you're not sure how to do something, feel free to ask questions, read up the `pandas` documentation, or refer to the code examples above!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1:\n",
    "What is the mean IGN review score for games released in October (where the `release_month` is 10)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2:\n",
    "How many games were reviewed by IGN that were released for the `'Genesis'` platform?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3:\n",
    "On average, does IGN rate games higher in the `'Platformer'` genre, or in the `'Action'` genre?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4:\n",
    "How many 'Mario' games have been reviewed by IGN? What about 'Kirby' games? Which games does IGN tend to review more highly?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5:\n",
    "Make a bar plot with the number of games reviewed, grouped by `release_month`. Which `release_month` has had the most reviewed games? The fewest?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6:\n",
    "Now make a bar plot with the mean score, grouped by `release_month`. Which `release_month` has the highest-reviewed games? The lowest?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 7:\n",
    "Which `'score_phrase'` was the most commonly used by IGN in the year 2008? The least common?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 8:\n",
    "Make a plot showing how the average scores for games in the genres `'Party'`, `'RPG'`, and `'Sports'` have changed from the year 2000 to the year 2010. How do the trends look for each genre? Does one genre appear to be consistently reviewed higher or lower than the other two? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}