lesson-02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Connect Intensive - Machine Learning Nanodegree\n",
    "# Lesson 02: Working with the Enron Data Set\n",
    "\n",
    "## Objectives\n",
    "  - Work through the Datasets and Questions lesson from [ud-120](https://www.udacity.com/course/intro-to-machine-learning--ud120) within the Jupyter Notebook environment\n",
    "  - Introduce [the `pickle` module](https://docs.python.org/2/library/pickle.html) from the Python Standard Library for preserving data structures in Python\n",
    "  - Practice loading data from different directories\n",
    "  - Review stacking and unstacking in `pandas` to [reshape a `DataFrame`](http://pandas.pydata.org/pandas-docs/stable/reshaping.html)\n",
    "  - Explore and understand data sets, particularly those with many missing values\n",
    "  \n",
    "## Prerequisites\n",
    "  - You should be familiar with `pandas` (lesson 01 in the [ConnectIntensive repo](https://github.com/nickypie/ConnectIntensive))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background Information\n",
    "Before you start working through this Jupyter Notebook, you should probably watch the following videos from ud-120 (they're all conveniently compiled in your Classroom under the Data Modeling section).\n",
    "  - Introduction\n",
    "  - What is a POI\n",
    "  - Accuracy vs. Training Set Size\n",
    "  - Downloading Enron Data\n",
    "  - Types of Data (Quizzes 1-6)\n",
    "  - Enron Dataset Mini-Project Video\n",
    "  \n",
    "[Katie Malone](http://blog.udacity.com/2016/04/women-in-machine-learning-katie-malone.html) provides background on the Enron scandal, introduces the email corpus, defines a person-of-interest (POI), and poses an interesting ML question to motivate the project (\"Can we identify patterns in the emails of people who were POIs?\"). The videos provide excellent insight into the workflow for a Machine Learnist or Data Scientist.\n",
    "\n",
    "This Jupyter Notebook is intended to provide a friendly guide through the \"Datasets and Questions\" lesson... but if you're feeling pretty confident about your Python skills, consider going off-script! Try to work through the lesson on your own -- you may encounter some snags, and you can always refer back to this Notebook if you need a little push forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started\n",
    "The Datasets and Questions lesson in the Data Modeling section draws from Enron finance and email data. These datasets are found in the [**ud120-projects** repo](https://github.com/udacity/ud120-projects) on GitHub. Please fork and clone the **ud120-projects** repo to your local machine if you haven't done so already (if you need a refresher on how to fork and clone from the command line, [check out this link from GitHub](https://help.github.com/articles/fork-a-repo/)). *Be sure to keep track of where you save the local clone of this repo on your machine!* We'll need the location of that directory in just a bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In a `pickle`\n",
    "Suppose you're working with Python and you assemble nice data structures (*e.g.* dictionaries, lists, tuples, sets...) that you'll want to re-use in Python at a later time. [The `pickle` module](https://docs.python.org/2/library/pickle.html) is a fast, efficient way to preserve (or pickle) those data structures without you needing to worry about how to structure or organize your output file. One nice thing about using `pickle` is that the data structures you store can be arbitrarily complex: you can have nested data structures (*e.g.* lists of tuples as the values in a dictionary) and `pickle` will know exactly how to serialize (or write) those structures to file! The next time you're in Python, you can un-pickle the data structures using `pickle.load()` and pick up right where you left off!\n",
    "\n",
    "For a better explanation of `pickle` than I could hope to put together, please check out [this reference on Serializing Python Objects](http://www.diveintopython3.net/serializing.html) from Dive Into Python 3.\n",
    "\n",
    "**Run** the cell below to import the `pickle` module. (Don't forget, **shift + enter** or **shift + return** runs the active cell in a Jupyter Notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import pickle\n",
    "    print(\"Successfully imported pickle!\")\n",
    "except ImportError:\n",
    "    print(\"Could not import pickle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `path` to success\n",
    "Do you remember where you cloned the **ud120-projects** repo to your local machine? We need that information now! The **ud120-projects** directory contains a lot of folders and files, one of which is the Enron data within a Python dictionary, preserved using the `pickle` module. However, we need to tell this Jupyter Notebook where it can find the **ud120-projects** repo on our local machine. In the cell below, I have a [Magic Function](http://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magics-explained) `%cd \"...\"` that changes the working directory to the **ud120-projects** directory on my machine. Chances are, you didn't save the **ud120-projects** directory to the same place on your machine (although you may have, in which case, hello fellow nick!)\n",
    "\n",
    "**Update** the Magic Function `%cd \"...\"` in the cell below to reflect the correct path of the **ud120-projects** directory on your local machine.\n",
    "\n",
    "Then **run** the cell below to load the Enron data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Be sure to write the full path, up to and including \"ud120-projects\"\n",
    "%cd \"/Users/nick/Documents/Udacity/ud120-projects\"\n",
    "\n",
    "try:\n",
    "    enron_data = pickle.load(open(\"final_project/final_project_dataset.pkl\", \"rb\"))\n",
    "    print(\"Enron data loaded succesfully!\")\n",
    "except IOError:\n",
    "    print(\"No such file or directory! (Is there a problem with the path?)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From Dictionary to DataFrame\n",
    "At this point, the variable `enron_data` is a dictionary object. Dictionaries are not displayed as nicely as `pandas` `DataFrame` objects within the Jupyter Notebook environment. So let's convert `enron_data` to a `DataFrame` object! In the Jupyter Notebook lesson-01, we saw how to construct a `DataFrame` from a .csv file... we simply used the method `pd.DataFrame.read_csv()`. Fortunately, it's just as easy to create a `DataFrame` object from a dictionary object: we could use [the method `pandas.DataFrame.from_dict()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_dict.html) or simply use [the constructor `pandas.DataFrame()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) -- either one works!\n",
    "\n",
    "**Run** the cell below to:\n",
    "  - import `pandas` and `display`.\n",
    "  - set some display options.\n",
    "  - create a `DataFrame` object for the Enron data.\n",
    "  - display the Enron data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import pandas as pd\n",
    "    print(\"Successfully imported pandas! (Version {})\".format(pd.__version__))\n",
    "    pd.options.display.max_rows = 10\n",
    "except ImportError:\n",
    "    print(\"Could not import pandas!\")\n",
    "\n",
    "try:\n",
    "    from IPython.display import display\n",
    "    print(\"Successfully imported display from IPython.display!\")\n",
    "except ImportError:\n",
    "    print(\"Could not import display from IPython.display\")\n",
    "    \n",
    "enron_df = pd.DataFrame.from_dict(enron_data)\n",
    "display(enron_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking, unstacking, and rearranging\n",
    "\n",
    "Oh no, it looks like we created our DataFrame the wrong way! Each row of the `DataFrame` should correspond to a unique instance or input, while each column of the `DataFrame` should correspond to a unique feature or variable. The functions [`pandas.DataFrame.stack()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.stack.html) and [`pandas.DataFrame.unstack()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.unstack.html) will come to the rescue! First, we need to `stack` the current column indices, moving them to the innermost level of the row index.\n",
    "\n",
    "**Run** the cell below to see the results of calling `enron_df.stack()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "enron_df.stack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the result of `enron_df.stack()` is a `Series` object, where the innermost (rightmost) level of the index is the person's name in the Enron data set, while the outermost (leftmost) level of the index is the feature. If we call `unstack()` on the resulting `Series` without specifying a level, we'll just revert to the original `DataFrame`.\n",
    "\n",
    "**Run** the cell below to see the result of calling `enron_df.stack().unstack()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "enron_df.stack().unstack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The trick is, we need to `unstack` the *outermost* level of the index, but by default, the function will `unstack` the *innermost* level of the index.\n",
    "\n",
    "**Run** the cell below *once* to correctly `stack` and `unstack` the Enron `DataFrame` to move the instances (names) to rows and features (variables) to columns. *Be careful!* If you run this cell an even number of times, you will lose your changes to the `DataFrame`... can you see why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "enron_df = enron_df.stack().unstack(0)\n",
    "display(enron_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now that our `DataFrame` has the features and instances in the correct orientation, we can start to explore the data. But before we dive into the exercises, I'll leave you with one last reference for [reshaping `Series` and `DataFrame` objects in `pandas`](http://pandas.pydata.org/pandas-docs/stable/reshaping.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "Now it's your turn to play with `pandas` to answer questions using the Enron data set. If you're not sure how to do something, feel free to ask questions, look through [the `pandas` documentation](http://pandas.pydata.org/pandas-docs/stable/api.html), or refer to the code examples above!\n",
    "\n",
    "You can check your solutions to each of these exercises by entering your answer in the corresponding Quiz in the \"Datasets and Questions\" lesson. I put the corresponding quizzes in parenthesis after each exercise, so you know where to go to check your answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1\n",
    "How many data points (people) are in the data set? (Quiz: Size of the Enron Dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2\n",
    "For each person, how many features are available? (Quiz: Features in the Enron Dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3\n",
    "How many Persons of Interest (POIs) are in the dataset? (Quiz: Finding POIs in the Enron Data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4\n",
    "We compiled a list of all POI names (in `final_project/poi_names.txt`) and associated email addresses (in `final_project/poi_email_addresses.py`).\n",
    "\n",
    "How many POI’s were there total? Use the **names** file, not the email addresses, since many folks have more than one address and a few didn’t work for Enron, so we don’t have their emails. (Quiz: How Many POIs Exist?)\n",
    "\n",
    "**Hint:** Open up the `poi_names.txt` file to see the file format:\n",
    "  - the first line is a link to a USA Today article\n",
    "  - the second line is blank\n",
    "  - subsequent lines have the format: `(•) Lastname, Firstname`\n",
    "      - the dot `•` is either \"y\" (for yes) or \"n\" (for no), describing if the emails for that POI are available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5\n",
    "What might be a problem with having some POIs missing from our dataset? (Quiz: Problems with Incomplete Data)\n",
    "\n",
    "This is more of a \"free response\" thought question -- we don't really expect you to answer this using code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6\n",
    "What is the total value of the stock belonging to James Prentice? (Query The Dataset 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 7\n",
    "How many email messages do we have from Wesley Colwell to persons of interest? (Query The Dataset 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 8\n",
    "What's the value of stock options exercised by Jeffrey K Skilling? (Query The Dataset 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Questions 9-12 can be answered by some Google sleuthing** \n",
    "\n",
    "## Question 9\n",
    "Which of these schemes was Enron **not** involved in? (Quiz: Research the Enron Fraud)\n",
    "\n",
    "- selling assets to shell companies at the end of each month, and buying them back at the beginning of the next month to hide accounting losses\n",
    "- causing electrical grid failures in California\n",
    "- illegally obtained a government report that enabled them to corner the market on frozen concentrated orange juice futures\n",
    "- conspiring to give a Saudi prince expedited American citizenship\n",
    "- a plan in collaboration with Blockbuster movies to stream movies over the internet\n",
    "\n",
    "## Question 10\n",
    "Who was the CEO of Enron during most of the time that fraud was being perpetrated? (Quiz: Enron CEO)\n",
    "\n",
    "## Question 11\n",
    "Who was chairman of the Enron board of directors? (Quiz: Enron Chairman)\n",
    "\n",
    "## Question 12\n",
    "Who was CFO (chief financial officer) of Enron during most of the time that fraud was going on? (Quiz: Enron CFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 13\n",
    "Of the CEO, Chairman of the Board, and the CFO, who took home the most money? How much money was it? (Quiz: Follow the Money)\n",
    "\n",
    "*Hint:* Which of the three individuals has the largest value in the `'total_payments'` feature?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 14\n",
    "For nearly every person in the dataset, not every feature has a value. How is it denoted when a feature doesn’t have a well-defined value? (Quiz: Unfilled Features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 15\n",
    "How many folks in this dataset have a quantified salary? What about a known email address? (Quiz: Dealing with Unfilled Features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Magic Functions!\n",
    "\n",
    "In the Jupyter Notebook `lesson-01.ipynb`, we introduced our first [Magic Function](http://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magics-explained):\n",
    "\n",
    "`%matplotlib inline`\n",
    "\n",
    "That allowed us to generate plots using `matplotlib.pyplot` that appeared directly within our Jupyter Notebook! Earlier in this notebook, we also introduced the Magic Function:\n",
    "\n",
    "`%cd \"...\"`\n",
    "\n",
    "Replacing the ellipses in the above Magic Function with the location of the **ud120-projects** repo allowed us to change the working directory for this Jupyter Notebook session. Let's learn one last Magic Function:\n",
    "\n",
    "`%load filename.py`\n",
    "\n",
    "This allows us to quickly load python code from the file `filename.py` into the current cell!\n",
    "\n",
    "**Run** the cell below to execute the Magic Function `%load tools/feature_format.py` from the **ud120-projects** repo. What happens?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load tools/feature_format.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that the Magic Function `%load tools/feature_format.py` copies the contents of `feature_format.py` into the same cell, below the magic command. Additionally, the Magic Function `%load tools/feature_format.py` is now commented out! Note that you've **copied** the contents of `feature_format.py` into the cell with the Magic Function, but you need to run the cell a second time to actually **execute** the code.\n",
    "\n",
    "If you want to try another `%load` Magic Function, **run** the cell below to load the contents of `poi_email_addresses.py` to this Notebook! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load final_project/poi_email_addresses.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple last helpful hints about Magic Functions:\n",
    "  - You can learn more about Magic Functions at any time with the Magic Function `%magic`\n",
    "  - You can get the list of available Magic Functions by the Magic Function `%lsmagic`\n",
    "  - You can learn about any Magic Function by typing a question mark after it, *e.g.* `%load?`\n",
    "  \n",
    "**Try it below!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%magic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Exercises -- Missing POIs\n",
    "As you saw a little while ago, not every POI has an entry in the dataset (e.g. Michael Krautz). That’s because the dataset was created using the financial data you can find in `../final_project/enron61702insiderpay.pdf`, which is missing some POI’s (those absences propagated through to the final dataset). On the other hand, for many of these “missing” POI’s, we do have emails.\n",
    "\n",
    "While it would be straightforward to add these POI’s and their email information to the E+F dataset, and just put “NaN” for their financial information, this could introduce a subtle problem. You will walk through that here.\n",
    "\n",
    "Again, you can check your solutions to each of these exercises by entering your answer in the corresponding Quiz in the \"Datasets and Questions\" lesson. I put the corresponding quizzes in parenthesis after each exercise, so you know where to go to check your answers.\n",
    "\n",
    "## Question 1\n",
    "How many people in the E+F dataset (as it currently exists) have “NaN” for their total payments? What percentage of people in the dataset as a whole is this? (Quiz: Missing POIs 1 (Optional))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2\n",
    "How many POIs in the E+F dataset have “NaN” for their total payments? What percentage of POI’s as a whole is this? (Quiz: Missing POIs 2 (Optional))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3\n",
    "\n",
    "If a machine learning algorithm were to use total_payments as a feature, would you expect it to associate a “NaN” value with POIs or non-POIs? (Quiz: Missing POIs 3 (Optional))\n",
    "\n",
    "(Think about your answers from Questions 1 and 2 to answer this)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4\n",
    "\n",
    "If you added in, say, 10 more data points which were all POI’s, and put “NaN” for the total payments for those folks, the numbers you just calculated would change.\n",
    "\n",
    "What is the new number of people of the dataset? What is the new number of folks with “NaN” for total payments? (Quiz: Missing POIs 4 (Optional))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5\n",
    "\n",
    "What is the new number of POI’s in the dataset? What is the new number of POI’s with NaN for total_payments? (Quiz: Missing POIs 5 (Optional))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6\n",
    "\n",
    "Once the new data points are added, do you think a supervised classification algorithm might interpret “NaN” for total_payments as a clue that someone is a POI? (Quiz: Missing POIs 6 (Optional))\n",
    "\n",
    "(Think about your answers from Questions 1-5 to answer this)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}