data_manipulation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\"> Python for the Humanities and Social Sciences <br> <em>Data Manipulation</em> </h1>\n",
    "\n",
    "## Info\n",
    "- Scott Bailey (CIDR), *scottbailey@stanford.edu*\n",
    "- Javier de la Rosa (CIDR), *versae@stanford.edu*\n",
    "- Ashley Jester (CIDR/SSDS), *ajester@stanford.edu*\n",
    "\n",
    "## Goal\n",
    "By the end of our workshop today, we hope you'll be able to load in data into a Pandas `DataFrame`, perform basic cleaning and analysis, and visualize relevant aspects of a dataset. We will work with a dataset of tweets collected during the release of the Apple Watch.\n",
    "\n",
    "## Topics\n",
    "- Pandas Series and DataFrame\n",
    "- Loading data in, null and missing data\n",
    "- Describing data\n",
    "- Column manipulation\n",
    "- String manipulation\n",
    "- Split-Apply-Combine\n",
    "- Plotting:\n",
    "  - Basic charts (line, bar, pie)\n",
    "  - Histograms\n",
    "  - Scatter plots\n",
    "  - Boxplots, violinplots\n",
    "\n",
    "## Setup and packages we need in our environment\n",
    "We'll be using Anaconda with Jupyter Notebooks for this workshop. For setting up both, please see the [setup](setup.ipynb).\n",
    "\n",
    "For this workshop, we'll need an environment with the following packages:\n",
    "- `matplotlib`\n",
    "- `pandas`\n",
    "- `requests`\n",
    "- `seaborn`, available in the `conda-forge` channel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas\n",
    "\n",
    "From Jake Vanderplas' book [**Python Data Science Handbook**](http://shop.oreilly.com/product/0636920034919.do) (from which some code excerpts are used in this workshop):\n",
    "\n",
    "> Pandas is a newer package built on top of NumPy, and provides an efficient implementation of a `DataFrame`. `DataFrame`s are essentially multidimensional arrays with attached row and column labels, and often with heterogeneous types and/or missing data. As well as offering a convenient storage interface for labeled data, Pandas implements a number of powerful data operations familiar to users of both database frameworks and spreadsheet programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np  # np becomes the namespace of numpy\n",
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "# Set some options\n",
    "pd.set_option('display.max_columns', 20)\n",
    "pd.set_option('display.max_rows', 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three main data structures in Pandas: `Series`, `DataFrame`, and `Index`. Pandas has a very decent [documentation](http://pandas.pydata.org/pandas-docs/stable/), and using Jupyter, any method help can be shown by appending the a `?` to the end and running the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# For example\n",
    "pd.isnull?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data I/O"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas provides a few methods to load in and out data in CSVs, Excel spreadsheets, HDF, or even JSON format.\n",
    "\n",
    "For example, click in the next URL of a CSV file containing twitter data during the release of the Apple Watch: http://bit.ly/python_workshop_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pandas can fetch data from a URL\n",
    "pd.read_csv(\"http://bit.ly/python_workshop_data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the previous data to a local file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"twitter.csv\", \"wb\") as file:\n",
    "    file.write(requests.get(\"http://bit.ly/python_workshop_data\").content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_csv(\"twitter.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's reload the CSV but this time specifying a index column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"twitter.csv\", index_col=\"created_at\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can just save the clean data to any format supported by Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"twitter_indexed.csv\", encoding=\"utf8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `DataFrame` and `Series`\n",
    "\n",
    "A `DataFrame` is a two-dimensional array with both flexible row indices and flexible column names. It can be seen as a generalization of a two-dimensional NumPy array, or a specialization of a dictionary in which each column name maps to a `Series` of column data. A `Series` is a one-dimensional array of indexed data. It can be seem as a specialized dictionary or a generalized NumPy array.\n",
    "\n",
    "A `DataFrame` is made up of `Series` in a similar way in which a table is made up of columns. The only restriction os that ach column must be of the same data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"twitter.csv\")\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing columns can be done using the dot notation, `df.column_name`, or the dictionary notation, `df[\"column_name\"]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"urls\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataFrame`s can be sliced to extract just a set of the columns you are interested in. We just pass in a list of the columns we need to the slice and get a `DataFrame` back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[[\"urls\", \"text\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All `DataFrame`s are indexed. If an index is not explictly provided Pandas will asign one, givinh each row a consecutive number. `Series` and slices keep these indices, which makes possible further operations such as merging or columns manipulation.\n",
    "\n",
    "`DataFrames` are designed to operate at the column level, not the row level. However, a subset of rows can be visualized easily using a slice like in any Python list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[10:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.urls[10:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[[\"urls\"]][10:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you can even access individual rows and mix index and rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[[\"urls\", \"text\"]].loc[2:5]  # for non numeric indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[[\"urls\", \"text\"]].iloc[2:5]  # for nummeric indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.ix[2:5, [\"urls\", \"text\"]]  # for mixed indices and columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Given the `DataFrame` defined above, write an expression to extract a `DataFrame` with the columns `text`, `user_screen_name`, `user_name`, `user_lang`, and `hashtags`. Show only the first 5 rows of it.\n",
    "<br/>\n",
    "<!-- * **Hint**: You could ...* -->\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write here your solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing and Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operations performed using a column or `Series` are broadcast to each of the elements contained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"id\"] * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"@\" + df[\"user_name\"] + \": \" + df[\"text\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"id\"] > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which allows for a more advanced and useful indexing as you can pass in an expression to a `DataFrame` to select content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df[\"id\"] > 575043732472528896]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically any expression that evaluates to a `Series` of `True` and `False` values and share the index can be used. And conditions can be put together using logical operators for \"and\", `&`, \"or\", `|`, and \"not\", `~`, making the filter more precise and expressive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[(df[\"id\"] > 575043732472528896) & (len(df[\"user_mentions\"]) > 5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some string operations are also available at the column level on the `.str` attribute of `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df[\"urls\"].str.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the previous selection could also be written as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[(df[\"id\"] > 575043732472528896) & (df[\"user_mentions\"].str.len() > 5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Given the `states` `DataFrame` defined below, write an expression to calculate the population density of each state.\n",
    "<br/>\n",
    "* **Hint**: Population density is defined as the number of people per unit of area.*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "population_dict = {'California': 38332521, 'Texas': 26448193, 'New York': 19651127,\n",
    "                   'Florida': 19552860, 'Illinois': 12882135}\n",
    "area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297,\n",
    "             'Florida': 170312, 'Illinois': 149995}  # these are in km²\n",
    "states = pd.DataFrame({'population': population_dict, 'area': area_dict})\n",
    "states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fundamental way of manipulating the contents of `DataFrame` columns is by using the `apply()` method, which allows to call a user defined function to each of the elements in the `Series`. Unlike the `.str` attribute, `apply()` is a general way of transforming values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_links(text):\n",
    "    links = text.split(\",\")\n",
    "    count = len(links)\n",
    "    return count\n",
    "\n",
    "df[\"urls\"].apply(count_links)  # urls are separated by comma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However our naive `count_links` function does not know how to handle missing data. We could ignore those values by dropping the `NaN`, which is the Pandas way of saying missing data, or by cleaning our dataset on import time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"urls\"].dropna().apply(count_links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cleaning the data at the beginning, at import time, and for the whole `DataFrame` is usually a good idea, since makes operating with it more consistent and lesss prone to error.\n",
    "\n",
    "This also avoids us the hassle to drop `NaN`'s everytime. In our case we will:\n",
    "- Filter out some columns we are not interested in\n",
    "- Specify and index for thr `DataFrame`\n",
    "- Provide data types for some columns\n",
    "- Parse dates as Python `datetime` for columns containing dates as strings\n",
    "- Replace `NaN` values by empty strings in string columns\n",
    "\n",
    "And then show the first 5, this time using the `head()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = [\n",
    "    \"created_at\", \"id\",\n",
    "    \"text\", \"lang\", \"possibly_sensitive\", \"user_screen_name\",\n",
    "    \"hashtags\", \"media\", \"symbols\", \"urls\",\n",
    "    \"place\", \"country\"]  # columns we want\n",
    "index_column = \"created_at\"\n",
    "column_types = {\n",
    "    \"id\": int,\n",
    "    \"possibly_sensitive\": bool,\n",
    "    \"lat\": float,\n",
    "    \"lon\": float,\n",
    "}\n",
    "fill_nans = {\n",
    "    'country': '',\n",
    "    'hashtags': '',\n",
    "    'lang': '',\n",
    "    'media': '',\n",
    "    'place': '',\n",
    "    'symbols': '',\n",
    "    'text': '',\n",
    "    'urls': '',\n",
    "    'user_lang': '',\n",
    "    'user_location': '',\n",
    "    'user_name': '',\n",
    "    'user_screen_name': ''\n",
    "}\n",
    "date_columns = [\"created_at\"]\n",
    "df = pd.read_csv(\"twitter.csv\",\n",
    "    parse_dates=date_columns,\n",
    "    index_col=index_column,\n",
    "    usecols=columns,\n",
    "    dtype=column_types).fillna(value=fill_nans)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, our `count_links` should work just fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"urls\"].apply(count_links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And since the result of `appply()` is another `Series`, we can even create a new column with the it to enrich a `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"urls_count\"] = df[\"urls\"].apply(count_links)\n",
    "df[[\"urls\", \"urls_count\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we now wanted to know the distribution or histogram of the number of links, we could use the `.value_counts()` method of `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"urls_count\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Given the twitter `DataFrame`, add a new column `length` with the length ot the `text`, and show the tweets with exactly 140 characters.\n",
    "<br/>\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n",
    "df[\"length\"] = df[\"text\"].apply(...)\n",
    "df[...][[\"text\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Series` also have some handy functions to compute basic statistics, like the sum or the mean. For example, given the new column created above, let's compute the average lenght of the tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"length\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping data\n",
    "\n",
    "But what about the most tweeted language? Or the most prolific user? For this kind of operations we need to use what is called the [Split-Apply-Combine](https://www.jstatsoft.org/article/view/v040i01/v40i01.pdf) approach:\n",
    "- *Split* up a dataset\n",
    "- *Apply* a function to each piece\n",
    "- *Combine* all the pieces back together\n",
    "\n",
    "<figure>\n",
    "  <img src=\"https://swcarpentry.github.io/r-novice-gapminder/fig/splitapply.png\" alt=\"Split-Apply-Combine\">\n",
    "  <figcaption>* Split-Apply-Combine - Source: [Software Carpentry](https://software-carpentry.org/lessons/). *</figcaption>\n",
    "</figure>\n",
    "\n",
    "In Pandas this can take the form of a `.groupby()` (split) operation followed by an `.aggregate()` (apply) function. Aggregates are like `apply()` that operate at the group level. Combining is done automatically for us by Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby(\"lang\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby(\"lang\")[[\"text\"]]  # no computation is made yet!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_nonzero(items):\n",
    "    total = 0\n",
    "    for item in items:\n",
    "        if item != 0:\n",
    "            total += 1\n",
    "    return total\n",
    "\n",
    "df.groupby(\"lang\")[[\"text\"]].aggregate(count_nonzero)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataFrames` can be sorted by the values of one or more columns, in either ascending or descending order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregated = df.groupby(\"lang\")[[\"text\"]].aggregate(count_nonzero)\n",
    "aggregated.sort_values(\"text\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, for complex groupings like, creating a pivot table can be more useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.pivot_table(\n",
    "    index=[\"lang\", \"user_screen_name\"],\n",
    "    values=[\"text\"],\n",
    "    aggfunc=count_nonzero\n",
    ").sort_values(\"text\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Given the twitter `DataFrame`, show the most popular retweet written in English.\n",
    "<br/>\n",
    "* **Hint**: In our dataset, retweets are tweets that start with \"RT @\".*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas also provides some utilities to create basic plots just by calling `plot()` on a `Series` or `DataFrame`. But first we need to tell Jupyter that we are going to plot some charts using the plotting library matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enables inline plotting in Jupyter using matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby(\"lang\")[[\"lang\"]].aggregate(count_nonzero).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each time you call `plot()` an `Axes` object is returned, and Jupyter knows how to paint those. `Axes` objects are objects of the underlying `matplotlib` library for plotting in Python, and as such, lots of different options can be given to customize the aspect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.groupby(\"lang\")[[\"lang\"]].aggregate(count_nonzero).plot(\n",
    "    kind=\"bar\",\n",
    "    figsize=(15, 5),\n",
    "    title=\"# Tweets per Language\",\n",
    "    legend=None\n",
    ")\n",
    "ax.set_ylabel(\"Languagae\")\n",
    "ax.set_xlabel(\"# Tweets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Axes` can also be created empty using `matplotlib` and then put some content in them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(15, 5))\n",
    "df.groupby(\"lang\")[[\"lang\"]].aggregate(count_nonzero).plot(ax=ax,\n",
    "    kind=\"bar\",\n",
    "    title=\"# Tweets per Language\",\n",
    "    legend=None\n",
    ")\n",
    "ax.set_ylabel(\"Languagae\")\n",
    "ax.set_xlabel(\"# Tweets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other styles available as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with plt.style.context('ggplot'):\n",
    "    df.groupby(\"lang\")[[\"lang\"]].aggregate(count_nonzero).plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Even a special one for XKCD!\n",
    "with plt.xkcd():\n",
    "    df.groupby(\"lang\")[[\"lang\"]].aggregate(count_nonzero).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`seaborn`, a convenience wrapper around `matplotlib`, changes the default style after being imported, but it can be reverted back easily setting the default style to `classic` using `plt.style.use(\"classic\")`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "df.groupby(\"lang\")[[\"lang\"]].aggregate(count_nonzero).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a hitogram with the lengths of tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(15, 5))\n",
    "df[\"length\"].hist(ax=ax, bins=15, normed=True, color='lightseagreen')\n",
    "df[\"length\"].plot(ax=ax, kind='kde', xlim=(0, 150), style='r--')\n",
    "ax.set_title(\"Histogram of lengths of tweets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplots are available by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(8, 6))\n",
    "df.boxplot(column=\"length\", grid=False, ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although violinplots can be used through `seaborn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(8, 6))\n",
    "sns.violinplot(y=df[\"length\"], grid=False, ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Given the twitter `DataFrame`, let's try to find out visually if there is any sort of relationship between the length of a tweet and the number of hastags it uses.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here\n",
    "df[\"hashtags_count\"] = \n",
    "\n",
    "fig, ax = plt.subplots(1, figsize=(15, 5))\n",
    "ax.scatter(x=..., y=...)\n",
    "ax.set_ylabel(\"Length\")\n",
    "ax.set_xlabel(\"# Hashtags\")\n",
    "ax.set_title(\"Tweet length by number of hashtags\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}