US-aiports-student.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory analysis of the US Airport Dataset\n",
    "\n",
    "This dataset contains data for 25 years[1995-2015] of flights between various US airports and metadata about these routes. Taken from Bureau of Transportation Statistics, United States Department of Transportation.\n",
    "\n",
    "Let's see what can we make out of this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import networkx as nx\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "pass_air_data = pd.read_csv('data/passengers.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `pass_air_data` dataframe we have the information of number of people that fly every year on a particular route on the list of airlines that fly that route."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a MultiDiGraph from this dataset\n",
    "passenger_graph = nx.from_pandas_edgelist(pass_air_data, source='ORIGIN', target='DEST',\n",
    "                                          edge_attr=['YEAR', 'PASSENGERS', 'UNIQUE_CARRIER_NAME'],\n",
    "                                          create_using=nx.MultiDiGraph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleveland to Chicago, how many people fly this route?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passenger_graph['JFK']['SFO'][24]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = [(i['YEAR'], i['PASSENGERS'])for i in dict(passenger_graph['JFK']['SFO']).values()]\n",
    "x, y = zip(*temp)\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Find the busiest route in 1990 and in 2015 according to number of passengers, and plot the time series of number of passengers on these routes.\n",
    "\n",
    "You can use the DataFrame instead of working with the network. It will be faster ;)\n",
    "[5 mins]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data['ORIGIN'] == 'LAX') & (pass_air_data['DEST'] == 'HNL')].plot('YEAR', 'PASSENGERS')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data['ORIGIN'] == 'LAX') & (pass_air_data['DEST'] == 'SFO')].plot('YEAR', 'PASSENGERS')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract year graph from passenger_graph\n",
    "def year_network(G, year):\n",
    "    temp_g = nx.DiGraph()\n",
    "    for i in G.edges(data=True):\n",
    "        if i[2]['YEAR'] == year:\n",
    "            temp_g.add_edge(i[0], i[1], weight=i[2]['PASSENGERS'])\n",
    "    return temp_g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_2015 = year_network(passenger_graph, 2015)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(pass_2015)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(pass_2015.edges())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in the GPS coordinates of all the airports\n",
    "lat_long = pd.read_csv('data/GlobalAirportDatabase.txt', delimiter=':', header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lat_long[lat_long[1].isin(list(pass_2015.nodes()))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_dict = {}\n",
    "for airport in lat_long[lat_long[1].isin(list(pass_2015.nodes()))].iterrows():\n",
    "    pos_dict[airport[1][1]] = (airport[1][15], airport[1][14]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Using the position dictionary `pos_dict` create a plot of the airports, only the nodes not the edges.\n",
    "\n",
    "- As we don't have coordinates for all the airports we have to create a subgraph first.\n",
    "- Use `nx.subgraph(Graph, iterable of nodes)` to create the subgraph\n",
    "- Use `nx.draw_networkx_nodes(G, pos)` to map the nodes. \n",
    "\n",
    "or \n",
    "\n",
    "- Just use a scatter plot :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 9))\n",
    "G = nx.subgraph(pass_2015, _______)\n",
    "nx.draw_networkx_nodes(G, pos=__________, node_size=10, alpha=0.6, node_color='b')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 9))\n",
    "x = [i[0] for i in pos_dict.values()]\n",
    "y = [i[1] for i in pos_dict.values()]\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about degree distribution of this network?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(list(nx.degree_centrality(pass_2015).values()))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot a log log plot to get a better overview of this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {}\n",
    "for i, j in dict(nx.degree(pass_2015)).items():\n",
    "    if j in d:\n",
    "        d[j] += 1\n",
    "    else:\n",
    "        d[j] = 1\n",
    "x = np.log2(list((d.keys())))\n",
    "y = np.log2(list(d.values()))\n",
    "plt.scatter(x, y, alpha=0.4)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Directed Graphs\n",
    "\n",
    "![title](images/pagerank.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G = nx.DiGraph()\n",
    "\n",
    "G.add_edge(1, 2, weight=1)\n",
    "\n",
    "# print(G.edges())\n",
    "# G[1][2]\n",
    "# G[2][1]\n",
    "# G.is_directed()\n",
    "# type(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G.add_edges_from([(1, 2), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2)])\n",
    "nx.draw_circular(G, with_labels=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G.in_degree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.pagerank(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G.add_edge(5, 6)\n",
    "nx.draw_circular(G, with_labels=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.pagerank(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G.add_edge(2, 8)\n",
    "nx.draw_circular(G, with_labels=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.pagerank(G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Moving back to Airports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.pagerank(pass_2015, weight=None).items(), key=lambda x:x[1], reverse=True)[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.betweenness_centrality(pass_2015).items(), key=lambda x:x[1], reverse=True)[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.degree_centrality(pass_2015).items(), key=lambda x:x[1], reverse=True)[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'ANC' is the airport code of Anchorage airport, a place in Alaska, and according to pagerank and betweenness centrality it is the most important airport in this network Isn't that weird? Thoughts?\n",
    "\n",
    "related blog post: https://toreopsahl.com/2011/08/12/why-anchorage-is-not-that-important-binary-ties-and-sample-selection/\n",
    "\n",
    "Let's look at weighted version, i.e taking into account the number of people flying to these places."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.betweenness_centrality(pass_2015, weight='weight').items(), key=lambda x:x[1], reverse=True)[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.pagerank(pass_2015, weight='weight').items(), key=lambda x:x[1], reverse=True)[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How reachable is this network?\n",
    "\n",
    "We calculate the average shortest path length of this network, it gives us an idea about the number of jumps we need to make around the network to go from one airport to any other airport in this network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.average_shortest_path_length(pass_2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait, What??? This network is not connected. That seems like a really stupid thing to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(nx.weakly_connected_components(pass_2015))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SPB, SSB, AIK anyone?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data['YEAR'] == 2015) & (pass_air_data['ORIGIN'] == 'AIK')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_2015.remove_nodes_from(['SPB', 'SSB', 'AIK'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.is_weakly_connected(pass_2015)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.is_strongly_connected(pass_2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strongly vs weakly connected graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G = nx.DiGraph()\n",
    "G.add_edge(1, 2)\n",
    "G.add_edge(2, 3)\n",
    "G.add_edge(3, 1)\n",
    "nx.draw(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G.add_edge(3, 4)\n",
    "nx.draw(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.is_strongly_connected(G)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(nx.strongly_connected_components(pass_2015))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data['YEAR'] == 2015) & (pass_air_data['DEST'] == 'TSP')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_2015_strong = max(nx.strongly_connected_component_subgraphs(pass_2015), key=len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(pass_2015_strong)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.average_shortest_path_length(pass_2015_strong)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise! (Actually this is a game :D)\n",
    "\n",
    "How can we decrease the avg shortest path length of this network?\n",
    "\n",
    "Think of an effective way to add new edges to decrease the avg shortest path length.\n",
    "Let's see if we can come up with a nice way to do this, and the one who gets the highest decrease wins!!!\n",
    "\n",
    "The rules are simple:\n",
    "- You can't add more than 2% of the current edges( ~500 edges)\n",
    "\n",
    "[10 mins]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.average_shortest_path_length(pass_2015_strong)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about airlines? Can we find airline specific reachability?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passenger_graph['JFK']['SFO'][25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def str_to_list(a):\n",
    "    return a[1:-1].split(', ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in str_to_list(passenger_graph['JFK']['SFO'][25]['UNIQUE_CARRIER_NAME']):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "for origin, dest in passenger_graph.edges():\n",
    "    for key in passenger_graph[origin][dest]:\n",
    "        passenger_graph[origin][dest][key]['airlines'] = str_to_list(passenger_graph[origin][dest][key]['UNIQUE_CARRIER_NAME'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Play around with United Airlines network.\n",
    "\n",
    "- Extract a network for United Airlines flights from the metagraph `passenger_graph` for the year 2015\n",
    "- Make sure it's a weighted network, where weight is the number of passengers.\n",
    "- Find the number of airports and connections in this network\n",
    "- Find the most important airport, according to PageRank and degree centrality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "united_network = nx.DiGraph()\n",
    "for origin, dest in passenger_graph.edges():\n",
    "    if 25 in passenger_graph[origin][dest]:\n",
    "        if \"'United Air Lines Inc.'\" in passenger_graph[origin][dest][25]['airlines']:\n",
    "            united_network.add_edge(origin, dest, weight=passenger_graph[origin][dest][25]['PASSENGERS'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(united_network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(united_network.edges())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.pagerank(united_network, weight='weight').items(), key=lambda x:x[1], reverse=True)[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(nx.degree_centrality(united_network).items(), key=lambda x:x[1], reverse=True)[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.average_shortest_path_length(united_network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "We are in New York so what should we do?\n",
    "\n",
    "Obviously we will make a time series of number of passengers flying out of EWR/JFK/LGA with United Airlines over the years.\n",
    "\n",
    "There are 2 ways of doing it.\n",
    "- Create a new multidigraph specifically for this exercise.\n",
    "\n",
    "OR\n",
    "\n",
    "- exploit the `pass_air_data` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data.ORIGIN == 'EWR') &\n",
    "              (pass_air_data.UNIQUE_CARRIER_NAME.str.contains('United Air Lines Inc.'))\n",
    "             ].groupby('YEAR')['PASSENGERS'].sum().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data.ORIGIN == 'JFK') &\n",
    "              (pass_air_data.UNIQUE_CARRIER_NAME.str.contains('United Air Lines Inc.'))\n",
    "             ].groupby('YEAR')['PASSENGERS'].sum().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pass_air_data[(pass_air_data.ORIGIN == 'LGA') &\n",
    "              (pass_air_data.UNIQUE_CARRIER_NAME.str.contains('United Air Lines Inc.'))\n",
    "             ].groupby('YEAR')['PASSENGERS'].sum().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}