02_data_analysis_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a8b3c41",
   "metadata": {
    "id": "6a8b3c41"
   },
   "source": [
    "<img src='images/Practicum_AI_Logo.white_outline.svg' width=250 alt='Practicum AI logo'> <img src='https://github.com/PracticumAI/practicumai.github.io/blob/main/images/icons/practicumai_python.png?raw=true' align='right' width=50>\n",
    "***\n",
    "\n",
    "# Data Analysis with pandas\n",
    "In this module, you will learn about one of the most popular and valuable Python libraries for data analysis and manipulation: ***pandas***!\n",
    "\n",
    "<img src='images/pandas_logo.png' alt='The pandas logo' align='center' width=200>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b3e4cd",
   "metadata": {},
   "source": [
    "### After this module, students will be able to:\n",
    "* Explain the various components of a pandas DateFrame.\n",
    "* Use `read_csv()` to create new DateFrames from data.\n",
    "* Subset data with pandas index and DateFrame management functions.\n",
    "* Use basic statistical functions to summarize a dataset.\n",
    "* Use matplotlib functions to generate clear graphs and plots.\n",
    "\n",
    "\n",
    "## <img src='images/get_started_icon.svg' alt=\"Let's get started header\" width=40 align=center> Let's Get Started!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b3af23e",
   "metadata": {},
   "source": [
    "As we learned in the last module, we must **import** the pandas library before we can use it.\n",
    "* We'll also import the scientific computing library NumPy for illustration purposes.\n",
    "* NumPy and pandas are usually imported with abbreviated aliases, `np` and `pd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e56bf5f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbf7bc4c-61fa-4863-8e6a-aa4a2d2497fb",
   "metadata": {},
   "source": [
    ">## <img src='images/alert_icon.svg' alt=\"Alert icon\" width=40 align=center> Did you get an error here?\n",
    ">\n",
    ">If you got an error running the cell above, it is probably because the environment you are working in doesn't have `numpy` or `pandas` installed.\n",
    ">\n",
    ">### For Google Colab\n",
    ">\n",
    ">On Google Colab, this shouldn't be an issue (for these libraries), and if you do run into this in the future, it is generally OK (and expected in Colab) to use `!pip install ___` to install a library. In Colab, each notebook is independent of other notebooks, so the installation of packages has limited impact. But, be especially careful doing this on HPC systems; installing things that cause compatibility issues is easy.\n",
    ">\n",
    ">### For Jupyter on HPC systems\n",
    ">\n",
    ">For most of what the *Practicum AI* courses cover, an environment (known as a **kernel**) will be ready for you to use. Check the kernel if you run into an error about a missing library. The current kernel is shown in the top right of the notebook:\n",
    ">![Screenshot of a Jupyter notebook showing the currently selected kernel](images/kernel.png)\n",
    ">For this notebook on HiPerGator, we will use the **UFRC Python-3.10** kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99857f15",
   "metadata": {},
   "source": [
    "## pandas Dataframes\n",
    "\n",
    "pandas provides one of the most useful data structures for managing data: the DataFrame.\n",
    "\n",
    "**DataFrames:**\n",
    "\n",
    "* Allow fast, flexible, and efficient data manipulation.\n",
    "* Are the natural data structure for **tabular** data.\n",
    "* Are two-dimensional, containing both **rows** and **columns**.\n",
    "* Consist of one or more **Series** (as columns).\n",
    "\n",
    "\n",
    "***pandas*** is built on top of the **NumPy** library, which in practice means that most of the methods defined for NumPy Arrays apply to pandas Series/DataFrames.\n",
    "\n",
    "**What makes pandas so useful for working with data?**\n",
    "1. Powerful and efficient interface to access and manipulate individual records.\n",
    "2. Tools for handling missing values and explicit values for missing data (`NaN` and `None`).\n",
    "3. Relational database-style operations between DataFrames."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc574944-2a60-429f-a6de-8e77d6498ddf",
   "metadata": {},
   "source": [
    "### <img src='images/note_icon.svg' alt=\"Note icon\" width=40 align=center> Note\n",
    "> Some of the examples used in this notebook are different from those used in the corresponding video."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55746954",
   "metadata": {},
   "source": [
    "## Creating a Series\n",
    "\n",
    "Let's start by creating our first pandas Series! (Remember: in the context of a **DataFrame** (table), each column is represented by a **Series**)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b01d9b5",
   "metadata": {},
   "source": [
    "First, let's create some example data to illustrate the creation of our Series. in this example, we will use data that relates a person's \"SpO2\" reading from a pulse oximeter (which shows the percentage of oxygen in someone's blood), and their risk of adverse outcomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4312e1c5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spo2 = [88, 99, 88, 85, 100, 98] # Review: this is a Python list, containing 6 values.\n",
    "risk = ['high', 'low', 'high', 'high', 'low', 'low']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c75cc46f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create our first Series.\n",
    "spo2_series = pd.Series(spo2)\n",
    "spo2_series # If a variable name is the last line in a code cell, Jupyter will display the contents of the variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c3bf8f7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Do the same for risk.\n",
    "pd.Series(risk) # Display the 'risk' Series"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f7cd66-cc93-4b17-978b-1c4d1403f5d3",
   "metadata": {},
   "source": [
    "### <img src='images/alert_icon.svg' alt=\"Alert icon\" width=40 align=center> Alert!\n",
    "> An important point to remember is that Python is \"zero-indexed.\" That means that the first item in a list, or the first row in a table, has the index of 0.\n",
    ">\n",
    "> Notice the output above. The first column shows the Series index, and the second column has the value. The indices start at 0!\n",
    ">\n",
    "> This applies to lists, strings, arrays, and almost everything in Python. Keep this in mind--a common error is mistakenly using index 1, thinking you are getting the first item, where you need to use index 0.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82ae812e",
   "metadata": {
    "id": "82ae812e"
   },
   "source": [
    "## Creating a pandas Dataframe\n",
    "\n",
    "* Now that we've seen a Series, we can see how these are combined as **columns** in a **DataFrame**.\n",
    "* By popular convention, we will name our DataFrame variable `df`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f9f24a",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "> Create a small DataFrame from our example data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "094f5a0a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'spo2': spo2, 'risk': risk}) # Create the DataFrame\n",
    "df # View the DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b3a34a",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "> Create a larger DataFrame from random data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f9d2f08",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "1f9d2f08",
    "outputId": "19429c60-6869-44c7-b761-f2fecd589310",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Make some random numbers for data. \n",
    "# This makes an array of random integers from 1-10, then reshapes it into a\n",
    "# 500x4 matrix (500 rows and 4 columns).\n",
    "random_data = np.random.randint(1,10, size=2000).reshape(500, 4)\n",
    "\n",
    "# Create some example column names\n",
    "column_names = ['columnA', 'columnB', 'columnC', 'columnD']\n",
    "\n",
    "# Create the DateFrame from the random data we generated.\n",
    "# Since the input matrix has 500 rows and 4 columns, the DataFrame will have\n",
    "# 500 rows and 4 columns.\n",
    "df = pd.DataFrame(data=random_data, columns=column_names)\n",
    "df # View the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e00c7d4",
   "metadata": {},
   "source": [
    "### <img src='images/note_icon.svg' alt=\"Note icon\" width=40 align=center> Note\n",
    "> When viewing a large DataFrame in a Jupyter notebook, sometimes the data will be **truncated** as above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "219cee8e-34f2-4e74-a31c-22f0a0a45562",
   "metadata": {},
   "source": [
    "We often want to see the first (or last) several rows of a DateFrame. We can use the `head()` and `tail()` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dbb7076-23a7-4d0d-a90b-addbccb55ded",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View the 1st 5 rows.\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbc449b0-39d4-4b9d-aaa0-104db61f4389",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View the last 3 rows (head also takes a number of rows as an optional\n",
    "# argument).\n",
    "df.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f9e38f9-9ef6-4d08-b24a-0218eb9845c9",
   "metadata": {},
   "source": [
    "We can also limit our views to particular columns using the column names. If the column names have no spaces or special characters, we can use the \"dot\" notation: `df.columnA`. Otherwise, we must pass a list of columns: `df[['columnA', 'columnB']]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2edec7c9-3024-4590-bd51-302adc491479",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View multiple columns or if there are spaces in names\n",
    "# Note the double square brackets!\n",
    "df[['columnA', 'columnB']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cf367e3",
   "metadata": {
    "id": "6cf367e3"
   },
   "source": [
    "# Creating a DataFrame from an existing data file\n",
    "\n",
    "* Instead of creating a DataFrame from scratch (tedious!), we usually create a DataFrame by loading an existing data file.\n",
    "* The most common file format for storing datasets is the **comma-separated value** `(.csv)` file.\n",
    "\n",
    "> To create a pandas DataFrame from a `.csv` file, we'll use the `read_csv` function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51a7d5d0",
   "metadata": {},
   "source": [
    "### Let's load some data\n",
    "\n",
    "The main input argument to the `read_csv` function can be a local filename or a remote file URL. In this example, we'll be using an online file.\n",
    "\n",
    "The dataset used is from the [GapMinder Project](https://www.gapminder.org/) and includes information on GDP, life expectancy, and population in various countries for different years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18a17ef6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "url='https://raw.githubusercontent.com/PracticumAI/python-for-ai/main/data/gapminder_all.csv' # Direct link to the dataset on GitHub\n",
    "df = pd.read_csv(url) # Create the DataFrame from the given URL\n",
    "df # View the DateFrame we just created"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e6be464",
   "metadata": {},
   "source": [
    "We can see that this dataset contains 142 rows and 38 columns corresponding to each country's GDP, life expectancy, and population between 1952 and 2007.\n",
    "\n",
    "### <img src='images/note_icon.svg' alt=\"Note icon\" width=40 align=center> Note\n",
    "> Stop for a second here. What happened to the variable `df` we defined for our SpO2 data?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47YUaLX0zc_8",
   "metadata": {
    "id": "47YUaLX0zc_8"
   },
   "source": [
    "## Viewing parts of a Dataframe\n",
    "* Dataframes often hold many rows (in many AI projects, you might be dealing with hundreds of thousands, or even millions, of samples!)\n",
    "* Sometimes, we want a preview or a sense of the data stored in our DataFrame.\n",
    "* In the below examples, we'll see a few ways to view parts of a DataFrame.\n",
    "    * This includes viewing a subset of columns, a subset of rows, or both.\n",
    "\n",
    "With so many columns, it can be a bit hard to see all of them, but we can get a list of the columns in the DataFrame with the `.columns` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee31a55d-398b-4f8c-857c-bc5f750917d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b380d7",
   "metadata": {},
   "source": [
    "Next, let's revisit viewing some of a DataFrame's **rows**. Since pandas is so flexible, there are several ways to accomplish this. The examples below use the pandas' functions `head` and `tail`.\n",
    "\n",
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 1\n",
    "\n",
    "> Use `head` to view the first 5 rows of our `df` DataFrame. (**Hint:** Given a DataFrame `X`, calling `X.head()` will show the first `5` rows of the DataFrame.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95f5d8dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View only the first 5 rows of the DateFrame.\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bd524d6",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 2\n",
    "> Use `head` to view the first **10** rows of `df`. (**Hint:** You can also provide an input argument `n` to the `head` function to show the first `n` rows (instead of the default `5` rows))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0be82b9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View only the first 10 rows of the DataFrame.\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44f9d9ca",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 3\n",
    "> Use `tail` to view the **last** 5 rows of our `df` DataFrame. (**Hint:** Given a DateFrame `X`, calling `X.tail()` will show the **last** `5` rows of the DateFrame).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4535f19a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View the last 5 rows of the DataFrame.\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4349d214",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 4\n",
    "> Use `tail` to view the **last** `10` rows of `df`. (**Hint:** You can also provide an input argument `n` to the `tail` function to show the last `n` rows (instead of the default `5` rows))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50f56b56",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View only the last 10 rows of the DateFrame.\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2244b476",
   "metadata": {},
   "source": [
    "Now, let's see some ways to view a subset of a DateFrame's **columns**. \n",
    "\n",
    "**Reminder:** If the last line inside a code cell is simply a variable name, Jupyter will display the value of that variable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9e7fe38",
   "metadata": {},
   "source": [
    "For a given DateFrame `df` and desired column `col`, one way to view a single column is to write `df['col']`. (**Important:** the column name `col` must be in quotes!).\n",
    "\n",
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 5\n",
    "> Use the above method to view the `gdpPercap_2007` column of our `df` DateFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfad1032",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# View the gdpPercap_2007 column of the DateFrame.\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68ec6a34",
   "metadata": {},
   "source": [
    "We can also adapt this method to view multiple columns. Instead of passing in a single string column name `'C'`, we will pass in a **list** of string column names, e.g. `['A', 'B', 'C']`. For example: `df[['A', 'B', 'C']]` \n",
    "\n",
    "### <img src='images/note_icon.svg' alt=\"Note icon\" width=40 align=center> Note\n",
    "> **Important:** Notice the double brackets! DataFrames use a similar syntax as lists. To get a single column, we can use the column's name: e.g., `df['A']` would give us column A. Here, we want multiple columns, so a list of column names is provided.\n",
    "\n",
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 6\n",
    "> Use the above method to view the `gdpPercap_2007`, `lifeExp_2007`, and `pop_2007` columns of our `df` DateFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f15bb025",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Display the columns `gdpPercap_2007`, `lifeExp_2007`, and `pop_2007`\n",
    "# Your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afc5dec7",
   "metadata": {
    "id": "afc5dec7"
   },
   "source": [
    "## Creating new columns \n",
    "\n",
    "* We can easily create new columns in an existing DateFrame.\n",
    "* When we use mathematical operations as below, they will be applied to every element of the Series.\n",
    "* This way, we can create new columns from different combinations of existing ones.\n",
    "\n",
    "Let's use an equation to calculate the change in GDP from 1962 to 2007."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "107af062",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "107af062",
    "outputId": "de3188a0-fc49-4c70-e8e0-1b6a9b193380",
    "tags": []
   },
   "outputs": [],
   "source": [
    "df['gdp_change'] = (df['gdpPercap_1962'] - df['gdpPercap_2007'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ebe98a4",
   "metadata": {
    "id": "3ebe98a4"
   },
   "source": [
    "## Deleting columns \n",
    "\n",
    "We can also **drop** (delete) columns.\n",
    "\n",
    "Let's remove the `gdp_change` column that we previously created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48b57eaa",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "48b57eaa",
    "outputId": "e4ff3c3c-e3a6-4499-c406-ad3e4afad251",
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.drop(columns=['gdp_change']) # Will this remove the column?\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af31be9",
   "metadata": {
    "id": "6af31be9"
   },
   "source": [
    "**What happened? `gdp_change` is still there!**\n",
    "\n",
    "### <img src='images/tip_icon.svg' alt=\"Tip icon\" width=40 align=center> Tip\n",
    "> Unless you specify `inplace=True` or assign the DateFrame back to itself (or a different variable name), **the `drop` method does not change the original DateFrame!**\n",
    "\n",
    "Below is one way to permanently remove a column from a DateFrame by assigning the result to a new DateFrame variable `df2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "840718b2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# After dropping the column, we assign the resulting DateFrame to a new variable.\n",
    "# This will persist the change.\n",
    "df2 = df.drop(columns=['gdp_change']) \n",
    "df2.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07188598",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 7\n",
    "\n",
    "> Create a new DateFrame called `df3`, that contains everything in the `df` DateFrame **except** the `continent` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79646cdc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Code it!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a113e56",
   "metadata": {},
   "source": [
    "## Descriptive Statistics\n",
    "\n",
    "### <img src='images/alert_icon.svg' alt=\"Alert icon\" width=40 align=center> Alert!\n",
    "> Before diving into an AI project, becoming familiar with the data you will be modeling is critical!  pandas has an extensive collection of optimized methods designed to help you analyze your data.\n",
    "\n",
    "A preliminary data discovery and exploration phase can help you better understand the data you will be modeling and has many downstream advantages:\n",
    "* You may identify potential problems with the dataset (e.g., missing values, outliers, incomplete columns, irrelevant information).\n",
    "* Results of data exploration may help you decide which preprocessing or modeling approach to take.\n",
    "* A better understanding of your dataset will help you interpret future results.\n",
    "\n",
    "#### Remember Python functions?\n",
    "pandas offers many **functions** that take as **input** a set of values (example: all of the values from the `gdpPercap_2007` column) and **return** a calculated **output** value based on which function was used.\n",
    "\n",
    "Here are a few examples (most are self-explanatory!):\n",
    "* `sum` (the sum of the values)\n",
    "* `count` (the number of values), `value_counts` (the number unique values)\n",
    "* `min` (the minimum value), `max` (the maximum value)\n",
    "* `mean` (the mean value), `median` (the median value), `mode` (the mode of the values)\n",
    "* `var` (the variance of the values), `std` (the standard deviation of the values)\n",
    "* `quantile` (the quantiles of the values)\n",
    "\n",
    "Let's take a look at a few examples using our dataset!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "085acc35",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "Here's how we can compute the **mean** GDP in 2007:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e45b5034",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Compute the mean GDP in 2007\n",
    "df['gdpPercap_2007'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e515f2c7",
   "metadata": {},
   "source": [
    "It's that easy! Let's compute a few more descriptive statistics:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb2ae2e7",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 8\n",
    "> Compute the **minimum** GDP in 2007."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "734a99a5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Compute the minimum GDP in 2007\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b75d65",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 9\n",
    "> Compute the **maximum** GDP in 2007."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "850dbf88",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Compute the maximum GDP in 2007\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8534aca",
   "metadata": {},
   "source": [
    "Finally, the `describe` function can be used on an entire DateFrame to display several standard statistics computed for each column. (**Note:** Be aware of each column's values and data types, as sometimes the statistics will not make sense.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2383c2f6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# (Almost) everything, (almost) everywhere, all at once\n",
    "# Note: will only compute statistics for columns with numerical data.\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8e532f0",
   "metadata": {},
   "source": [
    "## Filtering data\n",
    "pandas makes it easy to select, analyze, and process subsets of your data based on some criteria, `C`. This is accomplished with the syntax: `df[C]`, which will return a subset of DateFrame `df` that **matches** the criteria `C` listed inside brackets `[]`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d90eb91",
   "metadata": {},
   "source": [
    "### Comparing values\n",
    "In pandas, we can create filtering criteria using familiar mathematical operators to compare values:\n",
    "* `>`: greater than\n",
    "* `<`: less than\n",
    "* `>=`: greater than or equal to\n",
    "* `<=`: less than or equal to\n",
    "* `!=`: is not equal to\n",
    "* `==`: is equal to a single value\n",
    "* **Bonus:** `.isin(L)`: equals any value in the list `L`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17853995",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "Display 2007 GDP is **greater** than $40,000.\n",
    "* In this example, the criterion `C` is `df['gdpPercap_2007'] > 40000`.\n",
    "* Recall the pandas filtering syntax: `df[C]`.\n",
    "* So, this is what we want: `df[df['gdpPercap_2007'] > 40000]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "effbed44",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Select rows where 2007 GDP is greater than $40,000\n",
    "df[df['gdpPercap_2007'] > 40000]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dcdd276",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 10\n",
    "> Display where 2007 GDP is **less** than $500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45465e39",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Display rows where 2007 GDP is less than $500"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69539eb3",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "Display the first 5 rows for African countries.\n",
    "\n",
    "* Recall that the symbol `==` will check for equality (we want the continent to be equal to the single value `Africa`).\n",
    "* In this example, the criterion `C` is `df['continent] == 'Africa'`.\n",
    "* Given the pandas filtering syntax: `df[C]`, this is what we want: `df[df['continent'] == 'Africa']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a17076c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Display the first 5 rows for African countries.\n",
    "df[df['continent'] == 'Africa'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cceae2a5",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 11\n",
    "> Display the first 5 rows for European countries (`'continent'`==`'Europe'`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d71ee165",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Display the first 5 rows for European countries ('continent'=='Europe').\n",
    "# Write your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7bc6f15",
   "metadata": {},
   "source": [
    "### Combining multiple criteria\n",
    "\n",
    "Multiple criteria can be combined using either the `&` or `|` symbols:\n",
    "* C1 `&` C2: This will select rows that match **both** criterion C1 **and** criterion C2.\n",
    "* C1 `|` C2: This will select rows that match **either** criterion C1 **or** criterion C2.\n",
    "\n",
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "Display rows where 2007 GDP is greater than or equal to 3,000 and 2007 life expectancy is under 60. In this example, we have two criteria:\n",
    "* C1: `df['gdpPercap_2007'] >= 3000`\n",
    "* C2: `df['lifeExp_2007'] < 60`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9d2d237",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Select rows where 2007 GDP is greater than or equal to 30,000 and 2007 life expectancy is under 60.\n",
    "df[(df['gdpPercap_2007'] >= 3000) & (df['lifeExp_2007'] < 60)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54e734e9",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 12\n",
    "> Display all European countries that have 2007 life expectancies over 80."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aedbbfd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Display all European countries that have 2007 life expectancies over 80.\n",
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e7a5eb1",
   "metadata": {},
   "source": [
    "### Analyzing filtered data\n",
    "We can also perform our descriptive statistics on a filtered subset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb9a1376",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df[(df['continent'] == 'Europe') & (df['lifeExp_2007'] > 80)].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d9c19e1",
   "metadata": {},
   "source": [
    "## Data visualization\n",
    "\n",
    "Let's conclude this lesson by learning how pandas can enhance our understanding of a dataset with its many built-in visualization tools.\n",
    "\n",
    "Let's see the data! This time, we'll load our dataset into a DateFrame variable named `X`. \n",
    "\n",
    "> ![A hand icon](images/hand.png) Wait! `X`?? Didn't we say to avoid single-letter variable names and generally use lowercase?? Well...rules are...there to be broken when there is a good reason. In much of machine learning and AI, we work with datasets where we want to make a model to generate a prediction. Part of the goal is to take input data and predict an output or outcome. Thinking of a graph with `x`- and `y`-values, we can think of the `x`-value as an input and the `y`-value as the outcome. So, `x` and `y` make sense as variable names when thinking about values on a graph and datasets more generally. Going further, we borrow a convention from math, where a matrix (or a table with several columns) is referenced with a capital letter. Since our DataFrame has multiple columns, we will use the capital X for the name. \n",
    ">\n",
    "> We also want to ensure you understand that while we frequently use `df` for a DataFrame variable name, `df` is just a name, and nothing is special about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "088473f7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = pd.read_csv(url) # url is defined above\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35b27a8",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "Let's visually examine the distribution of 2007 GDPs with the pandas `hist` (histogram) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf817b3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hist = X['gdpPercap_2007'].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28f61863",
   "metadata": {},
   "source": [
    "We can modify many parameters of the `hist` function to create a more informative figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c67d123b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hist = X['gdpPercap_2007'].hist(bins=72, grid=False)\n",
    "hist.set_title('Distribution of 2007 GDP of Countries')\n",
    "hist.set_ylabel('Count')\n",
    "hist.set_xlabel('GDP in 2007')\n",
    "display(hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44145fe4",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 13\n",
    "> Visualize the distribution of the 2007 life expectancies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ac17451",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Visualize the distribution of the 2007 life expectancies.\n",
    "# Your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ff5eb1-85b5-4802-adaa-3e4e4065764f",
   "metadata": {},
   "source": [
    "This can also be written as: `X.hist(['lifeExp_2007'], bins=40)`\n",
    "\n",
    "That syntax makes it easier to plot multiple histograms side-by-side. Below, we compare the distributions of life expectancies in 1962 and 2007."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e81d1106-801f-4318-b7a6-7216dc416a68",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X.hist(column=['lifeExp_1962','lifeExp_2007'], bins=40, sharey=True, sharex=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a622496",
   "metadata": {},
   "source": [
    "### <img src='images/example_icon.svg' alt=\"Example icon\" width=40 align=center> Example\n",
    "We can use the `df.plot.box` function to create box plots from columns in our DateFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2524525c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X.plot.box(column='lifeExp_2007', by='continent')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa1e495",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 14\n",
    "> Practice with the `hist` and `box` functions by exploring different columns in our DateFrame. Experiment with data filtering methods to refine your figures further. **Hint:** We can see all available columns with `print(X.columns)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "705b9371",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Experiment with code!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66e920e6-0941-4f38-8b38-63341fdff3a4",
   "metadata": {},
   "source": [
    "----\n",
    "## Push changes to GitHub <img src=\"images/push_to_github.png\" alt=\"Push to GitHub icon\" align=\"right\" width=150>\n",
    "\n",
    " Remember to **add**, **commit**, and **push** the changes you have made to this notebook to GitHub to keep your repository in sync.\n",
    "\n",
    "In Jupyter, those are done in the git tab on the left. In Google Colab, use File > Save a copy in GitHub.\n",
    "\n",
    "\n",
    "***\n",
    "#### Attribution \n",
    "Some content in this learning experience was adapted from Ben Shickel's [AI for Medicine bootcamp](https://github.com/gatorai/scripps) content.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d204c609-a815-4768-888e-6a1e6f55b0c7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "UFRC Python-3.10",
   "language": "python",
   "name": "python3-3.10-ufrc"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}