Day_9_Pandas_Part_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Day_9_Pandas_2.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P2-S_RPdx5Nz"
      },
      "source": [
        "## **Dealing with NaN values:**\n",
        "As mentioned earlier, before we can begin training our learning algorithms with large datasets, we usually need to clean the data first. This means we need to have a method for detecting and correcting errors in our data. \n",
        "\n",
        "While any given dataset can have many types of bad data, such as outliers or incorrect values, the type of bad data we encounter almost always is missing values. As we saw earlier, Pandas assigns ```NaN``` values to missing data. In this lesson we will learn how to detect and deal with ```NaN``` values.\n",
        "\n",
        "We will begin by creating a DataFrame with some ```NaN``` values in it.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cdbo5DqCv_fz",
        "outputId": "aa84e080-0d13-47b3-a9f0-8ccde009a7f2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# We create a list of Python dictionaries\n",
        "items2 = [{'bikes': 20, 'pants': 30, 'watches': 35, 'shirts': 15, 'shoes':8, 'suits':45},\n",
        "{'watches': 10, 'glasses': 50, 'bikes': 15, 'pants':5, 'shirts': 2, 'shoes':5, 'suits':7},\n",
        "{'bikes': 20, 'pants': 30, 'watches': 35, 'glasses': 4, 'shoes':10}]\n",
        "\n",
        "# We create a DataFrame  and provide the row index\n",
        "store_items = pd.DataFrame(items2, index = ['store 1', 'store 2', 'store 3'])\n",
        "\n",
        "# We display the DataFrame\n",
        "store_items"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8</td>\n",
              "      <td>45.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1     20     30       35    15.0      8   45.0      NaN\n",
              "store 2     15      5       10     2.0      5    7.0     50.0\n",
              "store 3     20     30       35     NaN     10    NaN      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WwTe9sY01ftK"
      },
      "source": [
        "In cases where we load very large datasets into a DataFrame, possibly with millions of items, the number of NaN values is not easily visualized. For these cases, we can use a combination of methods to count the number of ```NaN``` values in our data. The following example combines the ```.isnull()``` and the ```sum()``` methods to count the number of ```NaN``` values in our DataFrame"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7-oMtUkO174S",
        "outputId": "7109e9f6-69a8-48df-d598-9b493261a9f7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "store_items.isnull()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1  False  False    False   False  False  False     True\n",
              "store 2  False  False    False   False  False  False    False\n",
              "store 3  False  False    False    True  False   True    False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wfprzvTu2ETi"
      },
      "source": [
        "> In Pandas, logical True values have numerical value 1 and logical False values have numerical value 0. Therefore, we can count the number of NaN values by counting the number of logical True values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W_AV_rKL0_kM",
        "outputId": "30028950-2b6e-4aeb-8f41-d61b10536b60",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "# We count the number of NaN values in the columns of store_items\n",
        "x =  store_items.isnull().sum()\n",
        "\n",
        "# We print x\n",
        "print('Number of NaN values in our DataFrame:', x)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of NaN values in our DataFrame: bikes      0\n",
            "pants      0\n",
            "watches    0\n",
            "shirts     1\n",
            "shoes      0\n",
            "suits      1\n",
            "glasses    1\n",
            "dtype: int64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDRunbOt2N4c"
      },
      "source": [
        "> In order to count the total number of logical True values we use the .sum() method twice. We have to use it twice because the first sum returns a Pandas Series with the sums of logical True values along columns, as we see below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r5DUDDh-1qdC",
        "outputId": "490e8ef5-93d1-48d7-de03-aff0f2c685bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# We count the number of NaN values in store_items\n",
        "x =  store_items.isnull().sum().sum()\n",
        "\n",
        "# We print x\n",
        "print('Number of NaN values in our DataFrame:', x)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of NaN values in our DataFrame: 3\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1GoFgxMx2iBw"
      },
      "source": [
        "> The second sum will then add up the 1s in the above Pandas Series.\n",
        "\n",
        "Instead of counting the number of NaN values we can also do the opposite, we can count the number of non-NaN values. We can do this by using the .count() method as shown below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0GnX8msl14sQ",
        "outputId": "0fc11eab-19d2-4ba8-92ff-7db85dc71d20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        }
      },
      "source": [
        "# We print the number of non-NaN values in our DataFrame\n",
        "print()\n",
        "print('Number of non-NaN values in the columns of our DataFrame:\\n', \n",
        "        store_items.count())"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Number of non-NaN values in the columns of our DataFrame:\n",
            " bikes      3\n",
            "pants      3\n",
            "watches    3\n",
            "shirts     2\n",
            "shoes      3\n",
            "suits      2\n",
            "glasses    2\n",
            "dtype: int64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TW02Ukj-3ggb"
      },
      "source": [
        "Now that we learned how to know if our dataset has any NaN values in it, the next step is to decide what to do with them. In general we have two options, we can either delete or replace the NaN values. In the following examples we will show you how to do both.\n",
        "\n",
        "We will start by learning how to eliminate rows or columns from our DataFrame that contain any NaN values. The .dropna(axis) method eliminates any rows with NaN values when axis = 0 is used and will eliminate any columns with NaN values when axis = 1 is used. Let's see some examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oIGm4Jft23sA",
        "outputId": "cf463378-0773-4dd5-8e4b-02450250d167",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 100
        }
      },
      "source": [
        "# We drop any rows with NaN values\n",
        "store_items.dropna(axis = 0)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 2     15      5       10     2.0      5    7.0     50.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5A83CBac-Kan",
        "outputId": "674a6735-6563-4267-ee6d-f045d7fb59ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We drop any columns with NaN values\n",
        "store_items.dropna(axis = 1)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shoes</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shoes\n",
              "store 1     20     30       35      8\n",
              "store 2     15      5       10      5\n",
              "store 3     20     30       35     10"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hu6m-KLW-O5u"
      },
      "source": [
        "Notice that the .dropna() method eliminates (drops) the rows or columns with NaN values out of place. This means that the original DataFrame is not modified. You can always remove the desired rows or columns in place by setting the keyword inplace = True inside the dropna() function.\n",
        "\n",
        "Now, instead of eliminating NaN values, we can replace them with suitable values. We could choose for example to replace all NaN values with the value 0. We can do this by using the .fillna() method as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eCVbempI-OpA",
        "outputId": "a5062f3d-2cbc-41db-df67-38f67227f44e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace all NaN values with 0\n",
        "store_items.fillna(0)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8</td>\n",
              "      <td>45.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>0.0</td>\n",
              "      <td>10</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1     20     30       35    15.0      8   45.0      0.0\n",
              "store 2     15      5       10     2.0      5    7.0     50.0\n",
              "store 3     20     30       35     0.0     10    0.0      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGRzC9cN-ZM8"
      },
      "source": [
        "We can also use the .fillna() method to replace NaN values with previous values in the DataFrame, this is known as forward filling. When replacing NaN values with forward filling, we can use previous values taken from columns or rows. The .fillna(method = 'ffill', axis) will use the forward filling (ffill) method to replace NaN values using the previous known value along the given axis. Let's see some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q6d-csaI-MNs",
        "outputId": "36d474ab-4f9c-4844-c028-58964d8c161b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace NaN values with the previous value in the column\n",
        "store_items.fillna(method = 'ffill', axis = 0)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8</td>\n",
              "      <td>45.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>2.0</td>\n",
              "      <td>10</td>\n",
              "      <td>7.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1     20     30       35    15.0      8   45.0      NaN\n",
              "store 2     15      5       10     2.0      5    7.0     50.0\n",
              "store 3     20     30       35     2.0     10    7.0      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ToUvvjSC-i2N"
      },
      "source": [
        "Notice that the NaN value in store 1 didn't get replaced. That's because there are no previous values in this column, since the NaN value is the first value in that column. However, if we do forward fill using the previous row values, this won't happen. Let's take a look:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2AkJfUbF-b8K",
        "outputId": "60ede1a5-70e6-4ea9-f4aa-d1a49b291a4e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace NaN values with the previous value in the row\n",
        "store_items.fillna(method = 'ffill', axis = 1)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>45.0</td>\n",
              "      <td>45.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1   20.0   30.0     35.0    15.0    8.0   45.0     45.0\n",
              "store 2   15.0    5.0     10.0     2.0    5.0    7.0     50.0\n",
              "store 3   20.0   30.0     35.0    35.0   10.0   10.0      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ji9ncbCm_FPg"
      },
      "source": [
        "We see that in this case all the NaN values have been replaced with the previous row values.\n",
        "\n",
        "Similarly, you can choose to replace the NaN values with the values that go after them in the DataFrame, this is known as backward filling. The .fillna(method = 'backfill', axis) will use the backward filling (backfill) method to replace NaN values using the next known value along the given axis. Just like with forward filling we can choose to use row or column values. Let's see some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W-wDgBVZ-7yJ",
        "outputId": "dbd2e194-533a-4c3c-d3ef-918b062d2123",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace NaN values with the next value in the column\n",
        "store_items.fillna(method = 'backfill', axis = 0)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8</td>\n",
              "      <td>45.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1     20     30       35    15.0      8   45.0     50.0\n",
              "store 2     15      5       10     2.0      5    7.0     50.0\n",
              "store 3     20     30       35     NaN     10    NaN      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "00-GZ-Jg_Hr4",
        "outputId": "4ef11e3b-5218-4116-a324-bbee50a7aeaa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace NaN values with the next value in the row\n",
        "store_items.fillna(method = 'backfill', axis = 1)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>45.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1   20.0   30.0     35.0    15.0    8.0   45.0      NaN\n",
              "store 2   15.0    5.0     10.0     2.0    5.0    7.0     50.0\n",
              "store 3   20.0   30.0     35.0    10.0   10.0    4.0      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ci1HJ1uz_cGF"
      },
      "source": [
        "Notice that the .fillna() method replaces (fills) the NaN values out of place. This means that the original DataFrame is not modified. You can always replace the NaN values in place by setting the keyword inplace = True inside the fillna() function.\n",
        "\n",
        "We can also choose to replace NaN values by using different interpolation methods. For example, the .interpolate(method = 'linear', axis) method will use linear interpolation to replace NaN values using the values along the given axis. Let's see some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "irAhB45W_XgS",
        "outputId": "ada45a20-541c-4f56-a194-3322db6a2885",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace NaN values by using linear interpolation using column values\n",
        "store_items.interpolate(method = 'linear', axis = 0)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8</td>\n",
              "      <td>45.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20</td>\n",
              "      <td>30</td>\n",
              "      <td>35</td>\n",
              "      <td>2.0</td>\n",
              "      <td>10</td>\n",
              "      <td>7.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1     20     30       35    15.0      8   45.0      NaN\n",
              "store 2     15      5       10     2.0      5    7.0     50.0\n",
              "store 3     20     30       35     2.0     10    7.0      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lt5jypg8_uBu"
      },
      "source": [
        "Notice that the two NaN values in store 3 have been replaced with linear interpolated values. However, notice that the NaN value in store 1 didn't get replaced. That's because the NaN value is the first value in that column, and since there is no data before it, the interpolation function can't calculate a value. Now, let's interpolate using row values instead:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QGnRalxo_kSA",
        "outputId": "944059c4-b41d-4453-de22-cda0c8d17e7f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "# We replace NaN values by using linear interpolation using row values\n",
        "store_items.interpolate(method = 'linear', axis = 1)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bikes</th>\n",
              "      <th>pants</th>\n",
              "      <th>watches</th>\n",
              "      <th>shirts</th>\n",
              "      <th>shoes</th>\n",
              "      <th>suits</th>\n",
              "      <th>glasses</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>store 1</th>\n",
              "      <td>20.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>45.0</td>\n",
              "      <td>45.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 2</th>\n",
              "      <td>15.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>50.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>store 3</th>\n",
              "      <td>20.0</td>\n",
              "      <td>30.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>22.5</td>\n",
              "      <td>10.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         bikes  pants  watches  shirts  shoes  suits  glasses\n",
              "store 1   20.0   30.0     35.0    15.0    8.0   45.0     45.0\n",
              "store 2   15.0    5.0     10.0     2.0    5.0    7.0     50.0\n",
              "store 3   20.0   30.0     35.0    22.5   10.0    7.0      4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YcXh1Wzu_zBa"
      },
      "source": [
        "> Just as with the other methods we saw, the .interpolate() method replaces NaN values out of place."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBo9YDea_57T"
      },
      "source": [
        "## **Question:**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xDLlfzrU_wAp"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "# Since we will be working with ratings, we will set the precision of our \n",
        "# dataframes to one decimal place.\n",
        "pd.set_option('precision', 1)\n",
        "\n",
        "# Create a Pandas DataFrame that contains the ratings some users have given to a\n",
        "# series of books. The ratings given are in the range from 1 to 5, with 5 being\n",
        "# the best score. The names of the books, the authors, and the ratings of each user\n",
        "# are given below:\n",
        "\n",
        "books = pd.Series(data = ['Great Expectations', 'Of Mice and Men', 'Romeo and Juliet', 'The Time Machine', 'Alice in Wonderland' ])\n",
        "authors = pd.Series(data = ['Charles Dickens', 'John Steinbeck', 'William Shakespeare', ' H. G. Wells', 'Lewis Carroll' ])\n",
        "\n",
        "user_1 = pd.Series(data = [3.2, np.nan ,2.5])\n",
        "user_2 = pd.Series(data = [5., 1.3, 4.0, 3.8])\n",
        "user_3 = pd.Series(data = [2.0, 2.3, np.nan, 4])\n",
        "user_4 = pd.Series(data = [4, 3.5, 4, 5, 4.2])\n",
        "\n",
        "# Users that have np.nan values means that the user has not yet rated that book.\n",
        "# Use the data above to create a Pandas DataFrame that has the following column\n",
        "# labels: 'Author', 'Book Title', 'User 1', 'User 2', 'User 3', 'User 4'. Let Pandas\n",
        "# automatically assign numerical row indices to the DataFrame. \n",
        "\n",
        "# Create a dictionary with the data given above\n",
        "dat = \n",
        "\n",
        "# Use the dictionary to create a Pandas DataFrame\n",
        "book_ratings = \n",
        "\n",
        "# If you created the dictionary correctly you should have a Pandas DataFrame\n",
        "# that has column labels: 'Author', 'Book Title', 'User 1', 'User 2', 'User 3',\n",
        "# 'User 4' and row indices 0 through 4.\n",
        "\n",
        "# Now replace all the NaN values in your DataFrame with the average rating in\n",
        "# each column. Replace the NaN values in place. HINT: you can use the fillna()\n",
        "# function with the keyword inplace = True, to do this. Write your code below:\n"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}