Lab 9 - Continuous Prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Prediction\n",
    "In this lab, we will learn how to use linear prediction methods\n",
    "to estimate continuous outcomes, and to evaluate the \n",
    "quality of competing models. We will explore both \n",
    "the \n",
    "\n",
    "# Predicting Loan Quality\n",
    "\n",
    "One of the most imporant aspects of lending is determining the\n",
    "interest rate to give a customer. Set rates too high, and the\n",
    "customer may choose another lender. Set rates too low, and \n",
    "lender may not earn enough interest to offset defaults and other expenses.\n",
    "\n",
    "The data for this exercise comes from Lending Club, a peer-to-peer lending company.\n",
    "They facilitate loans and allow individuals to make loans or borrow money (you \n",
    "can read more about them on \n",
    "[Wikipedia](https://en.wikipedia.org/wiki/Lending_Club).\n",
    "\n",
    "We can get historical data from the\n",
    "[Lending Club data page](https://www.lendingclub.com/info/download-data.action).\n",
    "\n",
    "Download the loan data that is on Blackboard. This is not the newest data, \n",
    "but it has the outcomes of many loans that have reached maturity. We can use the first dataset\n",
    "to train and the second to test. You should also download the data dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import statsmodels.api as sm\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "First, let's view some of the columns in dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ld = pd.read_csv('data/lendingclub_2015-2018.csv')\n",
    "ld.head()\n",
    "tmp = ld.tail()\n",
    "display(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interest Rate\n",
    "Since we are interested in the interest rates that customers receive, let's plot a histogram to see how rates are distributed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ld['int_rate'].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loan Duration\n",
    "The loan duration column is formatted as a text string, and must be cleaned up for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# view unique values\n",
    "ld['term'].unique()\n",
    "\n",
    "# split rows into parts\n",
    "term_split = ld['term'].str.split(' ')\n",
    "\n",
    "# view first five rows\n",
    "print(term_split[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the str function can retrieve a specific list element for all rows\n",
    "term_split.str[1]\n",
    "ld['duration'] = term_split.str[1]\n",
    "\n",
    "# add this to the dataframe\n",
    "display(ld['duration'].head())\n",
    "# this column is not in integer format. Must fix!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert column to integer\n",
    "ld['duration'] = ld['duration'].apply(int)\n",
    "display(ld['duration'].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rescaling\n",
    "Some of the columns that we will use are on very different scales. For example, loan amount and annual income range from 0 to tens of thousands of dollars,\n",
    "whereas the debt-to-income (`dti`) range is much smaller. This can cause issues when fitting the models.\n",
    "\n",
    "We will transform the income and loan amount variables using a log transformation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ld['log_loan_amnt'] = np.log(ld['loan_amnt'])\n",
    "ld['log_annual_inc'] = np.log(ld['annual_inc']+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlations\n",
    "Let's run some correlations to see how some columns relate to one another"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = ['int_rate', 'log_loan_amnt', 'installment', 'log_annual_inc', 'duration', 'fico_range_low', 'revol_util', 'dti']\n",
    "corr = ld[cols].corr()\n",
    "corr.style.background_gradient(cmap='coolwarm')\n",
    "\n",
    "# ld[cols].corr() # <--- use this if you just want the table in non-graphical format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of these values, interest rate has the strongest correlations with duration and FICO score. The correlation between loan amount\n",
    "and installment size is quite high, so we should drop one of these from our subsequent analysis (highly correlated variables can \n",
    "cause issues with linear regression).\n",
    "\n",
    "Create a list of the variables to use for the prediction of interest rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_vars = ['log_loan_amnt', 'log_annual_inc', 'fico_range_low', 'revol_util', 'dti', 'duration']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop rows with missing values\n",
    "\n",
    "There are some rows in this dataframe that are missing values for at least one of our predictor columns.\n",
    "We will drop these from the dataframe before proceeding to avoid downstream errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"before dropping rows with missing data\", len(ld))\n",
    "ld = ld.dropna(subset=pred_vars)\n",
    "print(\"after dropping rows with missing data\", len(ld))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a dataset that is cleaned and ready for analysis.\n",
    "\n",
    "# Training and testing sets\n",
    "With this dataset, the observations are ordered from newest to oldest. We can \n",
    "simulate a real-world situation by splitting our data into train and test subsets\n",
    "by their position in the series. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# use index-based sampling since we have time series data\n",
    "train, test = train_test_split(ld, test_size=0.25, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, view the start and end dates for the two samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# earliest and latest dates in train\n",
    "print(\"training data starts\\n\", train['issue_d'].head())\n",
    "print(\"training data ends\\n\", train['issue_d'].tail())\n",
    "# earliest and latest in test\n",
    "print(\"testing data starts\\n\", test['issue_d'].head())\n",
    "print(\"testing data ends\\n\", test['issue_d'].tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Linear Regression\n",
    "\n",
    "The syntax for creating models using the `statsmodels` package\n",
    "is similar to that of `sklearn` (`sklearn` has linear regression\n",
    "functions, but it is somewhat barebones in it's model summaries\n",
    "compared to `statsmodels`). The documentation for \n",
    "ordinary least squares (OLS) regression using\n",
    "`statsmodels` is \n",
    "[here](https://www.statsmodels.org/devel/generated/statsmodels.regression.linear_model.OLS.html#statsmodels.regression.linear_model.OLS).\n",
    "\n",
    "Predict the interest rate as a function of credit score\n",
    "`fico_range_low`. This variable has the strongest correlation\n",
    "with interest rate from those in our correlation table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg_fico = sm.OLS(train['int_rate'], train['fico_range_low']).fit()\n",
    "reg_fico.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, add additional predictors from our list from earlier. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add additional variables\n",
    "Next, build a regression using the variables in the list of `pred_vars`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg_multi = sm.OLS(train['int_rate'], train[pred_vars], hasconst=False).fit()\n",
    "reg_multi.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This new model performs better on out of sample data.\n",
    "\n",
    "# Evaluation\n",
    "\n",
    "For OLS, the $R^2$ of a model is often the statistic that people look at first. This \n",
    "describes the amount of variance in $Y$ that is explained by the predictors. It will always\n",
    "increase with additional predictor variables, so we will want to look at some additional measures that penalize \n",
    "having too many predictors in the model.\n",
    "\n",
    "One measure is the Adjusted $R^2$, which considers the number of variables in the model. For large samples, this\n",
    "is essentially the same as $R^2$. Two measures that consider both the number of variables *and* the quality\n",
    "of the model are Akaike's Information Criterion (AIC) and Bayesian Information Criterion (BIC). These values\n",
    "do not have much significance on their own, like $R^2$, but they are very good for comparing models. Lower \n",
    "values of AIC or BIC are better.\n",
    "\n",
    "Which model has the lowest AIC?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(reg_fico.aic)\n",
    "print(reg_multi.aic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(reg_fico.bic)\n",
    "print(reg_multi.bic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another comparison method is to use an ANOVA. This compares models for statistically significant\n",
    "improvements. A $p$-value less than $0.05$ is generally considered a significant improvement.\n",
    "This method can determine if the addition of variables improves the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.stats.anova_lm(reg_fico, reg_multi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "To further evaluate the quality of these models, we will look at out-of-sample prediction\n",
    "with the test data. We will use the root mean squared error (RMSE) to evaluate \n",
    "the performance of this model. Lower values indicate smaller prediction errors and\n",
    "a better model. \n",
    "\n",
    "First, get the predictions from the model that only uses FICO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fico_pred = reg_fico.predict(test['fico_range_low'])\n",
    "\n",
    "fico_rmse = metrics.mean_squared_error(test['int_rate'], fico_pred, squared=False)\n",
    "print(\"RMSE:\", fico_rmse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then do the same for the model using multiple predictors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_pred = reg_multi.predict(test[pred_vars])\n",
    "\n",
    "multi_rmse = metrics.mean_squared_error(test['int_rate'], multi_pred, squared=False)\n",
    "print(\"RMSE:\", multi_rmse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second model performs better on out of sample data. \n",
    "\n",
    "# Summary\n",
    "We built regression models in an attempt to predict interest rates for loans from Lending Club\n",
    "using data about the loan request and borrower information. First, we cleaned and transformed the\n",
    "data, then viewed the correlations between a subset of variables. Then, we built models on\n",
    "on a training set of data and compared model fit measures (AIC, BIC, ANOVA). Lastly, we compared\n",
    "the models on a holdout set of data.\n",
    "\n",
    "# Exercises\n",
    "1. Can you build a model that performs significantly better than the models \n",
    "   already built? Train the model and compare it. Which variables did you \n",
    "   use and why do you think they improved the model? Provide the statistics you used \n",
    "   to evaluate. \n",
    "\n",
    "   **Do not use the `installment` column.**\n",
    "   \n",
    "2. What level of RMSE would you consider acceptable would you consider appropriate in this situation? Provide justification for your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "aa3e7337b8d36894b7a51014394139a0eba3728352dce4cced1005d81d5028b1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}