Lab 5 - Algorithms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5 - Classification Algorithms\n",
    "In this lab, you will learn how to use train and evaluate classification algorithms\n",
    "on a dataset. The reason we do this is because it is usually difficult to know \n",
    "which algorithm will perform best for a given dataset. It is important to\n",
    "understand how to compare multiple models in order to select the best model.\n",
    "\n",
    "We will need to import a variety of models from `sklearn`. Descriptions\n",
    "of these models and links to documentation will be provided when \n",
    "we fit each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn import tree\n",
    "from sklearn import ensemble\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn import svm\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data\n",
    "The data comes from [Kaggle](https://www.kaggle.com/mlg-ulb/creditcardfraud).\n",
    "The data dimensions are anonymized to protect the identity of the individuals.\n",
    "The process to create the dimensions is \n",
    "[Principal Components Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ccfraud = pd.read_csv('data/creditcard.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "This dataset is already cleaned and ready for use. Let's view some\n",
    "descripteve statistics to verify this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the display() function works better for displaying dataframes in Jupyter Notebooks than print().\n",
    "display(ccfraud.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train/test split\n",
    "Next, we will use a 75/25 split for this data. We will view\n",
    "some summarization by outcome `Class` and a few variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(516)\n",
    "\n",
    "# create train and test\n",
    "train, test = train_test_split(ccfraud, test_size=0.25)\n",
    "print(\"Rows in train:\", len(train))\n",
    "print(\"Rows in test:\", len(test))\n",
    "\n",
    "# view some stats by different variables\n",
    "train_stats = train.groupby('Class')[['Time', 'Amount', 'V1']].agg(['mean', 'count'])\n",
    "print(\"Training data:\\n\", train_stats)\n",
    "test_stats = test.groupby('Class')[['Time', 'Amount', 'V1']].agg(['mean', 'count'])\n",
    "print(\"Testing data:\\n\", test_stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will select the variables we want to use for prediction. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_vars = ['Time', 'Amount', 'V1', 'V2', 'V3', 'V4'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train models\n",
    "In this section, we will train a several different types of classifiers. They\n",
    "will be presented one-by-one in this lab, but it is possible to set up the \n",
    "training in a `for` loop to make your code simpler and more flexible.\n",
    "\n",
    "## Decision tree\n",
    "The first model we fit is a decision tree, which we have been working with for some \n",
    "time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtree = tree.DecisionTreeClassifier(criterion='entropy', max_depth=10)\n",
    "dtree.fit(train[pred_vars], train['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest\n",
    "A random forest classifier consists of many decision trees\n",
    "with randomized parameters (hence the name). One of the parameters\n",
    "is the number of trees in the classifier. The default is 100, but \n",
    "this can be reduced for large datasets if training is slow.\n",
    "Full documentation on this implementation is \n",
    "[here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = ensemble.RandomForestClassifier()\n",
    "rf.fit(train[pred_vars], train['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "There are many types of neural networks in use today. We will \n",
    "use perhaps the most common type, the Multi-Layer Perceptron (MLP). \n",
    "The documentation for this method is \n",
    "[here](https://scikit-learn.org/stable/modules/neural_networks_supervised.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp = MLPClassifier(hidden_layer_sizes=(20,20))\n",
    "mlp.fit(train[pred_vars], train['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machines\n",
    "Next, we will train a support vector machine. The basic \n",
    "model works well for linearly separable classes. You may also\n",
    "want to try an RBF kernel, which can perform better in some situations.\n",
    "\n",
    "The documentation for SVMs is \n",
    "[here](https://scikit-learn.org/stable/modules/svm.html). By default, \n",
    "the model only produces label outputs. We will adjust the probability\n",
    "parameter so that the model can also report class probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc = svm.SVC(probability=True)\n",
    "svc.fit(train[pred_vars], train['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "The Naive Bayes classification algorithm is based on conditional probabilities\n",
    "from Bayes' Theorem. It is one of the simplest algorithms, yet tends to \n",
    "perform quite well in many scenarios.\n",
    "\n",
    "Documentation for this is \n",
    "[here](https://scikit-learn.org/stable/modules/naive_bayes.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb = GaussianNB()\n",
    "nb.fit(train[pred_vars], train['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "The last method we will use in this lab is the logistic regression.\n",
    "This a method that has existed in statistics for a long time and,\n",
    "like decision trees, results in models that are human-interpretable.\n",
    "\n",
    "The documentation for this method is \n",
    "[here](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression()\n",
    "lr.fit(train[pred_vars], train['Class'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "Now that we have trained these models, the next step \n",
    "is to evaluate their performance on out-of-sample data (our test dataset).\n",
    "We will use multiple evaluation statistics.\n",
    "\n",
    "The code for this section is adapted from \n",
    "[this tutorial](https://abdalimran.github.io/2019-06-01/Drawing-multiple-ROC-Curves-in-a-single-plot),\n",
    "which seems to be an adaptation of \n",
    "[this StackOverflow answer](https://stackoverflow.com/questions/42894871/how-to-plot-multiple-roc-curves-in-one-plot-with-legend-and-auc-scores-in-python).\n",
    "\n",
    "It works by iterating through the models using a for loop,\n",
    "and then storing the statistics in a dataframe. These models \n",
    "perform poorly, and may result in warning message from `sklearn`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of our models\n",
    "fitted = [dtree, rf, mlp, svc, nb, lr]\n",
    "\n",
    "# empty dataframe to store the results\n",
    "result_table = pd.DataFrame(columns=['classifier_name', 'fpr','tpr','auc', \n",
    "                                     'log_loss', 'clf_report'])\n",
    "\n",
    "for clf in fitted:\n",
    "    # print the name of the classifier\n",
    "    print(clf.__class__.__name__)\n",
    "    \n",
    "    # get predictions\n",
    "    yproba = clf.predict_proba(test[pred_vars])\n",
    "    yclass = clf.predict(test[pred_vars])\n",
    "    \n",
    "    # auc information\n",
    "    fpr, tpr, _ = metrics.roc_curve(test['Class'],  yproba[:,1])\n",
    "    auc = metrics.roc_auc_score(test['Class'], yproba[:,1])\n",
    "    \n",
    "    # log loss\n",
    "    log_loss = metrics.log_loss(test['Class'], yproba[:,1])\n",
    "    \n",
    "    # add some other stats based on confusion matrix\n",
    "    clf_report = metrics.classification_report(test['Class'], yclass)\n",
    "    \n",
    "    # add the results to the dataframe\n",
    "    result_table = result_table.append({'classifier_name':clf.__class__.__name__,\n",
    "                                        'fpr':fpr, \n",
    "                                        'tpr':tpr, \n",
    "                                        'auc':auc,\n",
    "                                        'log_loss': log_loss,\n",
    "                                        'clf_report': clf_report}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the results\n",
    "For easy formatting later, reset the dataframe index to be the classifier names\n",
    "rather than a numeric index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_table.set_index('classifier_name', inplace=True)\n",
    "display(result_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The raw dataframe is not very pleasing to look at, so let's work on that next.\n",
    "First, run a for loop to show the classification report and log loss for \n",
    "each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in result_table.index:\n",
    "    print('\\n---- statistics for', i, \"----\\n\")\n",
    "    print(result_table.loc[i, 'clf_report'])\n",
    "    print(\"Model log loss:\", result_table.loc[i, 'log_loss'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the ROC curve\n",
    "A common technique in comparing models is to \n",
    "plot the ROC curve for each model and compare\n",
    "the shape and the AUC for each. We will use another\n",
    "for loop to generate the plot, and then\n",
    "format the axes and titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(14,12))\n",
    "\n",
    "for i in result_table.index:\n",
    "    plt.plot(result_table.loc[i]['fpr'], \n",
    "             result_table.loc[i]['tpr'], \n",
    "             label=\"{}, AUC={:.3f}\".format(i, result_table.loc[i]['auc']))\n",
    "    \n",
    "plt.plot([0,1], [0,1], color='orange', linestyle='--')\n",
    "\n",
    "plt.xticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.xlabel(\"False Positive Rate\", fontsize=15)\n",
    "\n",
    "plt.yticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.ylabel(\"True Positive Rate\", fontsize=15)\n",
    "\n",
    "plt.title('ROC Curve Analysis', fontweight='bold', fontsize=15)\n",
    "plt.legend(prop={'size':13}, loc='lower right')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you recall from the last lab, a better model will have a line that is closer to the \n",
    "upper-left corner. The AUC is calculated by measuring the area underneath each curve.\n",
    "The baseline AUC is 0.5 (see the dotted line). The SVC model performed impressively poorly bad, \n",
    "and none performed particularly well. The\n",
    "best model by AUC is Naive Bayes, but the decision tree had highest recall for fraud cases. This is interesting,\n",
    "because the classification report tells a much different story, with the Naive Bayes\n",
    "having lower recall. This is evidence that relying \n",
    "on just one evaluation statistic or algorithm is usually not a good idea."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "1. Run the model with at least 10 predictors. Does it improve the overall performance? \n",
    "2. Adjust some parameters of the random forest model. Note how similar to decision trees thes parameters are.\n",
    "3. Overall, Which model would you choose? Why?\n",
    "\n",
    "\n",
    "## Optional\n",
    "1. Construct a voting classifier (class or prob). Voting classifiers\n",
    "    take the predictions from several models and \"vote\" on the most\n",
    "    likely class. They can also use the average probability over\n",
    "    all models. There is a built-in function in `sklearn` to make\n",
    "    this fairly simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "aa3e7337b8d36894b7a51014394139a0eba3728352dce4cced1005d81d5028b1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}