Week 14 CRISP-DM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 14 - CRISP-DM\n",
    "We will implement the stages of the CRISP-DM pipeline in this lab. We will use the `netattacks` dataset to demonstrate the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn import tree\n",
    "from sklearn import ensemble\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn import svm\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Business Understanding\n",
    "We need a way to identify malicious traffic on our network and automatically block it. The model should be fast and accurate. \n",
    "\n",
    "Criteria (I made these up, but they are reasonable):\n",
    "\n",
    "1. Recall for benign traffic should exceed 99.99% to avoid blocking legitmate traffic.\n",
    "2. Recall for malicious traffic should exceed 98%.\n",
    "3. Precision for malicious traffic should exceed 90% (to avoid excessive false positives).\n",
    "4. Other criteria (like AUC) could be used as a tie-breaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data \n",
    "We will use the KDDCup Network Attack dataset from Labs 2 and 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load txt file\n",
    "names = pd.read_csv('data/kddcup.names', header=None, delimiter=':',skiprows=1)\n",
    "\n",
    "# make column 0 into a list\n",
    "name_list = names[0].tolist()\n",
    "\n",
    "# add the last column with type\n",
    "name_list.append('type')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "netattacks = pd.read_csv('data/kddcup.data_10_percent_corrected', names=name_list, header=None, index_col=None)\n",
    "\n",
    "# use a 0 (normal) or 1 (malicious) to code bad traffic\n",
    "netattacks['label'] = np.where(netattacks['type'] == 'normal.', 0, 1)\n",
    "\n",
    "netattacks = netattacks.select_dtypes(include=np.number)\n",
    "\n",
    "# train-test split\n",
    "train, test = train_test_split(netattacks, test_size=0.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "This dataset is already cleaned and ready for use. Let's view some\n",
    "descripteve statistics to verify this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the display() function works better for displaying dataframes in Jupyter Notebooks than print().\n",
    "display(netattacks.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train/test split\n",
    "Next, we will use a 75/25 split for this data. We will view\n",
    "some summarization by outcome `Class` and a few variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(516)\n",
    "\n",
    "# create train and test\n",
    "train, test = train_test_split(netattacks, test_size=0.25)\n",
    "print(\"Rows in train:\", len(train))\n",
    "print(\"Rows in test:\", len(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use all of the available predictors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get columns not label\n",
    "pred_vars = list(netattacks.columns)\n",
    "\n",
    "# remove 'label' because it is what we are trying to predict\n",
    "pred_vars.remove('label')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# rename train and test for easier copy/paste in future projects\n",
    "\n",
    "train_X = train[pred_vars]\n",
    "train_y = train['label']\n",
    "\n",
    "test_X = test[pred_vars]\n",
    "test_y = test['label']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define models\n",
    "In this section, we will train a several different types of classifiers. They\n",
    "will be presented one-by-one in this lab, but it is possible to set up the \n",
    "training in a `for` loop to make your code simpler and more flexible.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Decision tree\n",
    "The first model we fit is a decision tree, which we have been working with for some \n",
    "time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtree = tree.DecisionTreeClassifier(criterion='entropy', max_depth=10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest\n",
    "A random forest classifier consists of many decision trees\n",
    "with randomized parameters (hence the name). One of the parameters\n",
    "is the number of trees in the classifier. The default is 100, but \n",
    "this can be reduced for large datasets if training is slow.\n",
    "Full documentation on this implementation is \n",
    "[here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = ensemble.RandomForestClassifier()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "There are many types of neural networks in use today. We will \n",
    "use perhaps the most common type, the Multi-Layer Perceptron (MLP). \n",
    "The documentation for this method is \n",
    "[here](https://scikit-learn.org/stable/modules/neural_networks_supervised.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp = MLPClassifier(hidden_layer_sizes=(20,20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machines\n",
    "Next, we will train a support vector machine. The basic \n",
    "model works well for linearly separable classes. You may also\n",
    "want to try an RBF kernel, which can perform better in some situations.\n",
    "\n",
    "The documentation for SVMs is \n",
    "[here](https://scikit-learn.org/stable/modules/svm.html). By default, \n",
    "the model only produces label outputs. We will adjust the probability\n",
    "parameter so that the model can also report class probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc = svm.SVC(probability=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "The Naive Bayes classification algorithm is based on conditional probabilities\n",
    "from Bayes' Theorem. It is one of the simplest algorithms, yet tends to \n",
    "perform quite well in many scenarios.\n",
    "\n",
    "Documentation for this is \n",
    "[here](https://scikit-learn.org/stable/modules/naive_bayes.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb = GaussianNB()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "The last method we will use in this lab is the logistic regression.\n",
    "This a method that has existed in statistics for a long time and,\n",
    "like decision trees, results in models that are human-interpretable.\n",
    "\n",
    "The documentation for this method is \n",
    "[here](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training & Evaluation\n",
    "Next, we will train and  \n",
    "evaluate their performance on out-of-sample data (our test dataset).\n",
    "We will use multiple evaluation statistics.\n",
    "\n",
    "The code for this section is adapted from \n",
    "[this tutorial](https://abdalimran.github.io/2019-06-01/Drawing-multiple-ROC-Curves-in-a-single-plot),\n",
    "which seems to be an adaptation of \n",
    "[this StackOverflow answer](https://stackoverflow.com/questions/42894871/how-to-plot-multiple-roc-curves-in-one-plot-with-legend-and-auc-scores-in-python).\n",
    "\n",
    "It works by iterating through the models using a for loop,\n",
    "and then storing the statistics in a dataframe. These models \n",
    "perform poorly, and may result in warning message from `sklearn`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of our models\n",
    "model = [dtree, rf, mlp, svc, nb, lr]\n",
    "\n",
    "# empty dataframe to store the results\n",
    "result_table = pd.DataFrame(columns=['classifier_name', 'fpr','tpr','auc', \n",
    "                                     'log_loss', 'clf_report'])\n",
    "\n",
    "for clf in model:\n",
    "    # print the name of the classifier\n",
    "    print(clf.__class__.__name__)\n",
    "    \n",
    "    print('fitting model')\n",
    "    clf.fit(train_X, train_y)\n",
    "\n",
    "    print('evaluating model')\n",
    "    # get predictions\n",
    "    yproba = clf.predict_proba(test_X)\n",
    "    yclass = clf.predict(test_X)\n",
    "    \n",
    "    # auc information\n",
    "    fpr, tpr, _ = metrics.roc_curve(test_y,  yproba[:,1])\n",
    "    auc = metrics.roc_auc_score(test_y, yproba[:,1])\n",
    "    \n",
    "    # log loss\n",
    "    log_loss = metrics.log_loss(test_y, yproba[:,1])\n",
    "    \n",
    "    # add some other stats based on confusion matrix\n",
    "    clf_report = metrics.classification_report(test_y, yclass, digits=6)\n",
    "    \n",
    "    # add the results to the dataframe\n",
    "    result_table = result_table.append({'classifier_name':clf.__class__.__name__,\n",
    "                                        'classifier': clf,\n",
    "                                        'fpr':fpr, \n",
    "                                        'tpr':tpr, \n",
    "                                        'auc':auc,\n",
    "                                        'log_loss': log_loss,\n",
    "                                        'clf_report': clf_report}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the results\n",
    "For easy formatting later, reset the dataframe index to be the classifier names\n",
    "rather than a numeric index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_table.set_index('classifier_name', inplace=True)\n",
    "display(result_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The raw dataframe is not very pleasing to look at, so let's work on that next.\n",
    "First, run a for loop to show the classification report and log loss for \n",
    "each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in result_table.index:\n",
    "    print('\\n---- statistics for', i, \"----\\n\")\n",
    "    print(result_table.loc[i, 'clf_report'])\n",
    "    print(\"Model log loss:\", result_table.loc[i, 'log_loss'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the ROC curve\n",
    "A common technique in comparing models is to \n",
    "plot the ROC curve for each model and compare\n",
    "the shape and the AUC for each. We will use another\n",
    "for loop to generate the plot, and then\n",
    "format the axes and titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(14,12))\n",
    "\n",
    "for i in result_table.index:\n",
    "    plt.plot(result_table.loc[i]['fpr'], \n",
    "             result_table.loc[i]['tpr'], \n",
    "             label=\"{}, AUC={:.3f}\".format(i, result_table.loc[i]['auc']))\n",
    "    \n",
    "plt.plot([0,1], [0,1], color='orange', linestyle='--')\n",
    "\n",
    "plt.xticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.xlabel(\"False Positive Rate\", fontsize=15)\n",
    "\n",
    "plt.yticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.ylabel(\"True Positive Rate\", fontsize=15)\n",
    "\n",
    "plt.title('ROC Curve Analysis', fontweight='bold', fontsize=15)\n",
    "plt.legend(prop={'size':13}, loc='lower right')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy\n",
    "\n",
    "Next, we will save the best model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the model that works the best and pickle it.\n",
    "import pickle\n",
    "best_model = result_table.loc[0, 'classifier']\n",
    "\n",
    "with open('netattack_dtree.pkl','wb') as f:\n",
    "    pickle.dump(best_model, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "aa3e7337b8d36894b7a51014394139a0eba3728352dce4cced1005d81d5028b1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}