09-10-future-employment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The future of employment\n",
    "\n",
    "How susceptible are jobs to computerisation? \n",
    "\n",
    "> We examine how susceptible jobs are to computerisation. To assess this, we begin by implementing a novel methodology to estimate the probability of computerisation for 702 detailed occupations, using a **Gaussian process classifier**. Based on these estimates, we examine expected impacts of future computerisation on US labour market outcomes, with the primary objective of analysing the number of jobs at risk and the relationship between an occupations probability of computerisation, wages and educational attainment.\n",
    "\n",
    "C. Frey, M. Osborne  *The future of employment: How susceptible are jobs to computerisation?* Technological Forecasting & Social Change 114 (2017) 254–280 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "First, together with a group of ML researchers, we subjectively hand-labelled 70 occupations, assigning 1 if automatable, and 0 if not. \n",
    "\n",
    "- For our subjective assessments, we draw upon a workshop held at the `Oxford University Engineering Sciences Department`, examining the automatability of a wide range of tasks. \n",
    "\n",
    "- Our label assignments were based on eyeballing the O*NET tasks and job description of each occupation. \n",
    "\n",
    "- The hand-labelling of the occupations was made by answering the question “Can the tasks of this job be sufficiently specified, conditional on the availability of big data, to be performed by state of the art computer-controlled equipment”. \n",
    "\n",
    "- Labels were assigned only to the occupations about which we were most confident.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Second, we use objective O*NET variables corresponding to the defined bottlenecks to computerisation. \n",
    "\n",
    "- We are interested in variables describing the level of perception and manipulation, creativity, and social intelligence required to perform it. \n",
    "- We identified **nine variables of O*NET** that describe these attributes. \n",
    "- We rely on the “level” rating which corresponds to specific examples about the capabilities required of computer-controlled equipment to perform the tasks of an occupation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T10:45:10.010738Z",
     "start_time": "2019-11-17T10:45:09.999374Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Computerisation bottleneck \n",
    "\n",
    "- **Perception and manipulation**\n",
    "    - Finger dexterity [variables of O*NET]\n",
    "        - The ability to make precisely coordinated movements of the fingers of one or both hands to grasp, manipulate, or assemble very small objects.\n",
    "    - Manual dexterity [variables of O*NET]\n",
    "        - The ability to quickly move your hand, your hand together with your arm, or your two hands to grasp, manipulate, or assemble objects.\n",
    "    - Cramped work space, awkward positions  [variables of O*NET]\n",
    "        - How often does this job require working in cramped work spaces that requires getting into awkward positions?\n",
    "- **Creative intelligence**\n",
    "    - Originality [variables of O*NET]\n",
    "        - The ability to come up with unusual or clever ideas about a given topic or situation, or to develop creative ways to solve a problem.\n",
    "    - Fine arts [variables of O*NET]\n",
    "        - Knowledge of theory and techniques required to compose, produce, and perform works of music, dance, visual arts, drama, and sculpture.\n",
    "- **Social intelligence**\n",
    "    - Social perceptiveness  [variables of O*NET]\n",
    "        - Being aware of others’ reactions and understanding why they react as they do.\n",
    "    - Negotiation [variables of O*NET]\n",
    "        - Bringing others together and trying to reconcile differences.\n",
    "    - Persuasion [variables of O*NET]\n",
    "        - Persuading others to change their minds or behaviour.\n",
    "    - Assisting and caring for others [variables of O*NET]\n",
    "        - Providing personal assistance, medical attention, emotional support, or other personal care to others such as coworkers, customers, or patients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We tested three Gaussian process classifiers using the **GPML toolbox** (Rasmussen and Nickisch, 2010) on our data, built around exponentiated quadratic, rational quadratic and linear covariances.\n",
    "\n",
    "- Rasmussen, C.E., Nickisch, H., 2010. Gaussian processes for machine learning (GPML) toolbox. J. Mach. Learn. Res. 11, 3011–3015.\n",
    "- Rasmussen, C.E., Williams, C.K.I., 2006. Gaussian Processes for Machine Learning. MIT Press.\n",
    "\n",
    "Having validated our approach, we proceed to use classification to predict the probability of **computerisation** for all 702 occupations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## GPy\n",
    "\n",
    "\n",
    "The Gaussian processes framework in Python. https://github.com/SheffieldML/GPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-16T08:43:41.737830Z",
     "start_time": "2019-11-16T08:41:29.393005Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting GPy\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4a/84/91dc7d63fa32d83a799d32071d56fe481bc1ce6b090509999e3463bfeeea/GPy-1.9.9-cp37-cp37m-macosx_10_9_x86_64.whl (1.5MB)\n",
      "\u001b[K    100% |████████████████████████████████| 1.5MB 8.2kB/s ta 0:00:016\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: six in /Users/datalab/anaconda3/lib/python3.7/site-packages (from GPy) (1.12.0)\n",
      "Collecting paramz>=0.9.0 (from GPy)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d8/37/4abbeb78d30f20d3402887f46e6e9f3ef32034a9dea65d243654c82c8553/paramz-0.9.5.tar.gz (71kB)\n",
      "\u001b[K    100% |████████████████████████████████| 71kB 25kB/s ta 0:00:0101\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: scipy>=0.16 in /Users/datalab/anaconda3/lib/python3.7/site-packages (from GPy) (1.2.1)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.7 in /Users/datalab/anaconda3/lib/python3.7/site-packages (from GPy) (1.16.2)\n",
      "Requirement already satisfied, skipping upgrade: decorator>=4.0.10 in /Users/datalab/anaconda3/lib/python3.7/site-packages (from paramz>=0.9.0->GPy) (4.4.0)\n",
      "Building wheels for collected packages: paramz\n",
      "  Building wheel for paramz (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /Users/datalab/Library/Caches/pip/wheels/c8/4a/0e/6e0dc85541825f991c431619e25b870d4b812c911214690cf8\n",
      "Successfully built paramz\n",
      "Installing collected packages: paramz, GPy\n",
      "Successfully installed GPy-1.9.9 paramz-0.9.5\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade GPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-16T08:43:43.852391Z",
     "start_time": "2019-11-16T08:43:43.540649Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pylab as plt\n",
    "import seaborn as sns\n",
    "import GPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-16T08:53:10.820467Z",
     "start_time": "2019-11-16T08:53:10.806681Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>soc</th>\n",
       "      <th>Element Name</th>\n",
       "      <th>id</th>\n",
       "      <th>label</th>\n",
       "      <th>Data Value</th>\n",
       "      <th>computerization</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Assisting and Caring for Others</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>2.205</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Cramped Work Space, Awkward Positions</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>1.415</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Fine Arts</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Finger Dexterity</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>2.000</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>11-1011</td>\n",
       "      <td>Manual Dexterity</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.015</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0      soc                           Element Name  id  label  \\\n",
       "0           0  11-1011        Assisting and Caring for Others  70      0   \n",
       "1           1  11-1011  Cramped Work Space, Awkward Positions  70      0   \n",
       "2           2  11-1011                              Fine Arts  70      0   \n",
       "3           3  11-1011                       Finger Dexterity  70      0   \n",
       "4           4  11-1011                       Manual Dexterity  70      0   \n",
       "\n",
       "   Data Value  computerization  \n",
       "0       2.205            0.015  \n",
       "1       1.415            0.015  \n",
       "2       0.915            0.015  \n",
       "3       2.000            0.015  \n",
       "4       0.000            0.015  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://github.com/computational-class/ccbook/tree/master/data/jobdata.csv\n",
    "df = pd.read_csv('../data/jobdata.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:49:19.364086Z",
     "start_time": "2019-11-17T08:49:19.359286Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_list=list(df['Data Value'])\n",
    "X=[]\n",
    "for i in range(0,585,9):\n",
    "    list1=data_list[i:i+9]\n",
    "    X.append(list1)\n",
    "X=np.array(X)\n",
    "\n",
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:49:19.757035Z",
     "start_time": "2019-11-17T08:49:19.751111Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0],\n",
       "       [0],\n",
       "       [0]])"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_list1=list(df['label'])\n",
    "Y=[]\n",
    "for i in range(0,585,9):\n",
    "    list1=data_list1[i]\n",
    "    Y.append(list1)\n",
    "Y=np.array(Y)\n",
    "Y=Y[:,np.newaxis]\n",
    "Y[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:49:20.389490Z",
     "start_time": "2019-11-17T08:49:20.341611Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<paramz.optimization.optimization.opt_lbfgsb at 0x1a1d24ec88>"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kernel = GPy.kern.RBF(input_dim=9, variance=1., lengthscale=1.)\n",
    "m = GPy.models.GPRegression(X,Y,kernel)\n",
    "m.optimize(messages=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:49:21.250748Z",
     "start_time": "2019-11-17T08:49:21.247309Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Name : GP regression\n",
      "Objective : 28.130643010540453\n",
      "Number of Parameters : 3\n",
      "Number of Optimization Parameters : 3\n",
      "Updates : True\n",
      "Parameters:\n",
      "  \u001b[1mGP_regression.         \u001b[0;0m  |               value  |  constraints  |  priors\n",
      "  \u001b[1mrbf.variance           \u001b[0;0m  |  0.3113242734729479  |      +ve      |        \n",
      "  \u001b[1mrbf.lengthscale        \u001b[0;0m  |   3.933616340596464  |      +ve      |        \n",
      "  \u001b[1mGaussian_noise.variance\u001b[0;0m  |  0.0964434513555219  |      +ve      |        \n"
     ]
    }
   ],
   "source": [
    "print(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:49:21.794413Z",
     "start_time": "2019-11-17T08:49:21.790514Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn.metrics import mean_squared_error, r2_score, explained_variance_score, roc_auc_score, accuracy_score\n",
    "\n",
    "X1, X2, y1, y2 = train_test_split(X, Y, random_state=0,\n",
    "                                  train_size=0.6, test_size = 0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:49:22.342245Z",
     "start_time": "2019-11-17T08:49:22.334694Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.8846153846153846, 0.8846153846153847)"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "bayes = GaussianNB()\n",
    "bayes.fit(X1, y1.flatten())\n",
    "y2_model = bayes.predict(X2)  \n",
    "accuracy_score(y2, y2_model), roc_auc_score(y2, y2_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T08:50:39.398687Z",
     "start_time": "2019-11-17T08:50:39.198743Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score =  0.8461538461538461\n",
      "ROC_AUC score = 0.8461538461538461\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "forest = RandomForestRegressor(n_estimators = 400)\n",
    "forest.fit(X1, y1.flatten())\n",
    "y2_model = forest.predict(X2)\n",
    "y2_model = [1 if i > 0.5 else 0  for i in y2_model ]\n",
    "print('Accuracy score = ', accuracy_score(y2, y2_model))\n",
    "print('ROC_AUC score =', roc_auc_score(y2, y2_model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:31:54.567805Z",
     "start_time": "2019-11-17T09:31:54.559429Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score =  0.8846153846153846\n",
      "ROC_AUC score = 0.8846153846153846\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC \n",
    "# \"Support vector classifier\"\n",
    "X1, X2, y1, y2 = train_test_split(X, Y, random_state=0,\n",
    "                                  train_size=0.6, test_size = 0.4)\n",
    "svc = SVC(kernel='linear', C=1E10)\n",
    "svc.fit(X1, y1.flatten())\n",
    "y2_model = svc.predict(X2)\n",
    "y2_model = [1 if i > 0.5 else 0  for i in y2_model ]\n",
    "print('Accuracy score = ', accuracy_score(y2, y2_model))\n",
    "print('ROC_AUC score =', roc_auc_score(y2, y2_model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:27:16.611981Z",
     "start_time": "2019-11-17T09:27:16.608482Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['explained_variance', 'r2', 'neg_median_absolute_error', 'neg_mean_absolute_error', 'neg_mean_squared_error', 'neg_mean_squared_log_error', 'accuracy', 'roc_auc', 'balanced_accuracy', 'average_precision', 'neg_log_loss', 'brier_score_loss', 'adjusted_rand_score', 'homogeneity_score', 'completeness_score', 'v_measure_score', 'mutual_info_score', 'adjusted_mutual_info_score', 'normalized_mutual_info_score', 'fowlkes_mallows_score', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted'])"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sklearn.metrics.SCORERS.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:38:41.102283Z",
     "start_time": "2019-11-17T09:38:28.061684Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8912698412698414"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "svc = SVC(kernel='linear', C=1E10)\n",
    "\n",
    "def cross_validation(model):\n",
    "    roc_auc= cross_val_score(model, X, Y.flatten(), scoring=\"roc_auc\", cv = 5)\n",
    "    return roc_auc\n",
    "\n",
    "np.mean(cross_validation(svc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:44:19.332339Z",
     "start_time": "2019-11-17T09:44:19.317059Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9103174603174604"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "svc = SVC(kernel='rbf', C=1E10, gamma = 'auto')\n",
    "\n",
    "def cross_validation(model):\n",
    "    roc_auc= cross_val_score(model, X, Y.flatten(), scoring=\"roc_auc\", cv = 5)\n",
    "    return roc_auc\n",
    "\n",
    "np.mean(cross_validation(svc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:44:37.133073Z",
     "start_time": "2019-11-17T09:44:37.129441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.0"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)/5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:55:17.207844Z",
     "start_time": "2019-11-17T09:55:17.172763Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score =  0.9230769230769231\n",
      "ROC_AUC score = 0.9230769230769231\n"
     ]
    }
   ],
   "source": [
    "X1, X2, y1, y2 = train_test_split(X, Y, random_state=0,\n",
    "                                  train_size=0.6, test_size = 0.4)\n",
    "m = GPy.models.GPRegression(X1,y1,kernel)#, normalizer = True)\n",
    "m.optimize(messages=False)\n",
    "y2_model = m.predict(X2)[0]\n",
    "y2_hat = [1 if i > 0.5 else 0  for i in y2_model ]\n",
    "print('Accuracy score = ', accuracy_score(y2, y2_hat))\n",
    "print('ROC_AUC score =', roc_auc_score(y2, y2_hat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:55:23.474806Z",
     "start_time": "2019-11-17T09:55:23.316121Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEXCAYAAACOFGLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHV5JREFUeJzt3XuYXXV97/H3J5FcBhJikglEkpiYTLw0SsF5BDlYUCjByuH0eMELIhYVWm3VPiCHU4qnPe1RTw/WIhaPETAotlq1pQcVuUjFC5A6QayPeHBCQ24HcMgQDDMhgcz3/PFb46wZ955Ze7L2bebzep79rL0ue+3vegb2N7+1fr/vTxGBmZlZWWY0OwAzM5tanFjMzKxUTixmZlYqJxYzMyuVE4uZmZXKicXMzErlxGJmZqVyYjEzs1I5sZiZWame0+wAmmHx4sWxcuXKZodhZtZWNm/e/HhEdE503LRMLCtXrqSnp6fZYZiZtRVJ24oc51thZmZWKicWMzMrlROLmZmVyonFzMxK5cRiZmalmpa9wialvx96e2HPHliwALq6YOHCZkdlZtZy3GIpor8fNm2C/fth0aK03LQpbTczs1GcWIro7YXDD08vaeR9b2+zIzMzazlOLEXs2QMdHaO3dXSk7WZmNooTSxELFsDg4Ohtg4Npu5mZjdI2D+8lzQWWR8TPG/7lXV3pmQqklsrgIAwMwLp1DQ/FzKxmDe581PItFknzJd0EPAZcWmH/Okk/lrRN0tWSyr+mhQvhhBNg9mzYvTstTzjBvcLMrPU1ofNRO7RYhoCrga8DJ1bYfw1wGXAbcCdwNnBT6VEMJxczs3bS2wsHD8LWrbB3L8yblxJMb2/dftNavsUSEU9FxLeBZ8fuk9QJrIqIWyLiIPBF4MxGx2hm1rJ27ICHHoIDB+DII9PyoYfS9jpp+cQygWXA9tz6TmBppQMlXSipR1JPX19fQ4IzM2u6X/4SZsyAuXPTcIm5c9P6L39Zt69s98Qyi3SrbNgQcLDSgRGxISK6I6K7s3PCeWrMzKaGefNgaAj27YOItBwaStvrpB2esYznEeCY3PoyoH7tOzOzdrNiRWql7N4NTz6ZEsrSpVDHf2C3dYslIrYDA5JOlTQTOA/4SpPDMjNrHV1dMHMmrFoFJ56YljNnpu110vItFknzgB8B84A5kk4FPgSsjogrgfOBG4AFwMaI+H6zYjUzaznDPVp7e1OrZcGCNAavjsMlWj6xRMReYM04++8DXtq4iMzM2kyDh0u0fGIxM7ND5JH3ZmZWmiaMvHdiMTObypow7YcTi5nZVNaEaT+cWMzMprImTPvhxGJmNpV1daVpPgYG0sj74fd1HMfixGJmNpU1YdoPdzc2M5vqGjyOxS0WMzMrlROLmZmVyonFzMxK5cRiZmalcmIxM7NSObGYmVmpnFjMzKxUHsdiZjbVuWy+mZmVxmXzzcysVC6bb2ZmpXLZfDMzK1UTyub74b2Z2VTW1QV33JGeqTzzDBx2WHpwf/rpdftKt1jMzKa6iPHXS+YWi5nZVNbbC0uXwpo1I9sGBtL2OpXSd4vFzGwq88P7yiSdI2mrpC2SLhiz73RJP8n2Xy9pZrPiNDNrOZ7z/tdJmgd8HDg5e31EUmfukM8CbwXWAGuBMxoepJlZq/Kc9xWtB+6KiF0R8ShwJ3Babv/+3PvZwC8aGZyZWUtbuBDWroUtW+DWW9Ny7dppX9JlObAtt74TWJpbfzvwZWATcGNEbK50EkkXSuqR1NPX11e3YM3MWkp/P2zeDAcOwPz5abl587Qv6TILGMqtDwEHc+vvAT4NvB94g6QVlU4SERsiojsiujs7OysdYmY29fT0wAMPwPbtI68HHkjb66QdEssjwDG59WXADgBJLwFeHhGfioi7gZuA9zU+RDOzFtXTk1onM2bAvHlp2d8/7RPLrcB6SUskHQ2cBNyW7dsPrJB0lKQZwHHAE02K08ys9fT1waxZMHt2KkI5e3Zar+MjgZYfIBkRj0m6HLgn23QxcIak1RFxpaS/An5ISjI/BD7ZpFDNzFpPZyds3Qq7d8Ozz8JznpNeq1bV7StbPrEARMRGYGOVfVcCVzYyHjOztrF2Lfz0p6m1Aqle2IEDaXudtMOtMDMzm6wjj4Rly9KzlaefTstly9L2OmmLFouZmU3S8HTEM2aM3AqbP7+uJV2cWMzMprK9e+GII2D58pFtTzyRtteJb4WZmU1l8+fD0BDs25dKuuzbl9bnz6/bV7rFYmY2lS1fDnPmpF5hTz6ZxrIsXZp6i9WJE4uZ2VTW1ZUGRK5alcrlDw66CKWZmR2ChQvThF6zZ6dWy+zZab2ORSjdYjEzm+qGk0uDuMViZmalcmIxM7NS+VaYmdlU198Pvb0jgyW7uqb9RF9mZjZZ/f2waRPs3w+LFqXlpk3TfqIvMzObrN5eOPzw9JJG3vf21u0rnVjMzKayPXvS+JW8jo661gpzYjEzm8oWLEiDIvMGB9P2OnFiMTObyrq60kj7gYFUK2z4vUfem5nZpHjkvZmZlc4j783MrJ05sZiZWal8K8zMbKrzyHszMyuNR96bmVmpPPLezMxK5ZH3lUk6R9JWSVskXVBh/59J2iHpYUknNSNGM7OW1ISR9y3/8F7SPODjwInAQeB+STdHRF+2/wKgG1gLPA3MblasZmYtp6srPVOB0XPer1tXt69shxbLeuCuiNgVEY8CdwKn5fb/MfDBiNgXydNNidLMrBU1YeR9OySW5cC23PpOYCmApMOAo4ELJD0o6Z8kLap0EkkXSuqR1NPX11f3oM3Mpqt2SCyzgKHc+hDplhjAYuC5wL8ALwK2A5dXOklEbIiI7ojo7uzsrGO4ZmYtpNW7G0u6U9IfTXDM30i6+tDCGuUR4Jjc+jJgR/b+ceCpiLg9IgL4Z+CFJX63mVl7a4PuxqcCE9VaPgJ452SCqeJWYL2kJZKOBk4CbgOIiGeATZLOzI49C/hhid9tZtbemtDduLReYZIEvBj4bWBfWeeNiMckXQ7ck226GDhD0uqIuBL4A+ALkj5FSipXlPXdozS4JIKZWSmGuxsffvjItjp3N1a6g1RlZ+rq+zi1JSABN0bEOw4xtrrp7u6Onp6e4h8Yvkd5+OGju+vVuWeFmdkhK/H3S9LmiOie6Lhxb4VFxF7gdmAwewE8m1vPvwaArcBngPfXFG2ra8I9SjOzUrTiRF8Rcdbwe0lDwGciYmoljons2ZN6U+R1dKQ/kplZq2vxib7uAqbfP9ObUBLBzKxd1fTwPiJeXa9AWloTSiKYmbWrWsexHC/pbZI6x2y/UNIPJd0t6fXlhtgCmnCP0sysXdXa3fgqUrHH5cMbJL0N+N+5Y74s6eSI2FRCfK2jwfcozczaVa3PWH4DuDsiDuS2/XdgL/Aa4LdIPcQuKSc8MzNrN7W2WA4DflXBMRvx/gLgLyPiO9m2W4BXlhWgmZm1l1pbLFtJc58M+y+kcS0bctsGgCWHGJeZmbWpWhPL3wG/Kenbkm4l3fr6akTszB2zllyrxszMppdab4VdDbwRGO52vAv40PBOSV2kmR6/Wkp0ZmbWdmodxzIg6ZXA6aTnLd+OiIHcIXOAdzNSMHLqcBFKM7NCaq5unJWqv6XKvp8APznUoFpOvojbokVpgOSmTR7LYmZWwaTK5ktaDZwNrAbmRMS7S42q1eSLUMLIsrfXY1vMzMaoeWpiSX8JPAhcCbwX+L3cvjdL+qmkY8sLsQU0YaIcM7N2VWtJl3cBfwLszJa3jznkJuAo4IOlRNcqXITSzKywWlss7yN1JX55RPxP4Of5nRGxn1QB+ZRywmsRXV2p6OTAAESMvO+aaJZmM7Ppp9bE8iLgjogYbyKS3cDSyYfUglyE0syssFof3j9N6lI8npXAM5OKppW5CKWZWSG1tlh+AJwpaU2lnZJeTLoNNvW6HJuZWSG1Jpa/IA2M/J6kdwDzASStydZvI7WCPl1qlGZm1jZqHXn/r5LeDlwPfC6368FsKeCqiLixpPjMzKzNTGbk/T9Iuhf4Q+BkYDFpPpb7gRsi4rvlhmhmZu1kUiPvI2I7cGnJsZiZ2RRQ6wDJ67OpiMc75k8lXXZoYf3aOc+RtFXSFkkXVDnmOklbyvxeMzOrXa0P799JKos/npVAaYlF0jzg46TbbicDH5HUOeaYVwNHl/WdZmY2eTXXChuPpCOBk4ChEk+7HrgrInZFxKPAncBpue+cQ+qtdkWJ32lmZpM07jOWrLVwHzAzt/kdks6qcPhMUp2ww4CvlxYhLAe25dZ3Mnpk/4dJ3Zv7S/xOMzObpHETS0TslfQYqRUCEKSxK/OrfOQAaSzLe0uLEGYxugU0BBwEkPRS4NiI+BNJK8c7iaQLgQsBVqxYUWJ4ZmaWV+RW2HqgE1hCGqdybbY+9rUYmBsRr42IXSXG+AhwTG59GbAje38+sEbS/cA3geWSvlzpJBGxISK6I6K7s7Oz0iFmZlaCCbsbZ1MPDwBI2gbsmqAIZdluBT4qaQkpEZ4EXJTFdglwSRbbSlKBzDfXJQpPTWxmVkhND+8jYlVE/HnR4yWdK+n62sMa9Z2PAZcD95BqlV0MnCHpkkM5b02Gpybevz9NTbx/f1rv92MdM7OxJjVAsgYnkm5XVRx7UlREbAQ2TnDMw0DF4piHzFMTm5kVVmp34ynLUxObmRXmxFKEpyY2MyvMiaUIT01sZlaYE0sRnprYzKywej+8nzo8NbGZWSFusZiZWalqLZu/vl6BmJnZ1FBri+WWbF6UKyQtK3D8FzjEMSxmZtZeak0sXyNVMP5zYKukb0j6XUkzKx0cEf8aETccapBmZtY+ai3p8ibShFrvIZVXOZOUbHZK+pgk9781M5vman54HxG/jIjrIuJU4PmkOl67gUuB/yvpO1mNsNnlhmpmZu3gkHqFRcTOiPhYRKwDjge+BLwK+DzwiKRPSDpm3JOYmdmUcsjjWCStAN6SvY4lzdlyP+lZzAeA35d0WURcdajf1VQum29mVsikWiySjpb0fkl3A1uBjwErSVMEHx8RxwPDCech4K8lvauckJvAZfPNzAqrdRzLeyTdSZrB8ROksvjfBc4DnhcRfxgR9wNExMGI+AfgFcC/k03I1ZbyZfOlkfe9vc2OzMys5dR6K+wz2fIR4Abguoh4aLwPRMSgpB7grEnE1xr27EktlbyOjlQ3zMzMRqk1sXydNOf9NyLiYA2fux24t8bvah3DZfOHJ/gCl803M6uipsQSEWdP5ksi4pCmJ266rq70TAVSS2VwMJXNX7euuXGZmbUgF6EswmXzzcwKc9n8olw238ysECeWojyOxcysEN8KK8LjWMzMCnNiKcLjWMzMCnNiKWLPntQbLK+jI203M7NRnFiKGB7HkudxLGZmFbVFYpF0TjZz5RZJF4zZ9wFJP5P0sKQvSCq/Q0JXVxq3MjAAESPvuzz9jJnZWC2fWCTNAz4OnJy9PiKpM3fIXlJV5TWkispvLj0Ij2MxMyusHbobrwfuiohdAFkRzNNIc7+MGtUv6d+A+vzaexyLmVkhLd9iAZYD23LrO4GlYw+S1AG8Dri50kkkXSipR1JPX19fXQI1M7P2SCyzgKHc+hAwqgCmpBmkWSuvjoiHK50kIjZERHdEdHd2dlY6xMzMStAOieURID+98TLSfDAASBKp4vIDEXFNg2MzM7Mx2uEZy63ARyUtISXCk4CLcvuvAR6NiA/XNQqXdDEzK6TlWywR8RhwOXAP8APgYuAMSZdIOpmUZM7JuiJvkfTW0oNwSRczs8LaocVCRGwENlbZXf/kmC/pAiPL3l73FDMzG6PlWywtwSVdzMwKc2IpwiVdzMwKc2IpwiVdzMwKc2IpwiVdzMwKa4uH9y3BJV3MzApxi8XMzErlxGJmZqVyYjEzs1I5sZiZWamcWMzMrFROLGZmVip3Ny7K1Y3NzApxi6UIVzc2MyvMiaWIfHVjaeR9b2+zIzMzazm+FVbEnj2p6OQdd8Djj8PixXDccb9e8djMzNxiKeSpp+Dmm2HfPjjqqLS8+ea03czMRnFiKWLXLpg1KxWflNJy1qy03czMRnFiKWLvXnjZy+Cww1Ir5bDD0vrevc2OzMys5fgZSxFLl6b5V1avHtn2xBNpu5mZjeIWSxGnnJISyRNPwNDQyPtTTml2ZGZmLceJpYjVq+Hcc1MX4x070vLcc0e3YMzMDPCtsOJWr3YiMTMrwC0WMzMrlROLmZmVqi0Si6RzJG2VtEXSBWP2rZP0Y0nbJF0tqS2uycxsqmr5H2FJ84CPAydnr49I6swdcg1wGfAC4GXA2Q0P0szMfqXlEwuwHrgrInZFxKPAncBpAFmCWRURt0TEQeCLwJnNC9XMzNohsSwHtuXWdwLDIxOXAdur7BtF0oWSeiT19PX11SVQMzNrj8QyCxjKrQ8BBwvsGyUiNkREd0R0d3Z2VjrEzMxK0A6J5RHgmNz6MmBHgX1mZtYE7ZBYbgXWS1oi6WjgJOA2gIjYDgxIOlXSTOA84CvNC9XMzFp+5H1EPCbpcuCebNPFwBmSVkfElcD5wA3AAmBjRHy/SaGamRltkFgAImIjsLHKvvuAlzYyHjMzq64dboWZmVkbcWIxM7NSObGYmVmpnFjMzKxUbfHwviX090NvL+zZAwsWQFcXLFzY7KjMzFqOWyxF9PfDpk2wfz8sWpSWmzal7WZmNooTSxG9vWk64sMPB2nkfW9vsyMzM2s5TixF7NkDHR2jt3V0pO1mZjaKE0sRCxbA4ODobYODabuZmY3ixFJEVxcMDKRXxMj7rq5mR2Zm1nKcWIpYuBBOOAFmz4bdu9PyhBPcK8zMrAJ3Ny5qOLmYmdm43GIxM7NSObGYmVmpnFjMzKxUfsZSlEu6mJkV4hZLES7pYmZWmBNLEb29cPAgbN0K996blgcPuqSLmVkFTixFbN8ODz0EBw7AkUem5UMPpe1mZjaKE0sRe/fCjBkwd24qQjl3blrfu7fZkZmZtRwnliLmz4ehIdi3L5V02bcvrc+f3+zIzMxajnuFFbF8OcyZk8q5PPkkzJsHS5dCZ2ezIzMzazlOLEV0daUeYKtWpXL5g4MuQmlmVoVvhRXhIpRmZoW1dItF0jHAl4HnAz8A3hkRT+f2Hwd8ElgK7M32/7guwbgIpZlZIa3eYvko8MWIWA48C/z+mP1LgPMiYg3wCeDKBsdnZmZjtHpiOQu4IXv/eeDM/M6IuDUiHs5W7wN8b8rMrMlaNrFIei6wLyKG5wTeSbrlVc15wFfHOd+Fknok9fT19ZUYqZmZ5bXEMxZJnwFePmbzxcBQbn0IOFjl8/8ZeCVwWrXviIgNwAaA7u7uOJR4zcysupZILBFx0dhtkmYACyTNiogDwDJgR4Xj1gOXAWdGxDN1D9bMzMaliNb9x7ukm4GvRcRGSTcC34qIG3P7X0N6wP+6iHi8hvP2AdsmGdZioPB3TRG+5ulhul3zdLteOPRrfn5ETDgyvNUTyyrg70mtlW8BFwEdwJeAs4EHgSOAp7KP/Cgi3lTnmHoiorue39FqfM3Tw3S75ul2vdC4a26JW2HVRMRW4MQxm/cCr8ver2lsRGZmNpGW7RVmZmbtyYmldhuaHUAT+Jqnh+l2zdPteqFB19zSz1jMzKz9uMViZmalcmIxM5uiJM2VtLbR3+vEUoWkcyRtlbRF0gVj9q2T9GNJ2yRdnQ3mbHsTXPMHJP1M0sOSviCppXsUFjXeNeeOuU7SlkbHVi8TXbOkP5O0I/tbn9SMGMs2wX/bp0v6Sbb/ekkzmxVnWSTNl3QT8BhwaYX99f0Niwi/xryAeaRR/scARwOPAp25/d8FXgvMBO4CfrfZMTfgmi8AZpG6qN8GnNvsmOt9zdkxrwa+AWxpdrwN/Dt/HZgLCJjT7JgbcM1bgXXZ/8/fB17b7JhLuOYjSCWu3g1cW2F/XX/DpsS/tOtgPXBXROyKiEeBO8nqkEnqBFZFxC0RcRD4ImOqLrepqtcMEBHXR8SBiHgW+DemRiXpca9Z0hzgL4ArmhRfPYx7zcAfAx+MiH2RPF3xLO1lomven3s/G/hFI4Orh4h4KiK+TZpuZJRG/IY5sVS2nNElX/KVlZcB26vsa2fjXfOvSOogDVC9uUFx1dNE1/xh4NNAfyODqrOq1yzpMNK/6C+Q9KCkf5K0qAkxlm2iv/PbSRMKbgJujIjNDYytGer+G+bEUtksqldWHm9fO5vwurL7sJ8Hro6ReXDaWdVrlvRS4NiI+GIzAquj8f7Oi4HnAv8CvIj043N5Q6Orj4n+234P6R8Q7wfeIGlFA2Nrhrr/hjmxVPYI6X7ssHxl5fH2tbNxr0uSgGuBByLimgbHVi/jXfP5wBpJ9wPfBJZL+nKD46uH8a75ceCpiLg90o34fwZe2OD46qHqNUt6CfDyiPhURNwN3AS8r/EhNlTdf8OcWCq7FVgvaYmko4GTSA+siYjtwICkU7PeI+cBX2leqKWpes2Za4BHI+LDTYmuPsb7O18SES+MiN8EfgfYERFvbmKsZRnvmp8BNkkavt9+FvDD5oRZqvH+294PrJB0VNYiPw54oklxNkQjfsM88r4KSe9k5KHtJdlydURcKel40pTJC4CNETElHu5Wu2bgXlIvkn/PHX5FRPx946Krj/H+zrljVgJ3RMSUKHo6wX/bLwC+ABxFSirvjoiBxkdZrgmu+RLSbbD9jFzz4K+fpX1Imgf8iNQjbg7QB3yIBv2GObGYmVmpfCvMzMxK5cRiZmalcmIxM7NSObGYmVmpnFjMzKxUTixmZlYqJxazaUrSGyWFpI3NjsWmFicWMzMrlROLmZmVyonFzMxK5cRi1gamwnS5Nn04sZjlSLo5e6B9UZX9X8n2v7bg+TZmx79R0rHZ+Z+QNCDpe5JOr/CZh7PPHCXpKkn9wO4xx7wid659kn4k6d1VYjhK0jXZPPb7Jf1c0sWkqYfNSufEYjbatdnyrWN3SDqSVEp+B6kUey1eDtwDdJLmOfkJcDLwLUn/qcpnrgDeAPwj8J1cHOcAPwBOz7bfRJpT47OSPjIm5ueTKvb+AbCHVLn4YeCvgP9a4zWYFfKcZgdg1mK+QZoI6VWSnhcR/y+3742kEuTXR8RQxU9X9yHg/PyMlJL+CPgkcI2kb2bzoeT9NvCyiOjPfeb5wOdIc4a8KiIezLZ3Aj3AZZI+FxG92UeuI03NeyVwaTaBF5JeBdxe4zWYFeIWi1lORDxLmqdiBnDOmN3nkaZxvX4Sp/7S2GmOI+Jq0pwZzyO1Xsb6ZD6pZN4HdJCSxIO5c/WRptcVcDaApBcDp5HmNP/TyM2RERHfA/52EtdhNiG3WMx+3XXAZcBbgL8BkLQc+C3gtmwGvlrdWWX7JtKshWtJc83nfb/C8a/Klq+R1D1m39psuTpb/odseUtE7K9wrs3VwzWbPCcWszEiYouku4BTJK2MiIeBc0mtgc9O8rS/qLL9qWw5q8K+RypsW5wtzxvnu+ZmyyXZstp85gfGOYfZpPlWmFllww/x35It305KDv9nkuebXWX7imz5WIV9lZ7j7M2WL4oIVXn9XnbMcCtlUZXvXlJlu9khcWIxq+xrpF5Ub5F0HPAbwOcrPGAvauxtKyQdQerZFcC9Bc9zf7Z8ZYFjf5YtT5VUqWvxawp+p1lNnFjMKoiIfcDfAceSenTBSCtmMt4v6YThFUkzgI8BC4Fv1vDc5oZs+T8kvSS/Q9IcSW/JbboT6CNdw6Vjjj2P1JXZrHROLGbV5W+HfS/fC2sS7gN+IOlbkq4ljWN5H6nH1nuLniQi7iJ1UX4ecL+k2yV9VtI/AruA/5Y79uns3EPAxyTdlw3YvJvUs829wqwunFjMqoiIH5FuPYlDa60AXEX6kV9JevC+ANgAvKLWXmYR8QHgfFKvrlcC7wCOJ43BeduYY78KrAe+C7wQeD0wQOrh9p3JXozZeJTr2m5mOZKeA2wnjRtZmt0eq/UcG0lJ4E3Zj7zZlOfuxmbVvR5YClydTyrZs5LxuvsC3BsRN9YzOLNW5cRiVoGkDuDDpLEe/2vM7heTno+M5wjAicWmJScWsxxJ7yLV6OomjWC/OCJGDTCMiI3AxoYHZ9Ym/PDebLR5wH8k9aS6KCL+usnxmLUdP7w3M7NSucViZmalcmIxM7NSObGYmVmpnFjMzKxUTixmZlYqJxYzMyvV/wc3Wszde4CA4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y2, y2_model, 'ro', alpha = 0.2)\n",
    "plt.xlabel('y_pred', fontsize = 20)\n",
    "plt.ylabel('y_test', fontsize = 20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:56:00.777209Z",
     "start_time": "2019-11-17T09:55:58.112594Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC_AUC score = 0.8779356399319636\n"
     ]
    }
   ],
   "source": [
    "roc_auc_list = []\n",
    "for i in range(100):\n",
    "    X1, X2, y1, y2 = train_test_split(X, Y, random_state=i,\n",
    "                                      train_size=0.6, test_size = 0.4)\n",
    "    m = GPy.models.GPRegression(X1,y1,kernel)#, normalizer = True)\n",
    "    m.optimize(messages=False)\n",
    "    y2_model = m.predict(X2)[0]\n",
    "    y2_hat = [1 if i > 0.5 else 0  for i in y2_model ]\n",
    "    roc_auc = roc_auc_score(y2, y2_hat)\n",
    "    roc_auc_list.append(roc_auc)\n",
    "    \n",
    "print('ROC_AUC score =', np.mean(roc_auc_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-17T09:56:25.630941Z",
     "start_time": "2019-11-17T09:56:22.901960Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC_AUC score = 0.8921962481962482\n"
     ]
    }
   ],
   "source": [
    "roc_auc_list = []\n",
    "for i in range(100):\n",
    "    X1, X2, y1, y2 = train_test_split(X, Y, random_state=i,\n",
    "                                      train_size=0.8, test_size = 0.2)\n",
    "    m = GPy.models.GPRegression(X1,y1,kernel)#, normalizer = True)\n",
    "    m.optimize(messages=False)\n",
    "    y2_model = m.predict(X2)[0]\n",
    "    y2_hat = [1 if i > 0.5 else 0  for i in y2_model ]\n",
    "    roc_auc = roc_auc_score(y2, y2_hat)\n",
    "    roc_auc_list.append(roc_auc)\n",
    "    \n",
    "print('ROC_AUC score =', np.mean(roc_auc_list))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}