applied_machine_learning_in_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applied machine learning in Python\n",
    "\n",
    "**Applied machine learning in Python** was the third course in the **Applied Data Science with Python** specialization. This notebook shows my work for the four assignments that I had to solve during this course:\n",
    "\n",
    "* **Assignment 1: Introduction to machine learning**\n",
    "\n",
    "In this assignment I've used the Breast Cancer Wisconsin Database to create a classifier that can help diagnose patients. \n",
    " \n",
    "* ** Assignment 2:  Supervised machine learning**\n",
    "\n",
    "This assignment was split into parts: regression and classification. In the first part I've fitted the data using Linear Regression and Lasso Regression. For the second part I've worked with the UCI Mushroom Data Set to train a model to predict whether or not a mushroom is poisonous. \n",
    "\n",
    "* **Assignment 3:  Evaluation**\n",
    "\n",
    "In this assignment I have trained several models and evaluated how effectively they predict instances of fraud, using Support Vector Machines and Logistic Regression.\n",
    "\n",
    "* **Assignment 4: Understanding and predicting property maintenance fines**\n",
    "\n",
    "For this assignment, my task was to predict whether a given blight ticket will be paid on time. I have obtained an AUC score of 0.809 on the test dataset, using Gradient Boosted Trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 1\n",
    "----\n",
    "\n",
    "For this assignment, we will be using the Breast Cancer Wisconsin (Diagnostic) Database to create a classifier that can help diagnose patients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore')\n",
    "%matplotlib notebook\n",
    "\n",
    "\n",
    "cancer = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object returned by `load_breast_cancer()` is a scikit-learn Bunch object, which is similar to a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many features does the breast cancer dataset have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n"
     ]
    }
   ],
   "source": [
    "print(len(cancer['feature_names']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the sklearn.dataset cancer to a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cancerdf = pd.concat([pd.DataFrame(cancer.data), pd.DataFrame(cancer.target)], axis=1)\n",
    "cancerdf.columns = np.append(cancer.feature_names,'target')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.8</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>0.2776</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.1471</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.6</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.1189</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38           122.8     1001.0           0.1184   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0            0.2776          0.3001               0.1471         0.2419   \n",
       "\n",
       "   mean fractal dimension   ...    worst texture  worst perimeter  worst area  \\\n",
       "0                 0.07871   ...            17.33            184.6      2019.0   \n",
       "\n",
       "   worst smoothness  worst compactness  worst concavity  worst concave points  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "\n",
       "   worst symmetry  worst fractal dimension  target  \n",
       "0          0.4601                   0.1189       0  \n",
       "\n",
       "[1 rows x 31 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancerdf.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the class distribution? (i.e. how many instances of malignant and how many benign?)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1    357\n",
      "0    212\n",
      "Name: target, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(cancerdf['target'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the DataFrame into X (the data) and y (the labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = cancerdf.ix[:,:-1]\n",
    "y = cancerdf.ix[:,-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split X and y into training and test sets (X_train, X_test, y_train, and y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Fit a k-nearest neighbors (knn) classifier with X_train, y_train and using one nearest neighbor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict the class label using the mean value for each feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "means = cancerdf.mean()[:-1].values.reshape(1, -1)\n",
    "pred = knn.predict(means)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict the class labels for the test set X_test.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "preds = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy of the knn classifier using `X_test` and `y_test`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "score = knn.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the test dataset:  0.916083916084\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy on the test dataset: ', score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the different predicition scores between training and test sets, as well as malignant and benign cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Find the training and testing accuracies by target value (i.e. malignant, benign)\n",
    "mal_train_X = X_train[y_train==0]\n",
    "mal_train_y = y_train[y_train==0]\n",
    "ben_train_X = X_train[y_train==1]\n",
    "ben_train_y = y_train[y_train==1]\n",
    "\n",
    "mal_test_X = X_test[y_test==0]\n",
    "mal_test_y = y_test[y_test==0]\n",
    "ben_test_X = X_test[y_test==1]\n",
    "ben_test_y = y_test[y_test==1]\n",
    "\n",
    "scores = [knn.score(mal_train_X, mal_train_y), knn.score(ben_train_X, ben_train_y), \n",
    "          knn.score(mal_test_X, mal_test_y), knn.score(ben_test_X, ben_test_y)]\n",
    "\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "# Plot the scores as a bar chart\n",
    "bars = plt.bar(np.arange(4), scores, color=['#4c72b0','#4c72b0','#55a868','#55a868'])\n",
    "\n",
    "# directly label the score onto the bars\n",
    "for bar in bars:\n",
    "    height = bar.get_height()\n",
    "    plt.gca().text(bar.get_x() + bar.get_width()/2, height*.90, '{0:.{1}f}'.format(height, 2), \n",
    "                 ha='center', color='w', fontsize=11)\n",
    "\n",
    "# remove all the ticks (both axes), and tick labels on the Y axis\n",
    "plt.tick_params(top='off', bottom='off', left='off', right='off', labelleft='off', labelbottom='on')\n",
    "\n",
    "# remove the frame of the chart\n",
    "for spine in plt.gca().spines.values():\n",
    "    spine.set_visible(False)\n",
    "\n",
    "plt.xticks([0,1,2,3], ['Malignant\\nTraining', 'Benign\\nTraining', 'Malignant\\nTest', 'Benign\\nTest'], alpha=0.8);\n",
    "plt.title('Training and Test Accuracies for Malignant and Benign Cells', alpha=0.8);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 2\n",
    "----\n",
    "This part of the notebook will explore the relationship between model complexity and generalization performance, by adjusting key parameters of various supervised learning models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Polynomial regression\n",
    "----\n",
    "The code below implements a function that fits a polynomial LinearRegression model on the training data X_train for degrees 1, 3, 6, and 9. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "np.random.seed(0)\n",
    "n = 15\n",
    "x = np.linspace(0,10,n) + np.random.randn(n)/5\n",
    "y = np.sin(x)+x/6 + np.random.randn(n)/10\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(x, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAEzCAYAAAAPYWahAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd8W/d97//XwQbBCe69t6y9JWpYnpKHvO3EI3VW4zapmzS9N7e37f016U3a25XWTdPYieOtyJItb8nW3ntzilPcA+ACSIyDc35/QIJMa5ESOPV9Ph58mCa+OOcLSiLf+I7PV1JVFUEQBEEQBGFsaMa7A4IgCIIgCLcSEb4EQRAEQRDGkAhfgiAIgiAIY0iEL0EQBEEQhDEkwpcgCIIgCMIYEuFLEARBEARhDN10+JIkKVWSpB2SJJVLklQqSdKfXaHNCkmSeiVJOnnh429u9r6CIAiCIAiTkS4I15CBH6mqelySpDDgmCRJX6iqWvaVdntUVb0vCPcTBEEQBEGYtG565EtV1VZVVY9f+LwfKAeSb/a6giAIgiAIU1FQ13xJkpQBzAIOXeHhRZIknZIk6TNJkoqDeV9BEARBEITJIhjTjgBIkhQKbAReVFW17ysPHwfSVVV1SJK0GtgE5F7hGt8BvgNgsVjmFBQUBKt7N6yvqxPZ7SIsOha9yTTe3REEQRAEYQI6duxYl6qqscNpKwXjbEdJkvTAx8AWVVX/ZRjt64G5qqp2Xa3N3Llz1aNHj950327Wztdfpqm8lOXPfJPUotvGuzuCIAiCIExAkiQdU1V17nDaBmO3owT8Fii/WvCSJCnhQjskSZp/4b62m733WNDq9AD4vN5x7okgCIIgCFNBMKYdlwDPAGckSTp54Wv/C0gDUFX118CjwPckSZKBQeBJNRhDbmNAq78QvmQRvgRBEARBuHk3Hb5UVd0LSNdp8xLw0s3eazyIkS9BEARBEIJJVLi/jsDIlwhfgiAIgiAEgQhf13ExfMli2lEQBEEQhCAQ4es6Lk07yuPcE0EQBEEQpgIRvq5DJ6YdBUEQBEEIoqAVWZ2qxG5HQRAEYaQabE5e3lPLphMtON0yFqOOtbOS+HZJFunRlvHunjDORPi6DrHbURAEQRiJHZUdvPDmcbw+BVnxV1VyuGXWHW5k47FmfvX0bFbmx41zL4XxJKYdr0OMfAmCIAjD1WBz8sKbxxn0+gLB6yJZURn0+njhzeM02Jzj1ENhIhDh6zoujnzJYuRLEARBuI6X99Ti9SnXbOP1Kbyyp26MeiRMRGLa8TpEnS9BEAThShx2G/22LjyuQbwuF17XIOXbj5PnU3BqLfTrQnFoQ/FoDCBdqkUuKyrvn2jmp2unjWPvhfEkwtd16MS0oyAIggAMOvpprzlHW3UVrTVVOLvtl7XJ6e7l4mSjdOFDkfQM6MJpMybRZE6mRxeKyy2jqiqSdM0DYoQpSoSv69Dq/N8iMfIlCIJw61EVhcayM5Tt3kFXY/2QxwwmM1GJyejNZkCPewC89k6MsoLJ50CvOND6HICXaJ+dVLedeX1nUTSR+AyprP+Zgdj0FKxJFv9HogVLpFEEsluACF/XIaYdBUEQbj0+2UvtiaOU7dpOv60T8K8Bjs3IIjEnj9iMbFzOUNpq++ls6MPZ6wEgNSKDbqcHGfACbhRcuND4OglzN2P2NqNVejB7emmvKaOnNQVL9GwM5gQAjCE6UgqiSJ8WQ3SyRQSxKUqEr+sQux0FQRBuHYrio3L/Hsp2b2ewvw8AS5SVopLbyZ4zH5dTpfZkJwc3deJydgaeZzBriU0NIyU6nh9/Xk63T2ZQAlUC0AIJQAIadSapPhs/mamlu/IsLmcXgz2bUTyJmMJm4HLGU3O8k5rjnYRGGUmfFk1acTRhVtN4fDuEUSLC13WIOl+CIAi3hp62VvZveBt7cyMAkQlJTFu+itTbZtBe62D/xjra6voC7SNizWTcFk1cRjiRcSFIGv8o1c+STLzw5nG0X6rzBaDTSOi1Bv7P02tYmR+He8BJxb7dVOzfhdfVhce5jfDYFKJTS7C3GXF0uynd00LpnhaS86MoLkkiMi5kbL8pwqgQ4es6xMiXIAjC1KYoPkp3bePMti0oPh+WyCjm3f8IyYXF2Fuc7Hr7HF2NDgC0OomUAivZs+OuOi24Mj+OzS+W8MqeOt4/0YzTI2Mx6HhoVjLfKskMVLg3hliYcee9FJasoPLAHir27qSvs4m+znXkLVhCcuEymqocNJXbaa7spqWqm5RCK8VLkwiPMY/p90gILklV1eu3Ggdz585Vjx49Ot7dwOtx84e//R9o9Qae+rt/HO/uCIIgCEHU097GgXffwnZhtCt3/mJm3/sArgE4s7OJpopuwL8Wq2BRIpnTYzCYR2fcwut2cWb755Tv2YGqqpjDI5h3/8PEpBdQcaCN2hOdKD4VSYK04mhuW5FCSLhhVPoijJwkScdUVZ07nLZi5Os6dIFpRw+qoiBpRF1aQRCEqaD+1HH2b3gbRZYJiYhk0SNPEZOew9ldzVQf60BVVLQ6ibz5CRQsSkRv1I5qf/RGE7PvfYDMmXM49P56uhob2P3Wq6QUFjN/7eMULEykbF8Ldae6aDhro7mqmxmrUsmaGSsW5k8yInxdh6TRoDMYkD0eZK8HvVEsehQEQZjMVFXl7M4vOPX5pwBkzZ7PvPsfprdL5ovfluLodiNJkDkjhuKS5DEfXYpKTObuP/4zzh3ez4nNH9NUXkpnwz+y+LGvM/feIgoXJXJi63laqno49lkDjWV25q7OJDTKOKb9FG6cCF/DoDMYkT0evG63CF+CIAiTmOLzcWjTemqOHgIk5qx+gNyFyyjb20rlwVZUFSLjzMy7L5OoBMu49VPSaMhbuJSUots4sOFtWs9VsuO131C4dAUz776PJY/k0Fhu58Tn5+lo6OfzV84ybXkyuXPjAwv/hYlLhK9h0BtNuBz9yB73eHdFEARBuEGewQF2v/V72mqq0Or0LHniGcJjc9j+Wjk9HYNIEhQsSqS4JAmtbmIsMQkJj+D2b3yXsj07OLnlY8r37qSjroYlTz5LWlEscenhnPjiPI1ldk5ubaS1ppeFa7MxjtK6NCE4JsbfrglOZ/APOXtdInwJgiBMRgN9vWz59b/TVlOFKTSMu777fRQlia2/L6OnY5DQKCMrnylg+sqUCRO8LpI0GoqXr+Ku7/4AS2QUtuZGPv2Pf+L82VOYLHoWrc1myaM5GEN0tNf1sfXVMrrbnOPdbeEaJtbfsAlKb/JPNYqRL0EQhMnH5XCw7ZVf0dvRRkRcAnd/989oqpQ4/HEdik8la2Ysd36zmJiUsPHu6jXFpmey5gc/Jm3aDGSPm91vvcrJLZ+gKgrJeVHc+Xwx1kQLzh43218vp/5M13h3WbgKEb6GQW/wL2L0ul3j3BNBEARhJNwDA2z73X/R29lORHwiK577Hie22ak81IakkZhzbzpzV2egN4zuTsZgMZhDKPnaN5iz+kEkSeLszi/Y+cZv8bgGCQk3sPKZAjJnxOCTVQ5/VMfxLQ34fMp4d1v4CjEpPAw6oz98yW4x8iUIgjBZeF0utr/6a7pbmwmLjmXBQ3/E3vXn6be7MIboWPxwDrFpE3u060okSaKwZCWRiUnsefs1mitKee9f/x9laSt4r2oQp1tmmqrndkx4DrXRZ3Ox5NGcSRMwbwVi5GsYLo18ifAlCIIwGXg9bna89htsTeexRFmZtfo59m1oot/uIjLOzB3fKJqUwevLEnPyufdPf4g7JIqT5XU4P3+T0N4mVOCM5GWd6uRMZz81FTZ2vlWByylOapkoRPgahsDIl1jzJQiCMOH5ZC+7Xv8tHfW1hIRHMP3OZznycTtet4/k/ChWPluIJXJq1MSyq2b+1TWb86ZktIqXZbZ95DqqAWjXKHxidlPWM0B7o4Mdb1Tg7BG/xyYCEb6GQW8UI1+CIAiTgaqqHHzvD4FdjcUrv86Jz7vwyQpZM2NY/FD2lJp+e3lPLS5Vy/6ohZSGFQIqs3pPMrvnBJKq4NDAlhAPbapMv93F9jfK6e0cGO9u3/JE+BoGnZh2FARBmBRKd22j7sRRdAYD+Yuf4PSOXhSfSt78eObcmzHlCpBuOtGCrKggSZSGF3Moah6KpCHHWcMy2z70ihcHKm/KDmLTwhjs97LjjQq6mhzj3fVbmghfwyBKTQiCIEx8jWVnOLnlEwDSZ6yh/MAgqgrFJUnMWJU6Jc8/dLrlIf/fEJLOjpjluDVG4t3trOragUV20ivLLHsyj+S8SDwuH7vXVWJrFgFsvIjwNQyBIqti5EsQBGFCsrc0s+8PbwAqibklNFaEADBjVSrFJclTMngBWIyXFy2wGaLZGns7ffpwwr193Nm5jRSlG61Ow6KHc0grsiJ7FHavq8LeKoqxjoebDl+SJKVKkrRDkqRySZJKJUn6syu0kSRJ+ndJkqolSTotSdLsm73vWLp4nqPsEXW+BEEQJppBRz+73ngF2eMhKrGIrtZUAObcm07+goRx7t3oWjsrCd0VplKdOgvbYlbSZorHqHq4r38/tSeOoNFIzH8gi5T8KLxu/whYT7tYAzbWgjHyJQM/UlW1EFgI/IkkSUVfaXMvkHvh4zvAfwXhvmNGlJoQBEGYmHyyl11v/BZnTzfmsASc/dORJImZd6SSPStuvLs36r5dkoVee+Vf5V6Nnj3WJdSH5RJr0bF//Vuc/PxTJFQWrM0iKTcSz6CPXe9U0ts5OMY9v7XddPhSVbVVVdXjFz7vB8qB5K80exB4XfU7CERKkpR4s/ceK6LIqiAIwsR07JMP6Dpfj0ZnQVYXg6SluCSJvPlTe8TrovRoC796ejZmvfayETCdRsJk0PPHP/gWSx5+zF8Rf8fn7F33OqriY9FD2SRkR+AekNn1dgV9NhHAxkpQ13xJkpQBzAIOfeWhZKDxS//fxOUBbcISpSYEQRAmnoYzJ6k6uBefLCHpliBJZvLmx1O0NGm8uzamVubHsfnFEp6an0aoUYckQahRx1Pz09j8Ygkr8+PIX1TCiue+g95oouHMST7/73/H5ehlycM5xGeE43LK7H6nioE+z3i/nFtC0I4XkiQpFNgIvKiqat9XH77CU9QrXOM7+KclSUtLC1bXbtrFNV9esdtREARhQui3d3Fw4zpkr4JGPwetPpasmTFTdlfj9aRHW/jp2mn8dO20q7ZJzi/k7u+9yI7XfoO9uZHP/vNfKPnaN1jyaA671lVha3KwZ30VK58uwGASpw+OpqCMfEmSpMcfvN5SVfW9KzRpAlK/9P8pQMtXG6mq+htVVeeqqjo3NjY2GF0Liou7HcW0oyAIwvjzyTJ7334N98Agii8RY2ghqUVW5tyTcUsGr5GIjE9g9Z/+iITsPFyOfra+8p/UHj/AkkdzCIs20dsxyP6N1fhkcRj3aArGbkcJ+C1Qrqrqv1yl2YfAsxd2PS4EelVVbb3Ze4+VLx8vpKqXDdgJgiAIY+jklo/pajqPy6knxFpCXHo48+/LnHIFVEeLMcTC7c9/l8KlK1EVhcMfbODkZxtZ8kgmplA9HQ39HP64DlURv+9GSzDGFZcAzwBnJEk6eeFr/wtIA1BV9dfAp8BqoBoYAP4oCPcdMxqNFq1Oj0/2Ins9gd2PgiAIwthqKjtL2Z6dOHu8hMbeSURsBEseyUGrE2UrR0Kj0TJnzYNEJSVx8L0/UH30IN2tzcy5+wkOfdROY5mdkDADM1alXv9iwojddPhSVXUvV17T9eU2KvAnN3uv8aQzGv3hyyPClyAIwnhw9nSzf8PbDPZ5MIbOJtSazNLHczGYxfqkG5U1ax4RsQnsfut32Job2fvOS+Qvvp9zx41UHmrDHKa/ZXaOjiXxVmGYAjseXaLQqiAIwlhTFYUDG96mr6sXVUogNGa6f52S1TTeXZv0olNSWf39H5NSdBtet4uzO94lLOIMqiJzalsjLed6xruLU44IX8N0qcq9WHQvCIIw1qoO7aOxtAKXU0N4/Arm359NbGrYeHdryjCGhLD86eeZ98AjaLRaus6fQh7cgtfVzcEPaujtFFXwg0mEr2ES5zsKgiCMj76uDo58uAlHj5vwuBKmr8wmvTh6vLs15UiSRP6iEu554YeERceiKj30d35IT+tR9vyhEpfTO95dnDLERPkwBUa+RPgSBEEYNQ02Jy/vqWXTiRacbhmLQcvX3AcI6+rHFJZDztzZt1wR1bFmTUpm9fd/xNGP36f6yCH6bcc5f6aW7a/dy93fXYb2KscZCcMnwtcw6QJV7sWaL0EQhNGwo7KDF948jtenIF8oc5BiK8PX28gAZsLyFzPv/swpUctLGRxEttnw9fai9Pbiu/jR148yOIA6MIjicvk/d7lRfT6QZVRFAZ8PVVGQNBrQaZG0OiSdFnQ6NOYQNCFf+rBY0EZGoI2MQhsVhS4qEq3VisZsvmb/9EYTix55iowZs9n/7jpaq5upOfw2HzprWPODr2O4MCBxWVg26lg7K4lvl2SRHm0Zi2/lpCTC1zBd3OEoe8TRC4IgCMHWYHPywpvHGfT6Al+L8PYyv7cUgwr9lvls7HGwst81KX6pq4qC3NWF3NKCt6UFb1s7ckcHcmcnclcXisMxrv3ThIaiT0hAl5CAPtH/X0NKCvrUVDTGSzv6E3PyefBHP+HAex9wcstWGksP8oe/rWHJ4w9Rb07lT94+OSQsO9wy6w43svFYM796ejYr88fmcHOX7OJc9zlKbaWU28uZlzCP+7LuG5N73wgRvoZJjHwJgiCMnpf31OL1XaqqrlF9rLAfwaKqeA057AyPw47CK3vqrnmEzniQu7vx1NfjaWjAU1+Pt7EJb0sL6jXerEt6PdqYaLQREWgjI9GGR/g/jwhHExKCZDKjCTGjMZuRTCYknQ40GiStFkmrBY3GPxrm86HKMqrsQ/V6UF0ulIEB/4dzAMXpwNfTi6+nG9lux9fdg89uR3E4cFdX466uHtoxjQZ9QgKGjHQMGRkYMrMw5uVR8uRjWJMK2P3W2/S0dbL1td9z0mkgInQabcZ4+NJopKyoyIqPF948zuYXS0YlLA/Kg1TaKym3l1NmK6Ompwafeim4m7QmEb6mgosjX2LBvSAIQvBtOtESGD0BmNlXSaKnF1UTyvHw22jSKaDA+yeaxzV8+fr7cZ+rxn3uHO5z5/DU1uLr7b1iW21EBPrkZPTJSf4Rprg4dLGx6GJj0UREjNv0qaqq+Hp6kNva8La24W1rRW5rx3P+PN7mZv9IXUsLzv0HAs/Rp6YSn5/H3Nl3UHquGZvtGBFyD8vce+k0xnI6fBo2w9BNEF5f8MLygHeAyu5KymxllNnKqO2pReFLYR0NWRFZFEUXUWgtpMBacNP3HE0ifA3Tl48YEgRBEILL6ZYDn0d6+pjTV4EENITO57jp0gJvp0e+wrNHh6qqyK2tuMrK/B9VVcitbZe105jNGDIzMKT7R4v0aWnok5LRhk7M6VFJktBFRaGLisJUWDjkMdXjwdPUjKehHk99gz9g1tTgbWzE29hINNuItSxlQJqPVa2l31dFgtpOrLuTTmMMlZY8WkyJIEnIinrDYdnpdVJuKw+MbNX31l8WtnIic4aErRB9yE1/b8aKCF/DFCiyKka+BEEQgs5i1OFwy6Cq3GU/jg4VpyGbreFRQ85QsRhG79fWxbA1eOqUP2yVll02qiUZDBiyszDl5mLIycGYk4suLnZKbAIA/+szZmVizMoMfE31eHDX1eOuqsRVUUFBaSntrtnoDFnEOnxIjgPYwk1EeJwkDbbRbYikMjSXenM6zmEuk3Z4HIGgVWYro6GvAZVLI6FaSUteZF4gbOVb8zHrrr1pYCIT4WuYRJFVQRCE0bN2VhLrDjcyu6eWaK8NVTLxubUYz5cyjU4j8dCs5KDe19ffz+Cp0wyePoXr1Gnkrq4hj2sjIjAVF2EqKsKYX4AhLdW//uoWIhkMmPLzMOXnEXH//cQpCn/x52+z0BmCOaoYq6eXzNazOM0u+i0aYvS9pDjb6NMfpy4qn572OUTGDz2iqNfdS4W9gjJbGeW2chr6G4Y8rpN05ETlUGgtpDi6mNyoXEy6qXOawa31N+gmiCKrgiAIo+fbJVlsO3COWX1nADgbPos2vX5IG71Ww7dKMq/09GFTVRVvQwMDx44zcOwY7qoqUC+NsGjCwjBPn45pWjHm4mJ0SUlTZlQrWCSNhuKVM9iyr5klAzo6M1ZTk1RIRlcp07pqiNS56Ld4Cdc7SfN2s+lvKrDmZqOZnkZbjIfyvkqaHE1DrqnX6APTiEXRReRG5WLUTt1zlEX4GiZRZFUQBGH0pEaF8Ky3gn5VpseQxN7QxMBjOo2EXqvhV0/PvqGdc6os4yorw3noEANHj+Lrsl16UKfFVFiIefoMzDNnYMjI8NfPEq7p2yVZbDzWzFlZZppHR7Ipk015yazLV8juaWaGrYzpvRWYVQ/2zkGaulrxHtXgCtGiZuux5JjJTCuiMLrQH7Yic9Fr9de/8RQhwtcwiVITgiAIo2f329vw2usIMZtxLl9NaHU/To+MxaDjoVnJfKskc0TBS/V4GDx1CuehwwwcOTKkrpY2IgLznNmEzJmLecb06xYcFS6XHm3hV0/P5oU3jhOjKCTIKsvlAXYkHKQ5tZlWfS/lUaEUdHjIKe8n7ryPAcWMS9ZgPe1DV6chsVBLwR1ZpEbkodXeWnHk1nq1N0EUWRUEQRgdjRVtlO78CAlY8OCDzLt/Mf/fDVxH9XoZPHMG5969OA8fRh289GZZn5JCyIL5WObPx5CVJUa3blLHQAeakHKeW1PF7qozRJXOJ85rYa7bQkPSAMkRMUyPK6RodhGFXyskw5iI9+gJmj/9hLqaKtr6+mg+dIDWUycIiYtnxoOPkre4BI1WO94vbUyI8DVMYuRLEAQh+NwDXrb97g/45AHiMjKZu+aOET1fVRRcpWU49+3FeeDgkBEuQ2YmlkULCVmwAENKSrC7fstQVZX2gXbKbeWU2f0L5DsHOwOPh0VD39yzpJxZRbJcwvTcx5g3rwitZmiQMi5fTv7y5WSeP4/9s0+p2bWDxsFBHOcb2P9fv+Tspg3MefJpMuYumPLr7ET4Giax5ksQBCG4VFVl97qD9HWUojfquPu73xj2iJSnqRnHrp04du3CZ7MHvq5PTSW0ZCmWxYvRJyZe4wrC1aiqSpuzzV/2we4v/WB32Ye0segtFFgL/AvkrUVkRGRQm93F8S0NtOz24MhyExF75bpbhrQ0Er77x8Q9+xzT9uyh+v2NVHW10tvYwI5//jnW5FTmP/M8STNnj8XLHRcifA1TYLejx42qqlM+lQuCIIy2ulMdVB/+DEmCWXffSVRi0jXb+xwOnHv24Ni5E3d1TeDrurg4LCVLCV26FENa2mh3e8pRVZUWZwvltnL/2Yi2crrd3UPahOnDKIwupNDqXyCfFp6GRhoalLNnx2JrdtBw1sb+jTXc8UdF6I1Xn0bUmM1E3HUXs++4g/xjRyl96w2qmhuwNTbw2c//D3l5xSz8wQ/Rx8aOyuseTyJ8DZNWp0Oj1aL4fCg+Ga3u1tmVIQiCEGyObhf71m/B6+rEmhTLnDWrr9hOVRRcZ8/Sv207A4cOoXq9gP8Xt2XJYkKXL8dYUCDWcI2Aqqo0OZoCQavcVk6vZ2gx2XBDeGAnYpG1iJSwlMvC1ldJksSce9Pp6Rigt2OQIx/Xsejh7OsOVkgaDaHz5rNg3nyKzp7l5Ou/5Vx9NVWVZ+n50+8w/877iH7kYXRRUTf92icKEb5GQG804h4YwOtyow0V4UsQBOFGKIrKvg1n6W07jMGso+SpxwNLOy6S7XYcO3bQv3UbckeH/4uShHnGDEJvv52Q+fPQXJiREK5NURUa+xsDBU3L7GX0e/qHtIkwRARqbBVGF5ISmnJDMzw6vZbFD+ew9dUymiq7qTrcTv6ChOs/8YKwadMo+cd/JX3vLna9+hs6+vvYteUDZmz9gtjVa4h4+CG0oaEj7tdEI8LXCOgM/vDlr3I/+f/wBUEQxkP5vhbOn9kBkofsWTNJmzYDuDTK1bdlCwOHj4DiP8tPFxND6O23E7pyBfq4uPHs+qSgqAr1ffX+oGUro8JegcPrGNImyhRFkbUoELgSLYlBW04TZjUx//5M9m2o5vT2RqyJFmLTwkZ0jbSly3kgL59tv/olvfV1HHEOMP2D9+nfto3Ixx4j/O67kPSTdxBEhK8R0F14ZyZ2PAqCINwYW7ODk58fx9VXRURcKAsefgzF6fSPcn3+Od6WVn9DjYaQhQsIu+MOzDNmiGnFa/ApPur76gPnIlbYKxiQB4a0iTZFB4JWUXQR8SHxo7p2OTkvioJFCVQcaOPAphrufL4Ic+jIRioj4hJY/eO/Ytcbv6O9spyT3d3c1t+H8uqr9G/ZTNQzzxAyb96kXIMtwtcIiMO1BUEQbpzs9XHog2p62/ditOiZNmcung3vYd+1C/VCDUVttJWwO+8kbNUqdFbrOPd4YpIVmbreOsrt5ZR2lVJhr8DlGzooEGeOCyyQL44pJtY89od/T1uegq3FSWdDPwffr2H51/LRaEcWok2WUO741vc4uHEddSePUZ0Ac/tlvC2tdPzDP2IqKiL6m89jyMgYnRcxSkT4GoGL4Uscri0IgjByZ3c10153DNXbRZjPTfj69+jHHwjMM6YTds89hMyZg3SLFNocLq/ipbanNjCyVdVddVnYig+JpzDafwh1obWQ2JDx3yGo0UgsWpvNF78tpbPRwZmdzcxYlTri62h1ehY99jX67Ta6ztdzfsFMZsYk0bP+XVxlZTT/+C+JuG8NkY8/PmlOKxDhawR0BjHyJQiCcCPaqzop3XyK/tb9mL02cnwadKYQwlauIOyeezGkJI93FycMr89LdU+1f4G8vZyq7ircvqG/dxItif7F8RdKP0Sbo8ept9dmsuhZ9HA2O96spPJQG9ZkC6kFIx/R1Gi0LH3iGT7+5T/SWH6WpIeLyXrpJXrWvUPfls/p/fAjHPv2Ef3884QsmPhFWkX4GgEx7SgIgjAyclcX9o82s2efTL+7Cr3cT4LZTN6jXyfs9pVoLCM/KHtqrKMSAAAgAElEQVSq8fq8VPVUBUa2znWfw6t4h7RJDk0OlH0ojC4kyjR5yi7EpIQxY1UKJ79o5MjHdUTEmgmPHvkIVag1mgUPPc6+P7zB0Y/eJy49i+hvfYvQlSux/eY3uKtr6Ph//4R5zmyiv/kt9PETd3OGCF8jcHHkSxYL7gVBmIrstbD/JTi9HjwOMITC9Mdh8Z+CNWtEl3LX1dH74Yc49+2nxjCdfp0VWWkiIiGK5f/77wiPjR+lFzHxuX1uquxVlNv9uxHP9ZxDVuQhbVLDUi+VfrAWEmGMGKfeBkfu3HhszU4ay+zs31jNqm8UoTeMfHo5c+YcWqrKqTtxlL3rXueeF17EmJ1N4s9/Tv+WLXS/9TaDx47T6fglSf/370fhlQSHCF8jEBj5Emu+BEGYas59AeufBZ8XLo66ePrh+Otw6h14/HXIvfOal1BVFdfZUno3bWLw5EkAegwJtFun43LtJSwmkml33nXLBS+X7KKyuzJQ+qGmpwZZvRS2JCTSw9MDo1qF0YWEG8LHscfBJ0kSc1dn0NsxSF/XIEc+Gl4B1iuZ/8CjdDbU0d3azMktnzBnzVokjYbwe+8lZOFC7K+9Rvi9Vy7aO1GI8DUCFw/XFuc7CoIwpdhr/cHLO3D5Y8qFMLb+WfjeviuOgKmKwsDhw/S+/37g2B/JaMS88k7O9hcjt1eiUx2ERscwbcXIDs6ejAa8A1R1VwUqyNf21uJTfYHHJSQyIzID67UKrYWEGqZ+7Ui9QcuSRy8VYK040Erh4msfKXXF65hMLH3yWbb8+peU791JUn4hiTn5AOiiooh78cVgdz3oRPgaAX2gzpcIX4IgTCH7X/KPeF2LzwsH/hPW/HPgS6os49izh973N+FtbgZAEx5G+OrVhN9zDyf2dOE81oyr/xgh4Xpm3/vAZZXspwKn10mFvSIwslXXW4eCEnhcg4bsiOxA9fgCawEW/a251i3MamLBA1nsffccZ3c1E5VgISFr5FOqManpTF91D6e++JRTX3wWCF+TRVDClyRJvwPuAzpUVZ12hcdXAB8AdRe+9J6qqn8XjHuPpcCaLzHtKAjCVHJ6/aWpxqtRvHD6D7Dmn1E8Hhzbt9O76QPkzk7AX4U+Yu2DhN5+Oxqjkfa6PmpPdjHQfRxTiI+4jBwyZswegxcz+hweBxX2Cv/Ilr2cht6GIWFLK2nJjcgNlH7It+Zj1k2OEghjISk3kuKSJEr3tHDwgxru/KNiLJHGEV+nYMkyyvfsoOt8PV3n64lJywh+Z0dJsEa+fg+8BLx+jTZ7VFW9L0j3Gxdit6MgCFOSx3H9NoAy6KD/o4/p3bQJX08PAPqkJCIefojQkhIknf9Xitfj4+hn9cjubhS5CmOonnkPPDLht/9fTZ+nL3AAdZm9jPN951FRA4/rJB25kbmBka28qDwRtq6jaEkS3a0DtFT3sG9jNbc/W4BOP7IF+HqjiZz5iynbvY3yfbsoudXCl6qquyVJygjGtSYyEb4EQZiSDKH+xfVXocgSfY0m+s6H4Sv9vf8pGRlEPvqIv6bSV47+OburGUe3C7fjMMYQHTnzFmFNmjx1vHrdvYEaW2W2Mhr7G4c8rtP4w9bFNVt51jyM2pGP3NzKJI3E/Acy2fb7cnraBzj8UR2L1mYjaUYW0AsWl1C+dwfnz5zEee8DWCInRwmOsVzztUiSpFNAC/AXqqqWjuG9g0JMOwqCMCVNf9y/q/ErU4+KLNF33kRvfQiKrIEQK8acbCIfewzznDlXHMnqanJQfbQdz0ADktSBMSSUGXdN7J1n3a7uwKhWma2MZkfzkMf1Gj25UbmB0g+5kbkYtCM7p1C4nMGkY8mjOWx7rZymim5K97YwbdnIQnpIRCTpt82i/tQxKvfvZvbqB0ept8E1VuHrOJCuqqpDkqTVwCYg96uNJEn6DvAdgLS0tDHq2vCJkS9BEKakxX/qLydxIXxdHOnqrQtB8foDltGqEPmXf4152ZqrTh/6ZIWjn9Sh+Hyo8kl0eg3T77gHk2Vi7eSzDdqGjGy1OluHPG7QGMi35gd2IuZE5qDX6sept1NbeIyZhWuz2bu+irK9LYRHm0grHlm1/sKly6k/dYxzhw9w26q7J8WmjjEJX6qq9n3p808lSfqVJEkxqqp2faXdb4DfAMydO1dlgrn4ByqKrAqCMKVYs+Dx11Hefpb+eg09tUYUz4XQFeUjKk/G9MLvkPLuuuZlyva20GdzociVSJKTiNh48hYsGYtXcE1dg12U2coCpR/aB9qHPG7Smsi35gemEbMis9BrRNgaK4nZEcy4I9VfAf+TOixRRqKThh/Yo1PSiMvIoqO+luqjhyhcsnwUexscYxK+JElKANpVVVUlSZoPaADbWNw7mMS0Y/Coqkp7zTlaa6qQJA1anQ6NVotWp8MQEkJq0W2T4t2LIEwFqsdDX6WH3uqV+FprQO7GGOkmskDFvOpRpGFUuO9uc1JxoBXFN4DsPoMkwZz71qIZ40OyVVWlc7CT0i7/TsRyWzkdgx1D2pi0JgqjCwPH9WREZKDTiMpL4yl3bjx9XS5qT3Syb0M1d3yjiJDw4U/tFi5dQUd9LRX7dlGwqOSydYgTTbBKTbwDrABiJElqAv4W0AOoqvpr4FHge5IkycAg8KSqqhNuZOt69CYx7XizLoau09s201Ffe9V2xpAQCpauoGDRMvQmEcIEYTSoskz/9u30bNiAz2YHwDhnBZFPPIl51sxh705UFJUjn9SjqqA3lCEPyCQXFJOUVzia3Qcu/EwZaA+MapXZyrC5hr63D9GFUGAtCKzZygjPQKsZ21AoXJskScy+K41+u4vOhn72vnuO258pQDfMI4hSCqcRao3BYe+isewMadNmjHKPb06wdjs+dZ3HX8JfimJS02h1SBoNis+HT5bR6sQ7peFSVZX22mp/6KrzV8A2mEPImbcQncGAIvu/pz7Zi725ia7Gek59/inle3ZSsGQZBYuXYTCHjPOrEISpQVUUHLt307P+XeR2/xScIT2dqKeexDx37ohLQlQdbqOnfQCdvgenrRKtVsucNaOz8FlVVVqdrYFDqMvsZXS7uoe0CdWHDglb6eHpaKSJPRIigEarYfHDOWz7fRk97QPsf7+GJY/moNVe/89O0mgoWLKMox+9R/neXbdG+LpVSJKE3mDE4xpE9nhE+BomVVHYv+Ft6k4cBfyhq7Dk6qNaqqrSVlPFmW1b6Kiv5fTWzZTv3cnCh58k/baZY919QZgyVFVl4PARut95B2+jv3yCPimJyCefwLJo0Q1N1Ti63ZTubkFVVST1JJLkL34ZHhMXtD43OZr8dbYuLJDvcfcMaRNmCKPIWhSos5UalirC1iRlNOsoeSKP7a+X01bTy5GP61hwf9awSlBkz5nPqS8+pbOhlq7GBmJS08egxzdGpIcR0hn94cvrdmEMESMx16OqKoc/2EDdiaPoDEaKV6y67lSiJEkk5uSTmJMfGC1rr61mz9uvMXBf76RYTCkIE83g6dN0v/U27upqAHSxsUQ+/jihy5ch3eC6LFVVOba5Hp+sEG7txHa+GaMllGm3X3th/rUoqkJTf1Og7EO5rZw+T9+QNhGGiEtrtqKLSAlNmbQFXIXLhVlNlDyex863Kzhfasdo1jPzztTr/hnrjSZy5y+mbPd2yvfupOSp58aoxyMnwtcIXdrxKNZ9Dcepzz/l3OH9aHQ6Vj73beKzckb0/PisHO7I/BNKd27l5OefcOzj9xns7WHWPfdP+AWVgjARuKursb/1Fq7TZwDQRkQQ+dijhK1ahWS4uVpVDWdttNf1oTMqOO2HAZh59xoMpuFXd1dUhYa+hkDQqrBX0O8dWvA1yhgVCFuF1kKSQ5NF2JrirEkWljySy571VZw72o7RoqNoyfUP4c5fVEJHfS2pxdPHoJc3ToSvEdJd+GHlFTser6t8zw7O7vwCSZJY9tQ3Rhy8LpIkiWkr7yQkIpIDG9+hbM8OBvp6WfToU2h1Yju4IFyJt6WF7nfewbn/AACakBAi1j5I+Jo1aIKwicXl9HJyq3/qMjKmiZaKXqISk8mZs+Caz/MpPur76gN1tirsFTi9ziFtrCZrYCdiUXQRCZYEEbZuQfGZ4Sx4MIuD79dwdlczxhAd2bOuPZ1tiYzinu+9OEY9vHEifI2QGPkanuqjhzj26QcALHrsa6QUXXbe+ohlzZ6HKTSM3W+9Sv2p47gc/Sx7+vkRvcsWhKlO7u6mZ/279G/dCoqCpNcTvno1EQ8/hDY0eMVOT21rxDMoE5Uo0V5zEIC59z102Yi0T/FR11sXmEassFcwKA8OaRNrjvWPbF0IW3EhcSJsCQCkFljx3C1zbHMDxzc3oNNrSZ82siKsE5EIXyOkC1S5F4VWr6ax7AyH3lsHwJz7HiJr1rygXTspr4C7vvt9tr/637TVnGP3m69y+/PfRSO2jQu3OGVggN4PPqD3o49R3W7QaAhddTtRTzyBLjq4v6zaantpOGtDq5PQac7i83pJLZ5OfFYOsiJT01MTqLFVYa/A5Rv68zIuJI6i6CKKo4sptBYSGxIb1P4JU0v27DjcgzJndzVz+KNaZK/vuiNgE50IXyN08Ygh2eMZ555MTIOOfg68+zaqqnLb7XeNyuJ4a1IKd3/vz9jyX7+kraaKY598wLz7Hw76fQRhMlC9Xvq3bqXn3Xfx9foXpofMm0fU01/HkJIS9PvJXh/HNjcAkFoAlftP4EGmo9DAzw7+jKruKty+oTMDiZbEQPX4ougios2Tf+RCGFtFS5KQJIkzO5s49lkDsttH/sLE8e7WDRPha4QuVrm/FUe+FJ+CCtesuXJyy8d4XIMk5hYw/Y57R60vYdYYlj/9PF/85iUq9+8mMiGR3HmLRu1+gjDRqKrKwIED2N96G7mtDQBjfj7WZ5/BVFAwavc9s7sRW1cP7lAH24+vx9vXTUeWlo7WjwJtkkOTA2GrMLoQq8k6av0Rbh2FixPRG7Uc39LAqe1NeD0KxSVJk3KKWoSvEZrqh2urikp32wBtdb047C5cDi+DTi8uhxf3gAyA3qjFYNZhDNFhMOsIjTQSkxqKVttDzdHDaDRa5t3/8Kj/g4hNz2T+Q49zcOM7HNm0gfCYOOIzs0f1noIwEbjKy7G/9jruc+cAf62uqKefJmT+vKD/u/P4PFR1V/nXa9XVYtiZCapKc9hmYtvtyEYwzkjnroRpgXVbkabIoPZBEC7KmROHzqDhyMd1lO1twev2MXNV6rDqgE0kInyNUOB8xykUvtyDMm21vbTV9NJW2xsIWV918We61+3D6fTS6XDTM+BBUUGDSvjgVgw4SS1aRHeHlpBIHzr96K7Fypm7gJ62Fir27WL3W69y75/8kNAo8S5bmJq8zc3Y33yLgcP+sg7aiAgin3zCXzYiSGcoumRXIGyV28up7qlGVmRQIe/UCgwqyKld5LZqMVriWfTIU0xfdHtQ7i0Iw5FxWww6g5aDm2o4d6SdgV4389ZkYjBPnkgzeXo6QQRGvqZAqQlHt4uKg23Un+5C8V06ajMkwkBCVgTWRAvmUD2mUD3mUAPGEP9fl22lbfzknVNoUNGbINInke2sx+zuRJbMdPVkceiDWvRG/66UrJmxRMaPXkHa2asfoLejndZzFex8/RXu/uMfiEO5hSnF19tL9/r19H/+hX8Ho9FIxIMPEvHA/WjMN7fbd1AepNJeGQhbNT01+FRf4HEJifTwdPK756LxxRGZGEp6dihlzdVEpSVz24IVN/nqBGHkUvKjWPp4Lgfeq6G5qoeejjIWrs0iOil4O3pHkwhfIzQVSk30dAxQcaCVxjI7quof0YrPCCchO4LE7AjCok1XnbposDn5/vpTDCo+0AAaaNd4yXafpFujUhoxDa3bzfOpUbhtbqqPdVB9rANrooXs2bGkT4tGM4xzukZCo9FS8tSzfPaf/0pPWwuHP9jAksefDuo9BGE8KB4PfR9/Qs97G1EHXaDREHbnHUQ+8QS6qKgbuuaAd4AKe0XgqJ7anloUlMDjGjRkR2RTGF1IobWQAmsBOo+Rz/77DF69j2lLozn6wR+AK5eWEISxkpAZwZ3PF3NwUw32VifbX69g+soU8ubHT/h1YCJ8jdBkLrLq7HVz8ovzNFf5z0WTNBIZt0VTsDCB8JjhvXt+eU8tXp8y5GvF/WUYFTedhmjOWlLRaWWOxmv40dpiak920nDWhr3Vif0TJ+X7W5m2PJnUAmtQ5+gN5hBWPvdtPvmPf6LuxFFSi26b8AerCsLVqIqCc+9eut96G7mrCwDzrFlYn3sWQ2rqiK7l8DiosFcEDqFu6G24LGzlROYEdiLmR+UToh86Un3g0xq8bh+JORG0V+/BJ3tJmzbjhgsnC0KwhEYZWflMAad3NHHuSDuntjXSeb6f+fdl0jrg5uU9tWw60YLTLWMx6lg7K4lvl2SRHm0Z136L8DVCk3HkS1VV6k93cXJrI163D61OInNGLPkLE7BEGEd0rU0nWpCVS1OU4d4+8hzVgMTxyFkgSciKyvsnmvnp2mnMvjudGben0lhhp3x/K/02Fwc31VKZ0MZtK1KIzwwP2juU8Ng4Zt97P0c+3Mih99cTm5GFOTQsKNcWhLHiKivD/tpruKtrADCkp2P9xnOYpw/vuJR+T38gbJXaSjnfdx6VS/9mtZKWvMi8wFE9+dZ8zLqrv/lqremlscyOTq8hvUhl91vH0eh0zL73gZt7oYIQJFqdhll3phGXFsaRT+poOdfD+pdP8/cdHXgVJfA7y+GWWXe4kY3HmvnV07NZmT9+tcJu6fDVt3kLxuwsjLm5w37OZCuyOujwcOzTBlqq/aNdyXmRzL4nHXPojZ3p5nQPXYw/q/ckoFJjyaJHf2mHk9NzqZ1WryHjthjSiqOpP9VF6d5mutsG2L2uiviMcGbfk06YNThrtPIWLKGx9AxtNVUcen89y59+fsIPPwsCgLe1FfsbbzJw6BAA2qgoor72FKErVlxzaq/X3RuYQiy3lXO+//yQx3WSjpyoHAqthRRHF5MblYtJN7x/b7LXx/Et/ppeRUsTKN257sLnKwm1ilpdwsSSnB9FZHwI29ZX8VJtC4P4LmsjKyqy4uOFN4+z+cWScRsBu2XDl2PvPmwvv4w2Koqkf/jFsCtAT6Yiq+fLbBzf0oBn0IfeqGXWXWmkT4u+qTBiMepwXAhgUZ5u4t0dyBodZ8KLh7YzXP5XS6ORyJoVS9o0K9VHO6g40Ep7fR+fv3KWactTyJsXf9NTkZJGw6JHn+Kjf/0FTWVnqDt5NKgV9gUh2HwOBz3vbqBv82cg+y4tpn/wgSuewdjj6glMIZbbymlyNA15XK/RkxOZQ3FMMUXWInKjcjFob+zNVvm+Vpw9biLjzOj0TdiaGzGHhVO8YtUNXU8QRpsl0sjhWJWWeh8oV2/n9Sm8sqeOn669+aPvbsQtG74sC+bTX1yMq7SU9l/8A4k//bthHTY7GYqsqorK6R1NVB7yF15MyIpg7uoMQsJv7Afwl62dlcS6w43IikqusxqA2pBMPJpL05c6jcRDs5Kveg2dXkvBokSyZsZycut56s/YOLWtkaYKO/PWZA57/dnVWCKjmHv/w/76Xx9uJCErl5AIUXdImFhUWaZvyxZ61r+L4nCAJBF6+0qinnoKnfVSuRTboC1wVE+ZrYwWZ8uQ6xg0BvKt+RRaCymMLiQ3Mhe99uYPnO/tHKTiYBuSBDNWJbL3nV8CMOue+8VuYmFC23SydcjymCv58vKY8XDLhi9Jryfux39By//8CZ7aWrpeeonYH/7wujt39BcC2kQtsuqTFQ5/XEdjmR1JIzHrzjSyZ8cGbert2yVZbDzWjNbrJG2wEYBzlqGFTfVaDd8qybzutQxmHfPvzyKl0Mqxz+qxNTv54nelFC9LJn9+wk2NgmXPmU9T2Wmayks5sPEdbv+jPxbTj8KEoKoqg0ePYn/9dbwtrQCYbpuG9bnnMGZm0jXYRVnT7sA0YttA25Dnm7Qm8qLyAgvksyKz0GtuPmwN6aOicmxzPaqikj07ltaqgwz29xGdkkbmzDlBvZcgBNtXl8dctZ1neO1Gwy0bvgC0YWHE/+R/0vo/f4LzwEH069cT9eST13xOYLfjBBz58rhk9m2sprOhH51Bw+JHckjIjAjqPdKjLfzq6dn823++iRaFZlMiTp2/ropOI6HXavjV07NHNI+elBNJzLencWpbI3Wnuji9vYn2uj4WPpiFMeTGfqlIksSCh56go/4XtJ6rpPrwAXIXLL6hawlCsLjr6rC/9hquM2cB0CclIj3+AFXpesrsmynbVkbHYMeQ55i0JgqsBYGjerIistBpRvdHd/2ZLroaHZgsOjKnh7D5v3YAorSEMDl8eXnMNdtdYXnMWLmlwxeAISWF2B/+kPaf/5yedzegT0kldOmSq7bX6Q1IkoQiyyiKD41mdCu4D9dAn4c9f6iit3MQU6ieksdziUoYnYWEy3OiaY620ew1cCgsH0ny/yV+aFYy3yrJvKEFjAaTjnlrMkkpsHL4o1ra6/r44tUyFj+UgzXpxl6HOSyc+Q8+yt51r3N884ekFN8mdj8K40Lu7qb7nXdwbN+B1+dhwADnlmWwK9dDZ+8rcPpSW7POHJhCLI4uJiM8A+0Y/pxxOb2c2u4f1Z5xRxpnd3yIIstkzpxDbPr1R7QFYbx9eXnM1Vxvecxou+XDF0DI7FlYn3sO+6uv0vXSS+jj4666A1KSJHQGI163C9ntxmAevcrtw9Vvd7HzrUoG+z2ERZtY9kQelsiRlZAYifNnToLLSVFeJn/54rNBnc5LzPYXzTvwXjW2Fifb3yhn1l1pZM28sanT9OmzqD1+hJaqco5/+iFLHv960PoqCNfjc7tp3PAWPe+9h3ugjwHFxanpoRxfYMVtagMPWPSWwMhWUXQRGeEZaKTxG106vaMJz6CP+IxwjCY758+eQqvTM+ue+8etT4IwEheXx8jK5bsdLxru8pjRIsLXBeFrVuNtaqT/i620//wXJP785+jjr1wDRGf0hy/vBAhfgw4Pu9dVMdjvITollKWP5WIc5fOtKvbvAaBg8bJRWUcVEm5gxdMFnNrWSPWxDo591oCt2cmcu9PR6kf2S0mSJOY98Agf/dsvqDtxhJx5C8Xh28KoUVWVZkcz5bYy2nd8jvXjAxh7BwGoz7JwqCQZJS6aGdGFgTpbaeFp4xq2vqzzfD/1p7vQ6iRm3pXCnjdfAqB4xR1i04owaVxcHvPCm8fx+pQhI2A3ujwm2ET4ukCSJKK/+U28bW24zpyl/e//nsT/+/doQy8/J0pvMDIIyONc5d7jktmzrgpnjxtrkoVlT+ahN4zu9ERXYwNdjfUYzCFkzBq9hbdanYbZd6djTbJw7LMG6k930W933VC4DIuOYdryOzi9bTOHP9jAmu//BZogHUIs3NoUVaG5v5lSeynlNv+ORGNDG4t2dZHY5l8X2hcfiv3BJSTPX85fW4tJDkueMGHry3w+hWOf1QNQsCiRtnMn6GlvxRJlpWiZODhbmFxW5sex+cUSXtlTx/snmnF65JteHhNMInx9iaTXE//jH9P613+Np+E87b/4BYl/8zdIhqElGgKHa4/jjkefrLBvQzU9HYOERZsoeTx31IMXQOX+3QDkzFuI3jB6U5sXZdwWQ2RcCHvfPYetycH218opeSKP0KiR3bto+SpqTxyht72Vyv27KSxZOUo9FqYyRVU433d+SFHTfm8/AKF9Xubvs5FfNYBJZ8YYl0bk175G+r2PTIqwX3mwjT6bizCricwZ4Xz8b58CMGf1g+j0wd1NKQhjIT3awk/XThu3chLXIsLXV2gsFuL/6q9o+clPcJdX0Pkf/0Hsn//5kB0+413rS1VUDn5QS+f5fsxhepY9mXfDuwJHYqCvl4bTJ5EkibyFS0f9fhdFxoew6rlC9qw/R0/7ANtfL2PpY3kjWoiv0+uZe//D7HztZU5t3Uz69FliGkW4LkVVqO+rp9xWTqmtlEp7JQ6vY0ibOE0EK89A1sEmQojEEJ1E+AP3E/nQQ2jMN1ezbqw4ul2U7/PXD5t9Tzpnd2zBMzhAQnYuqcXDO9ZIEIThE+HrCnTR0ST81V/R+lf/G+f+A2ijo4n+xjcCj+tN41flXlVVjm1poLmyG4NJy7In80Z8PuONOndoP4riI7V4OqFR1us/IYjMYQZWPl3A/veqaa/rY+dbFSxcm01S7vADVEpBMSmF02gqP8uxTz+g5KnnRrHHwmTkU3zU99X7K8jbyqiwVzAgDwxpE2OOodBaSFFUATllvfDeZnzd3aAJwbJkCVFPfx193PidGTdSqqpy/PPz+GSV9GnRGIz9nDu0D0mSmHvfw6I+niCMAhG+rsKQnk7c//hL2n72M/o++hhdTCwR960BQGe4WGh17Ee+Kg+1UXuiE61OYsmjuUTEjs2Cf8Xn49zh/YB/of140Bu1lDyey9ELa8D2bTjH3NUZZM6I/f/ZO+/wtup7/7/O0ZY8Jc/Y8R6xnelsMkgCCRkEEgqEVUpv6b0tpf21t7fjPrctdN629N62FHpboC0UCiHMAgnZe+9lO57xdmxH8tSWzvn9oUSJyfaSnZzX8/hJ7PPVOV9ZsvTWZ7w/132OSUvvo6m8lJrjR8iaPI3ErNwB3LHCUMcn+ajqqAq6x5+yncLl7/l3HWeIC3Yi5lnyiDPG4SwqwvbCq3hOnwZAl5WJ+V/+BX3u8Hs+1Z9q40xlB1q9irHzktn55kvIskzu9FlEJSSGensKCjclivi6CoYxY4h96ilan/8Dtr/9DVVEOGGzZ1+Y7zjINV+tdV2c2BKY4zb13kxiUwbPs+pMZRmu7i4iYuKIC2G3oKgSmbwkMCqpeGcjB1ZX4/fJZE28vkhDWLSZ0fPmc2z9Gg5+9D5LvvndIePVpjDweCUvVe1VwchWWUO+QkEAACAASURBVFvZJWIrwZhA3rluxHxLPjGGmAu3P3OG5heeuzD82mLG/NhjmGbOHJbmo163n6MbAoO4x8xNpqWqmObTFeiMRsbeuSjEu1NQuHlRxNc1CLv9dvzt7dj+/jqtf3gB0WC4qOZr8MSXy+5l74eVyDLkTksgOTd60K4NUHP8KABp4wpDnoYQBIHRs5PQ6lUc3VjH4XU1SH6JnCkJ13X7/FlzqTy4n47WZir27xnU+jWFwcXr91LeXh6MbJW1leGRepYLJJoSKbAUkGfJI8+ch8VgueQ8/m47He+9S8eaNReGXy9fRuQ99yDqBiftPxCc2FaPs9uLJclESl4kn/z+/wAYt2AJOmPoPQwVFG5WFPF1HUTeey/+7m463v+Alv/5X+TF84EBtJqwVcHuF+D4KvB0I2vC2S/+GKcvh5hUC2NuH1xXXr/PS23RMSBgWjpUyJmSgKgSObyuhqMb6/B5JfJnjLjm7VRqDYWLlrL9H3/j2IZPSRs/Ea1+eBRGK1wdj99DeVs5xbZAZKu8rRyv5O2xJjksORDZMgfSiNH6K3+Qkf1+utavp+3tVUhdXYHh13PnEv1Iz+HXwxFbo53KQy0IAkxcmEbR9k3Y29uITkwie/L0UG9PQeGmpl/ElyAIfwXuBlpkWb6kp1MIhEp+DywGHMATsiwf7o9rDxbRjzyCZHfQtW4djrVrkeItAxP5Kt8Aqx4HvxfOvWmUdEzmjNOPTjzMtHvyEFV5/X/dq9BUVorX5SI6MYnIuPhBvfa1yJoYh0otcHBNNSe3NSD5ZQpmjbhmdG5kwViMiSmUl5Ty6Hf+wEFTASadmmUTRvDlWRkh94BRuD7cfjdltrKA7YOthPL2cnxSz5luKeEpwTRinjmPSN21553KsozzyBFsr/0db30g1a/Pz8f8xSfQZWQMyH0ZTGRJ5tC66kAkfWoCospO8fZNAEy+53PDMoWqoDCc6K/I16vAC8Dfr3B8EZB97msq8H/n/h02CIKA5ckvIdntiLu24Wtqwt3acu0b3gi2qoDw8l7ormr2ZlHkvAtBlphqeh3jJ/UwcheYB+8NoPr4EQBSx4wftGveCOnjYhFVIvs/rqJ4ZyOSX2bMnKSrCrCtZa08V5/AbPsJUimlWJdKN2Gs3F/He4ca+ONjhczNHT4da7cKTp+TUltp0Gerqr0Kn3xBbAkIpEakBtKI5+YjhmtvrDbSU1OD7bXXcB4LDFxUJyRg/vznMU6dEvKUe39RcbiFtiYHhnAt+bNGsPPNvyD5/aRPmExc2vAXlwoKQ51+EV+yLG8XBCHtKkvuBf4uy7IM7BUEIUoQhERZlpv64/qDhSCKxH79aeqbG5CrTtG+eQveu5aiGXHtVNd1sfuFQMTrHC4pnH1djyLLAvnGjSRoy8CvgT0vwpL/6Z9rXgOfx0N9yUkAUscNnZTjZ0kdbUFUC+z9oJJTe5pQa8UrpiBrrHaeeuMwTjGSakMKqY5axnWeZLd5Gj5Jxif5eeqNw6z95iwlAhZiHF4HpW2lwQL5qvYqJKTgcRGR9Mj0gPXDuchWmPbSqRTXg6+tjfa336Zr02aQJESTiagH7idi4UKEm8hk1NHp4cTWQDSv8K4UmitKaCgtRqPTU7j4nhDvTkHh1mCwar6SgLqLvq8/97Me4ksQhH8F/hUgJSVlkLZ2YwhqNTGPPIL4m//G63TS9OyzJP74x2gS+6El+/iqYKpRluFA9wO4pDDiNJUUGNYH1kheOP72oImvhtJifB43lqSRhJtjrn2DEDJylBn5Xpl9/6zi5LYGVGqR3KmXFuG/vKMKrz/wBn48YjTJzgaSnfXEuM9yVhe4j16/xCs7Tg9JZ+SbGbvXHhjTYwuM6jndcfoSsZUZmRnsRBxlHoVR07fCcMntpvOTT2j/4ANkpwtUKiIWLybqwQdQhQ9eR/FgcXRDLT6PxIicKOLTw/j4t+8DMHb+IgxhN9/9VVAYigyW+LpcrF6+5Aey/BLwEsCkSZMuOT5U0IWFo05MgLYO/FYbTc88Q+JPfoIm4fq67a6I54Jzdp1nPE2ePDSCi6nhbyII8mXXDTQ151KOaeMKB+2afSEl34LfJ3Pgk9Mc21SHSi1eYkPx4ZHG4KBVp8rIqbAcCrpKGN95jI0x80AQ8EkyHxxpUMTXANPt6Q6mEIutxdR01iBf9NKgElRkR2YHxVauOReDun+aI2RJwr5jB7Z//AO/1QaAcfJkzI9/vv+i2UOMxop26kvbUGtFChekUrxtE/Y2G1HxieROV7p+FRQGi8ESX/XAyIu+TwYaB+na/Y5aq0MQRTR5eejOduE+VXpOgP0UTXwf6oS0YeDpwiWZOGJfBsA408cYxM5L1w0CXreL+lNFAKQM0Xqvy5E+Nga/V+LwuhoOr6tBrRFJG3shamd39yzILg3LIdNxGrOnjVRnLTXG1MA6T891Cn2n09MZtH0osZZQ21XbQ2ypBTVZ0VnBNGJOdA56tb7f9+E8WYTttdfwVFUBoE1Px/yFL2AYc/OKbZ/Hz5F1NQAUzEpC8nVxcttGIFBkr/jdKSgMHoMlvj4CnhYEYSWBQvuO4VbvdTFBk1W/j4Qf/IAzP/0Z7tJSzjzzDAk//nHvBdjYB+Hw3znadS9uyUScpoJ03f6ea0QNjF3Rx3twfdSXFCH5fMSmZmCKGlxfsb6SNTEOv0/i2KY6Dqw+jUotMjI/YA1g0qnpvkiA+UQNJ8ILmNx+iLGdJ6kzJCMJKkxaxYmlr3S4OyiyFgUFV313fY/jGlFDVlQW+ZZ8CiwFZEdno1Vpr3C2vuOpb6Dt9ddxHDwIBExSox95hLDZs2/6Dr+inY3YOzxExRvJnhzP9jdeQfL5SB8/kfiMrFBvT0HhlqK/rCbeAuYAMYIg1APPABoAWZb/BKwhYDNRQcBq4ov9cd1QodZdMFkVDQYSfvBfAQFWVhYQYM8+07sU5G1P07j/MLXuCagEL5PC3uGS5iqVBqZ/re934jqoOR5wA0kbwoX2VyN3agJ+r8TJ7Q3s+7gKrUFNfHoEyyaMYOX+umDqEeC0MY0cewWR3g5yuiuoiBzF8gmD66d2M9DmagumEEtsJTR0N/Q4rhE15ETnBK0fcqJy0KgGvpjd395O26p36NqwASQJQa8navkyIpYuHdYmqddLe4uDsv3NAU+vRak0lhVTX1KEWqtjwiKlyF5BYbDpr27Hh69xXAYGRzEMAsHI1zmTVdFoDAiwn/0cd1kZTT/4IQnP/AjtyJFXO80leE2pHDL9ENpOM8a0njCV7cJBURMQXg/+fVBsJjxOB42lpxAEgZTR4wb8egNF3oxEPC4fZfub2fVeOXMeHcWXZ2Xw3qEGfJL/wkJB4FjEGGZbd5LXfYrGyCyenJUeuo0PE6xOaw+x1WTvGdDWqXTkROcEa7YyIzMHRWyd55JielEkfP58olY8iDp6eEVze4ssyRxaW4MsBcZwRcZo2f76uwCMm78IY8S1fc8UFBT6FyWv0gvUmkBaxOfxIEsSgigimkwk/OiHNP/3L3EVFdH0ox+R8MMf3pAh4/EtdTj9EZhHjyE7qQROHAoU12vDAqnG6V8bNH+vuuITSJKfhMxsDOERg3LNgUAQBMbNG4nb4aPmpJUdb5cx7/E8/vhYIU+9cRivXwpGwM7o4mnVxxHvbuG/croUm4nL0OpopcRWEkwlNjuaexzXq/TkmnODtg8ZURloxMG3aZAlie6tW2lbufJCMf2kiUQ/9tgNfyga7lQda8Va340hTMPo25M4vunToJN97m2zQr09BYVbEkV89QJBFFFrdfg8brwed3A0jWgwEP+D/6Ll18/hPHKEMz96hvgf/Bf6UaOuec7W2i4qD7ciqgQm3zcGIXYq3D04dhKXo/rYOWPVITROqLcIosCkJWm4HT7OVHWwfWUZ8x4fxdpvzuKVHaf54EgDdo8Pk05Dysy7SD35If6Kw3S3LSIseniPkOkLsizT4mgJRrWKrcW0Olt7rNGr9BeGUJvzSY9MRxXCwu2gM/3rr+OtDbjbBIrpH8cwZkzI9hUqnN0ejm8O1NmNn5+Co72Vkp1bAIGpyx5QiuwVFEKEIr56ic5oxOdx43bYe8wFFLVa4r/3XVqffx777j2c+clPif/edzGMu3LqTpJkDp/rQsq7LZHI2NAOtHU7HJypKEUQxWGdcrwYlUpk+n2ZbPtHKbYmOzveLmfOo7n8dNnoS+wkdq6sp/rYYY5tWMOMBx8L0Y4HH1mWOWM/E0gjnpuNaHPZeqwxaUyMMo8KdiOmRaSFVGxdjLuiAtvrb+A6GTAFVsfGEv3Iw5hmzrzpi+mvxNENtXjdfhKzIknKiWTDyy8iSxLZU2cQk5IW6u0pKNyyKOKrl+iMJuztbXgcDvhMcETQaIj91rcQ9Hq6N2+h+Rf/Tcw3vk7YjBmXPdfpo610tDoxRekYNb0fzFr7SFP5KWRZJiEjC53x5km9abQqZj6YzZbXT9He7GD3exXMeigHlarnG/P4BUuoPXGM00cOkTdzLuYRN2fhvSzLNNobL9RsWUtoc7f1WBOmCQuO6cm35JMakYooDC0h421qou2tldh37QJADAsj6v7PEXHXXQjageucHOo0VrRTV9KGWiNSeFcqVUcO0lpThT4snAl3LQn19hQUbmkU8dVLdKaAKHHZ7Zc9LogiMV/9KqLBSOfq1bT+72/x29qIXHp3j3Uel4+T2wMdYWPnJaNSh/6NraG0GIARufkh3kn/ozdpmP1wDptfK6GlpouDq6uZsjS9x8y+MLOFnOkzObVrG0fWfsQd//LVEO64/5Blmfru+h5iq8PT0WNNuDacfHN+sEA+OTx5yImt8/ja2mh/9126NmwEvx9BoyFi8WIi71uOKmxwvPCGKl6PPxhNL5idhErt5fCafwIwcckytIbQRtcVFG51FPHVS3TGwIu723Flt3lBFDF/8QlU5mjaXn8D26uv4jt7FvMXHg+mQUp2N+F2+IgdGUZybui7r2RJorHsFABJuXkh3s3AYIrUBSJgb5yi5qQVU5SO0bN7RrdGz51P5cF9NJWX0lReSmJ2boh223skWaKuq+6CqamthE5PT8PeSG3khZotSz7JYclDfni05HDQ8dFHdHz8CbIr0MEYNm8u0StWoI4Z2iOwBoui7Q04OjxEJwQ8vfa9vxKP00FCZs6wmVahoHAzo4ivXqIzBj45uq8Q+TqPIAhELVuG2mKh9YUX6PzkE/w2G7Fffxq7Xab8QKBbbNwdKUPiTc/WWI/b3o0p2kxEbHyotzNgRCeYmL4sk53vlFO8sxFTpJb0cbHB43pTGAW338HR9as5/OlHLM789pCvG5JkiZrOmmBUq8RWQre354eDaH10MLKVZ8ljhGnEkHjeXQ+yx0PnunW0v/c+UlcXAMZJk4h+7NFbroPxarSdsVN+4LynVxqt1ZVUHtqHqFIxZdn9w+bxVlC4mVHEVy85n3a8lvg6T9isWaiiomj51a+x796Nv72dyoKHkPwyqWMsmEcMjdqqhlPnUo45eTf9i3RiVhSFC1M59GkNBz+twRChJSH9gufRqBm3U7Z3J21NDVQfP0L6+Ikh3O2l+CU/1Z3VwajWKdsp7N6ez0ez3kyBpYA8Sx4FlgLijfHD7nGV/f6AbcTbbwdtI3SjcjF//vPX1Ul8KyFJMgc/rUaWIWdKPBExWlb//m0ARs+ZT0RMH8afKSgo9BuK+Ool5wvR3Y7rE18AhjFjSPzZTznz81/QXNZCdWsJ+uRExtyePFDbvGHO13vdrCnHz5I5IY7uNjele8+w+70K5n0+j6j4QFRTrdUy9s5F7H1/JUfXryZl9FhU6sH3rDqPT/JxuuN00PahxFqCy+/qsSbOEEeeJS/YjRhnjBt2Yus8siRh37OH9pVv420MjILVpqYS/egjGAoLh+39GkgqDjbT1uTAGKGlYHYSJzatocvaSmRcAgVz7gj19hQUFM6hiK9e0hvxBaBNSyPxF7/gyI8/RXZ6iavYBBUGKAy9n5aruxtrfR2iWk18ZnaotzNojJ2TjKPTQ12xjR2ryrnzi3kYwgJdchkTJ1OyYwsdrc2U79vNqBm3D9q+vJKXqvaqYGSr1FZ6idiKN8YHo1r5lnxiDMO/5kmWZZyHDtH21ko81dUAqOPjiX74IUwzZgz59G+osLe7Obkt0LxTeFcqXWebKN6+GRCYdt+KkH5wUFBQ6IkivnpJMO14g+ILoL4J3Ik5GK2NjKg5SfN/n8D8+ONE3L0kpJ/mG8tLAJn49Cw02pt/3t15BFFgyt3pODs9nK3vZtc7Fcx5LBe1RoUoqpiwcClbX3+FE5vXkzlxKhq9fkD24fV7qWivCEa2ytrKcPvdPdYkmhKDxfF55jwsBsuA7CVUOE+coO3Nt3CXlQGgMpuJuv9+wufNRdAo4uFKyHIg3ejzSozMN5OQGc7aF19BlmVyb5tNbKoyKktBYSihiK9eEux2vM6ar/P4/RJFOxpAEJn4xEwiS7y0r1qF7dVX8dTWEPPlL4fMm6ixtASAEbdIyvFiVGqR2z6XxabXSrA12dn/0WmmL89EEAWS8gqITU2nteY0Rds3M37B4n65ptfvpay9LJhCLGsrwyt5e6xJCksKusfnWfKI1oe+I3YgcBUX0/b2qqBBqioygsjl9xF+1wLEW9ir63qpOWml+XQnWoOKCfNTKNmxFVtjPaao6H57viooKPQfivjqJfpeRr6qj53F0eEhIsZA6ugYhLEPoh2ZTOsfXqB78xa8tbXEfvvbaOIGtzD2VrCYuBZ6k4aZD2Sz+e8l1Je2cXJ7A2PmBKwXChfdw7o//Z6SnVvJmTajV8OI3X43ZbayYGSrvL0cn+TrsWZk+MhgVCvfkk+k7uYeeuwqLaVt5Upcx08AIJpMRN57DxGLFyMaDNe4tQKAy+7l6MZaAMbfmYLH2c6xjZ8CMGXZg2h0AxOpVVBQ6D2K+Ool2otqvmRZvq50od8nUbyrCYCCWSMQxMBtTLfdhjohgZbnfoO7opLG73yX2P/3/zAOYh3Y2boaPE4H4ZaYW7ojKjLWwPTlmex4u4yS3U2Em/WkjY0hNjWd5Pwx1Bef4MSmdUxd/uA1z+XyuShrKwuamla2V+KTe4qt1IjUYFQrz5JHhHb4DjG/EVxlZbSvegfnkcAMUdFgIGLpUiKWLEEVNjQ6f4cLRzbU4nH6SUiPIKXAzKa//BHJ5yN9wqRb9oOUgsJQRxFfvUSt0aDSaPF7Pfg87uv6dHn62FmcXR4i4wyXGKrqMjIY8dyvaf398zgPH6b5F78g6v77iXrwgUEpMG44n3LMUV6sEzIimbAglcPrajj4aTWmKB2xKeFMuOtuGkpOUnFgD3kz5xAR21OkOrwOytrKKLIWUWItoaqjCr/sDx4XEEiLSOtRsxWmvbWc2F2lpQHRdfQoAIJBT+SSJUQsXXrLu9L3hsbyduqKbag1IhMXpVF5cC/NVRXojCYmLlkW6u0pKChcAUV89QGd0Yijw4Pbbr+m+PL7JEp2n4t6zbwQ9boYVVgY8f/5fTo++IC2t1bS/s47uMvKiP3G11FFRQ3IfThP4008Uqg3ZE2Mo8vmovxAM7veq+DOJ/KJjIsnc9I0Kg7s4ci6T5i84iFKbCVBB/nTHaeRkILnEBHJjMwMOsiPMo/CpLk1ozqukhLaVq0KphcFg56IhYuIvGcpqohbI9rX33jdfg6tDYwQGn17ErLUzaHVgRFCk+/5HHqTImYVFIYqivjqAzqjCUdHO26HnTDz1bvOqo624uzyEBVnICnnykXTgigS9bnPocvOpuW3v8V57BgN//5tYp5+esDSkM6uTmyN9ajUGuIzsgbkGsOR8XeMpNvmoqmyg53vlDP14ZFIYxOw7mqjfu8aXpPXYo++IKJFRLKissi35FNgKSAnOgej5tadoSfLMq4TJ2h/7/1gIb1g0AfmLy5diio8PMQ7HN4c31KHs8uDOdFEVmEsm/72f/g8blJGjyN1bOitaxQUFK6MIr76wPV6ffl9Eqf2BKJe+bOSLhv1+iyGsWNJ+s1vaP3987iKimj++c+JWLIE82OP9ns3ZGNZIOWYkJmNWmnnD9Ll60I1zYatrpX6ah/bX9xEZcFu4kZ4ia3wEV+iQr24gIKY0eRZ8siJzsGgVorEZVnGefAg7e+9j7u8HDhX03X33UTcvURJL/YDLTWdVB5uRVQJTFqSRvmB3YF0oymMKfc+oBjQKigMcRTx1Qeud8RQ1ZFWnF1eouKNJOVcf/pQbbGQ8OwzdHzwIW1vv03n6tW4Tp4k9lvf7NdZdg23sMXExXS4O4KdiMXWYuq66gDQphrJbZtHZFsChc13krrAhLNjPyqPzJyoB0gZNS7EOx8ayH4/9j176Xj/fTw1gXSYGB5O5NK7Cb9roVJI30/4PH4OrK4GIO+2RETRzuFPPwZg6rIH0CviVkFhyKOIrz5wPvLlukrky++VKNlzUYfjDX4iDaQh78Mwbiwtv/sdnpoaGr/zXaIffZSIJYv7XIwv+f00lQcsJm418dXuag8ILVtAbDV0N/Q4rhE1ZEdnk2/JZ2RuNrWrPdAiMMmfhndRCgc+epcjaz8hOW80okoVonsReiSPh+7NW+j46CN8zYFB8Sqzmch77iF8/p2IA2RKe6tyYmsD9nY3UfFGRk2LZ+NfXsTv9ZA2biIpo5UPAgoKwwFFfPWB8wWtnquIr8ojLbi6vUQnGBmR3fuieV1WFknPPYf1r3+le/MWbK++in3PHmKeegptclKvz3u2rgavy0VETBzh5uE/muZqWJ3WHgXyjfbGHse1opZcc27QYysrKguN6kIaNk5q5eCaag6vq2HWg6MJt2yjy9pK+f7d5E6fNdh3J+T4u7vpWreOztVr8Hd0AKBOSCDy3nsInzMnZGbBNzOtdV1UHGpGEAUmL0mjbO8OWmtOYwiPYPI9nwv19hQUFK4TRXz1gfNpR9cV0o5+v0TpvjNA76Jen0U0GIj92tcwTZnC2T+/hLu0lMb/+A+iVqwg8p6lCL2IvjRXVQCQkJXTp70NRc46zwZTiCXWEs44zvQ4rlfpyYnOCVo/ZERloBGvXPOWMT6WzrNOyvY3s+fD0+TPWsj+D1/n+Ma1pE+YhFZ/a9R7eZtb6PzkE7o2b0Z2BWZNajMziFq2DOO0acrsxQHC5/VzcHU1sgz5tyUgit0cWb8agKn3rUBnvHWbOxQUhhuK+OoDOkPgxe5Kka+6IhvOLi+RsQYSs/rPKsI4eTJJefnYXnuV7s1baHvjDRx792D5ylfQpd/YDLczlYEZegnDfJC2LMu0OluDQqvYWkyLs6XHGr1KzyjzqKDYSo9MRy3e2J/AuHkj6bS6OFPZQeVRPZaRaVjrqinatokJd93dn3dpyOEqLaPz44+w79sPUsBSQz92DFHLl6MfM0Yp8h5girY30mVzERFjIGdqLBteeh7J5yNz4lSSRxWEensKCgo3gCK++oDuXNrxct2OsiwHo165UxP6/Y1JFWYKRMGm38bZP/8p4Iz/3e8RsWgR0Q+tQLyOT8F+n5fW2moA4tIz+3V/A40syzQ7moORrWJrMVaXtccao9rYQ2ylRaShEvtWmyWIAtPuzWDzayV0Wl1ExRaCXB0YOzR1Bqaom2v2ouzzYd+3j87Va3CXlgZ+qFYRNut2IpbefcNiX6F3WBu6Kdt/BkGAKXenc3LLOtqaGggzW5h09/JQb09BQeEGUcRXH9CeEzhuh+OSY2eqOulodWII1zCywDxgezAWTiD5t7+l7a236Fy7js7Vq7Hv2oX5iS9gmjnzqqKvtaYayecjOjFpyBsyyrJMk73pQhrRVoLNZeuxJkwTFhRbeZY80iLSEIX+T4Fp9WpmPJDNpteKaW81YIjKxNlRybENa7jtgUf7/XqhwN/ZSdeGjXSuW4vfGvg9iyYT4QsWELF4EWrzwD2nFXri8/rZ//FpZBlypyXgdtRTvH0zgiAw48HH0CgNDQoKww5FfPWB84LFbe++5FjZuahX9qR4VKqBrYERjUYsX/oSYXPnYX35ZdxlZbT+7vd0bdiI5ckvoU1JueztmqsCHkxD0VhVlmXqu+uDKcQSWwnt7vYea8K14cG5iPmWfEaGjxwQsXU5ws16pi/PYvtbpbjd+Xjd5VQdPkjubbOxJPWfDchg466qonPtWuzbdyB7vQBokpKIWLyYsNtnK8OuQ8CJLQ3BdGP2pCg+ffEVAMbccRexqUrkUUFhOKKIrz5wsdXExcO1287Yaa7uRK0VyZgQO3j7yUgn8ec/o3vLFmyvv4GrqIiGb/8H4fPmEvXQQ6ije6bEzlQGxFdCRujrvSRZor6rPmj7UGItodPT2WNNpDYyKLTyLfkkhSUNmti6HPFpEcEZkC6y8XnKObT6Q+Z/+elhVf8keTzYd+2ia936oCkqgGFiIZGLF6MfN25Y3Z+biZaaTsoPBrobpyxNY/8/38bZ2UFsajqj584P9fYUFBR6iSK++oBaq0VUq5F8PnxeDxqtDiBY65UxPhatfnB/xYIoEn7HHRinTKH97bfpXLeero2b6N6xk8h77yXynqWIBgNet4uzdTUIgkBcxuDXe0myRG1nbTCqVWItocvb1WNNtC6aPEseBZYC8ix5jDD1vWO0v8maGHeuA3ICtrpyzlRUUHvyGKljxod6a9fE29BA18aNdG3egtQdiN6KJhNhc+cScdcCNCNGhHiHtzZet58Dn5wGIH9GItba49QXn0Cj0zPjwccQ+1i/qKCgEDoU8dVH9EYTjs6OwHBtrQ57h5u6YhuCANmT40O2L1V4OJYnnyRi0SJsb/wDx/79tK9aRdf69UQ9cD9dI5OQJQlL0shBsUiQZInqjupgZOuU7RR2b89GBbPeHIhqmQORrQRT/zcqDATj7wx0QLq6JtLdtptDq/9Jcl4BKvXQG9UkeTw49uyha+NGXMUlwZ9rMzOIyyf6VwAAIABJREFUWLgQ04wZiDpdCHeocJ6jG2uxdwRmN47IFln74gdAwMX+WrNkFRQUhjaK+OojOlNYQHw57IRFmyk/0IwsQ0q+GVNk6N/ENElJxH/vu7iKi7G99hruikqsL79CVZgWv0lP/IyMAbmuX/JzuuN0D7Hl9Dl7rIkxxAQNTfMt+cQb44eF2PosokrktuWZ2Nud1HQU01J9huLtWxkzb2ikhWRZxlNZSffWrXRv34F0zpdO0OsxzbiNiPnz0WWHPvWscIHG8jZOHzuLSi1QuDCJXSv/hN/nJX3CJNLGTwz19hQUFPpIv4gvQRAWAr8HVMArsiz/8jPHnwCeA87Pb3lBluVX+uPaoSY4XNtux+PyUXW0FQh0JQ0l9Pn5JP7ylzj27qVt1Sps9ZX4nF0IH/yTTo2J8Ll9cyT3ST4q2yuDsxFLbaW4/K4ea+KMccHIVp4ljzhjXF/v1pBBa1Az68FcOltn0Hr6Yw58tJrMSVMwRkSGbE++tja6t22je+s2vHV1wZ/rsjIJu/NOwmbOVArohyBuh5eDa6oBGDMnmdJda2hvbiLcEsuUe+4P7eYUFBT6hT6LL0EQVMCLwHygHjggCMJHsiwXf2bp27IsP93X6w01dEG7CTtVR1vxeSTi0iKIThh6Q4QFQcA0fTrqcWNxfv+biDYb4e1dWF96ifZ33yVi0SLCF8xHdR2Deb2Sl6r2KoqtxRRZiyhvK79EbCUYE4K2D/mWfGIMN/f4oogYA3M/fzsf/e9xuttr2P7Guyx86kuDugfJ6cSxfz/dO3biPHYsaIaqiozANHMWYXPnKN5cQxhZljn0aQ0uu4/YlHBEoYbKQ/sQ1WpmP/qEYiuhoHCT0B+RrylAhSzLVQCCIKwE7gU+K75uSs4brbrsdioOBxzVc6cOrajXZ2mtPo1oMhGXV0Di+Km0v/MOnppa2v7xD9rfe4/wefOIuPtuNPEXIlNev5fy9vJgJ2JZWxkeydPjvElhScE0Yp4lD7P+1vOCSsiIZOry+9j+xu8pP7CP7CMzyZyQO6DXlD0eHEeOYt+5A8fBQ8iec4+LWoVxylTC58zBMGECglqpMhjqnD52lvrSNjQ6FblT9Gx7/S8ATLn3AaITez/DVUFBYWjRH6/GSUDdRd/XA1Mvs+5zgiDMBsqAb8myXPfZBYIg/CvwrwApV/CmGkrUWO2sLW/H2djJur/vI0MzgQiLHnfE0O5COnPO3ysxKwfT9OkYp03DefQonR9/jPPYcTrXrKHj0zU4x2RQNSGeQ+ZOKjor8UreHudJDku+ENky5xOl778RSsOZsfPyOH10GrUndrP5r/8g4Wf/1e/1f7LHg/PYMex79uA4cBDpIqNffX4ephkzMc24DVV4eL9eV2Hg6LK5OLqhFoCx8xI4+PGr+H1eMgqnkDXpci+pCgoKw5X+EF+Xq5CWP/P9x8Bbsiy7BUH4CvAaMO+SG8nyS8BLAJMmTfrsOYYUW0pbeOqNw6S22xkryST6fEhq2NzdzSvP7+SPjxUyN3do1jQFh2lnBoZpC4KAOCaP1mQ15cct+NdtJupoNeKOKow7YHyEGt3oSOxT88lMHU++JZ9R5lFE6kJXzzSUEQSBhV95kL9/9zjOribW/mk193zrXjTavolyyekMCK7de3AcPoTsvJDm1aanY5o5g7AZM1DHDp63nEL/4PdL7PtnFT6vxMj8aBpPbaDzbAuR8YlMuVep81JQuNnoD/FVD1xs6Z0MNF68QJbli4fuvQz8qh+uGzJqrHaeeuMwTq8fh6BFBYT5PdiRKVP78XrhqTcOs/abs0i1DK3aL5e9m7amBlCJNOjb2XzqLYqtxVS2V+KX/YFFs9UYJ6ZzW6WagqJu4rv8jD6pR3XqDMZJZwmbLWO0KIXaV0NvMjH70QfY9NfXaCrbyu73cpi9ogBBvLFuTl9rK46DB3EcPITr5Elkny94TJuRgWnaNEzTpymeXMOcou2N2JrsmCK1RETXc2TfEdRaHbc/+kXUfWiEUVBQGJr0h/g6AGQLgpBOoJvxIeCRixcIgpAoy3LTuW/vAUoYxry8owqvP1DI7BG16GUBQXZTpZHwnntv9folXtlxmp8uGx3CnV7A4XVQ2lbKkQObae1uwBbl4+PDzwWPi4hkRGYEa7ZGmUcRpg1DliScR4/RtX49jkOHcOzbh2PfPkSDAeP0aYTNmoV+9GgEMXRO80OV3GnTKN+/h5rjpZTv20RUXCTj7rj66CHZ68V1qhTn0aM4jxzBU1Nz4aAgoMvNxTR1CsZp03vU5CkMX1pqOind24QgQFYhHPzkIwCm3beCiFjlMVZQuBnps/iSZdknCMLTwDoCVhN/lWW5SBCEnwAHZVn+CPiGIAj3AD7ABjzR1+uGkg+PNOKTAllRn6BDJ4MsuynT+INrfJLMB0caQia+uj3dnLKdCjrIV3dUIyGReNKD2efHYdaQFZUV9NjKjc7FqDFech5BFDEWTsBYOAGfzYZ91y66t+/AU1VF9+YtdG/egioyEsPEQkxTpqAfO1Yx6TyHIIrMePAhrPW/ostaxMntuYRb9GSMv5AWlGUZb309rhMncBw9iquoGNl1IZ0oGPQYxo3DOHESxsIJqKKUurqbCY/Tx76PAkOzMyYYOb7xTWRJIn/WXNLGFYZ6ewoKCgNEv7Q/ybK8BljzmZ/96KL//yfwn/1xraGA3X0h9RMv6xEAj+zCpupZpmb3+Bgsuj3dQY+tImsRtZ21yBeV3qkEFTlROSS4zqIx+fnivf/OyIxRN3QNtdlM5NKlRC5diqe+AfvOHXTv2InvzJmgEBO02oBYmDQR/Zixt3x0xjwiidG3z+HYxs10Nu/k4KdmdE4r4W1VOIuKcBUXI3X2HKukTU3BMG4chnHj0Ofn98l/TWHoIssyB9dU4+zyEJ2g5UzZatwOO4nZo5iwcGmot6egoDCAKL3nvcCkU9Pt9oEM6b5A7ZMb96XrtAP36+1wdwQjW8XWYmq7anscV4vqQGTr3Kie7OhsJIeb9z95BrVJS1Jq3xzNtclJaB96iKgVK/DW1ATqkg4cwF1RiePAARwHDgT2ER+PYewY9KPHYBhdcMtFbvzd3WTHJlEhefDaT9NZup3t5fGM7dqM2tNBa7ebWr+W0ogkquPSyZozjccXjcc8xGoFFfqfikMt1Je2odaKiByg/Uwj4ZZYZj70uJLGV1C4yVHEVy9YNmEEK/fXEeWFaEmDhAofPlSSD78Y+JWqRYHlE/rPl6fD3UGRtYgSayC6Vd9d3+O4RryQRiywFJAdnY1W1TNiUn06YL0Wm5qBqOofOwxBENCmpaFNSyPq/vvxWa04Dh7CefQorpMn8TU307Whma4NG4GAGNNlZwe+crLRpaXdFJEdWZbxt7Xhqa7Gc/o07qoqPNU1+M4EhqynCX6KRB8e8RjeiEUcTbybTd01FKfHc0YXCefGKu0q7uDN0h1DultWoe/YGu0c2xRw24kZUUvtiZOotTrmPP6loHGzgoLCzYsivnrBl2dl8N6hBnLtAggCdpUGWfahlT04z/1KNSqRJ2f13knc5rIFhVaxtZhGe48GUrSilpzonKB7fHZUNhrV1Qc5t1RXARCXPjDzHAHUFgsRdy0g4q4FyJKEp6oK5/ETOE8cx11ahq+5GV9zM/adOwM3UKnQJI1AOzIFbWoK2pQUNCkpqGNiEPpJIPYnsteL7+xZvE1NeBsa8NbX46mvx9vQiNTVdcl6QaNBm5VJdnY2bXWVtLbb8KtbqewuQGcaTavO28OsxSfJ+CT/kO2WVeg7HpePPR9WIvllYpI6qD2xHRCYueLzRMYNbYNmBQWF/kERX70g1WLiD/ePY9vLRQC0qbSYfE50fjdejQmNSuSPjxXe0BvnWefZoHt8ia2EJntTj+M6lY6c6BwKLAXkWfLIjMpEI15dbH2W5tOVAMSnZ93Q7XqLIIrosrLQZWURdd9yZL8fT20t7vLyc18VeOvr8dbW4a2tw75r14Ubq1SoY2JQx8ehiU9AHReHKjoKVWRU4N+oKFQREf0m0GRJQnI4kTo78Le3B7987e34z57F29KCr6UFv60N5Mtb0IkmUyAKmJ6ONj0NXXo6mqSkoLP8bS1nWP3752g6W4RLjCVGjGOmS802ve8St7yh1i2r0A/YqpB3vcCB7SJ2Zw4GoZrmxnIwxDBu8XKS85XHWkHhVkERX70k2Qm5ceE0aiTOVOsJ83UQpfazdEoKT85Kv6bwanG0BCJbtkBkq8XR0uO4XqVnlHlUMLKVEZmBWuz9w+V2OOhobkJUq7EkX93uYKAQVCp06emB2YILFgAguVyB6FFtLZ6aWjy1NXjrG/DbbMEomYsTVz6nQY+oNyAa9IhGI4LeEBBkKlWgbkalQhAFZElG9nnB50P2+pB9PiSXE8nhQLY7kJzO67sTohgQhQnxaJOT0SQlBb6Sk1FFRyMIV/bxioxLIH/2PE68+R5OYS+YFpPiUzPRLXNI7++xNtTdsgr9TPkGWPU4Fd2TaXDejSC143Duxi9BulDJ6KTlod6hgoLCIKKIr14gSzKVh1vRqkUeeyCXiv2l1J5w8dWHxpA27tI3S1mWaXY0B8VWibWEVmdrjzUGtYFR5lEB6wdzPumR6ajE/ku7tZ5LOcYkp6BS31jEbCAR9fpgdOxiJI8HX0srvpaAAPO1tASiUO3tSB3nolOdXchOF36nC39b3/ciGPSoIiIDUbXgVyRqiwV1XBzquHjUFnOfZiSOuWMBf165jnBfF/W+Y6SpJ5LvVdMtQqm2pwAbzG5ZhQHEVgWrHsfqsHCsexGS5EVwbcQrySTou5lmrkJ45wvw1V1gHriSAAUFhaGDIr56QUttF/Z2N8ZILYkZkdSdCBTIuh12ICC2ztjPBOq1zkW2bC5bj3OYNKYeYistMg1RGLgOp5bqQMoxLi1zwK7Rn4habaCjMvnKTQuyJCG7XEhOJ5LTiXz+X58fZAnZ7wcp8K+gUiGo1QHhpNYgaNSIOh2C0YhoNCEaDYPSYaZSazgZN5XpjRtJcpRyPDqZ8f54JrvVOASZOo0UXDuQ3bIKg8juF3B5tezpfhy/JCC7NuPzu4nUuJgddxqVIIPfC3tehCX/E+rdKigoDALKq3svqD5+FoC0MTEIooDOaMLj93Ck9gBrdYcpsZbQ5u4ZignXhJNnyQs6yKdEpAyo2Posg1FsP9gIonhOPA2v7rDbp4+leH0NWV3ljOg+wLGI+Yzz6pjlUrNR8NKilvu9W1YhdEjH3mFvxyPYfZF4XfuQ/VaMKi/zEirRqs5FOyUvHH9bEV8KCrcIivi6QTwuH3WnbHj8HqqjT7Dh0Crqqw4R2dWJtbqJM6aAbUKENiJYr5Vvzic5PHlQxdbFeD1urA11CIJAbGrvOzAV+ocvz8pgyYGxJDobifR2UOcppkw7lhyvirkuDWsNXjyavnXLKgwdjrfPocWbidtdAt5qtKKfufFVmNTengs93aHZoIKCwqCjiK/rQJIlajtrKbGVUHqwEa8tgvbIM1TUBOwSIkUfZlFFqi6ZJaOXkm/JJyks6arF14PJ2dpqZEnCnDQSjU4f6u3c8qRaTPzh8Sn88BUbM5q3UNBdwrqYJAxqCyk+FQvcWhY8nK/YTNwEVJ84S5l7Dg5XJXiK0Ah+ZsWdxqy7TIOHNmzwN6igoBASFPF1GSRZorqzOuizdcp2im5v4FNpbukcjLKEO9nKjBEzyLfkE5es43j1eySE53Jn2oIQ7/5SWk6fSzmm3Twpx+HO3Nw4/vH9z/H6nzuxlx5masdB9o+YT4EunBxJhf+gFU9eHFqD8ic6XGk7Y+fQp9U4pDb87mPoRDe3xdaQZLzUDw5RA2NXDP4mFRQUQoLyyg74JT/VndVBQ9NTtlM4fI4eayx6C/nqcRh82YTFGPmPFfej1ga6Ea2qWo4DbvvQTBsMt2L7W4VUi4nv/fuTfPy7X+LoaOfrc0RG3TaFza+fovOsk53vlDP74RzUmqFnNqtwdVx2L7veq6DbVorbV4ZJ5WCquY70sCu05ao0MP1rg7tJBQWFkHHLii+r08rOhp1BseXyu3ocjzPEkW/JD9ZtxRpiObapjjJtMxmjY4PCC0BnDKSHPM6egm0o4Pf5aK2tBpTI11BEo9cz/f6H2fSXP3Fi01oSs3KY/VAOm14r4Wx9N3ver+S2+7NQqZRZf8MFv19iz4eV2OpLcdh2EmY2UjjxPrLLfgF+TaC4/jyiJiC8Hvy7YjOhoHALceuKL5eVN0+9Gfw+wZhwoUDekk+MIabHer9fouakFYD0cT2PnRdfLrt9gHd949ga6pB8PiJj49GHKTUlQ5HErFzyZ8+lePtmdr79Oku+8R1ufziHza+foqmyg/0fVTHt3kwEcWjUECpcGVmWOfRpDfXFxXS1bCbcomPsHXeRP38R2JYE7CSOvx0orteGBVKN07+mCC+Fa+L1eqmvr8flcl17scKAotfrSU5ORqPpvWfmLSu+MiIzmJ86P+Aib87DYrBcdX1TRQduh4+IGAPmxJ6F0GqdDlFU4fd68Hm9qPvwgPQ3FywmlJTjUGbc/MU0V5Zjbahj3wermPnQ48x+KIdtb5ZSV9KGWlvNpEVpigAb4pTsbqJ0z2E6mjYSZtaSP3sOY+9cGDhozghYSSh2Egq9oL6+nvDwcNLS0oZMM9etiCzLWK1W6uvrSU/vfUf6LZvLUItqnhzzJDOTZl5TeAFUHwt4e6WPi7nkiS8IAjrTudSjY2hFv1pOn6/3Uj5ZD2VUajUzHvo8aq2WmuNHqDq8H3OiiZkPZqNSi5w+dpajm+qQrzBXUiH01BZZObh6F+2N6zFGqsmbOYuJS5Ypb5QK/YLL5cJisSjPpxAjCAIWi6XPEchbVnzdCM4uD02V7QiiQOroyws17RBMPcqSREuNEvkaLkTExDH5nvsBOPDRe3SebSF2ZDgz7s9CVAmUH2imaEdjiHepcDla67rY9uZGOpo2YghXM/aOeUy5937ljVKhX7ne51ON1c4PPjzB6GfWkf791Yx+Zh0/+PAENdah8/40nOmPv2tFfF0HNSetyDKMyIpEb7p8SlF/Tny5HUOn47HtTBNelwtTVDSmqOhQb0fhOsgonEzauEJ8Hg87V76O3+cjISOSacsyEQQo3tlIye6mUG9T4SK6bC7Wv7yG9oZN6AwqChfdpUS8FELGltIWFv5uByv319Ht9iED3W4fK/fXsfB3O9hS2tKr87a3t/PHP/6xV7ddvHgx7e3tV13zox/9iI0bN/bq/Ffj1Vdf5emnn77qmq1bt7J79+5+v/bVUMTXNZBlmdPHL6Qcr4T23Igb9xCKfAUtJpSo17BBEASmLHsAU7QZW0Mdh1Z/CEBybjST705HEODE1npKdisRsKGAy+5lzYv/xFq7CbVOZMrypUxYuFQRXgohocZq56k3DuP0+vFJPUsUfJKM0+vnqTcO9yoCdjXx5ff7r3rbNWvWEBUVddU1P/nJT7jzzjtveF/9gSK+hiC2JjtdVhd6k5qEzCs/efSmQCeh2zF07CaCxfZKvdewQqs3MOvhLyCqVJTt3Un5gT1AYJbo5CXnBVgDxbsUARZK3A4vH/7PP2iu3IhaIzD9/uWMn79IEV4KIePlHVV4/dJV13j9Eq/sOH3D5/7+979PZWUl48eP5zvf+Q5bt25l7ty5PPLII4wZMwaAZcuWMXHiRAoKCnjppZeCt01LS+Ps2bNUV1eTl5fHl7/8ZQoKCliwYAFOZ2DawxNPPMG7774bXP/MM89QWFjImDFjOHXqFACtra3Mnz+fwsJC/u3f/o3U1FTOnj17yV7/9re/kZOTw+23386uXbuCP//444+ZOnUqEyZM4M4776S5uZnq6mr+9Kc/8dvf/pbx48ezY8eOy67rbxTxdQ1qT9oAGJlvQbxKp5luiKUdZVkOFtvHp2eFeDcKN0rMyFSmLHsQgAMfvktrTeDFMm3sBQF2cpsiwEKF1+XlvV+9TEvVLlRqkVkPr2D8/LtCvS2FW5wPjzReEvH6LD5J5oMjDTd87l/+8pdkZmZy9OhRnnvuOQD279/Pz3/+c4qLiwH461//yqFDhzh48CDPP/88Vqv1kvOUl5fzta99jaKiIqKionjvvfcue72YmBgOHz7MV7/6VX7zm98A8OMf/5h58+Zx+PBhli9fTm1t7SW3a2pq4plnnmHXrl1s2LAhuDeAmTNnsnfvXo4cOcJDDz3Er3/9a9LS0vjKV77Ct771LY4ePcqsWbMuu66/uWWtJq4HSZKpKwk8eVILrt4Reb7bcahEvrqsrbi6u9CHhRMeExvq7Sj0gqxJU2lraqB093a2vfFXFj/9bYyRUaSNjQEBDnxympPbGkCG/JkjQr3dWwa30827P/sD1oZSVBoN85/8F3KmTgr1thQUsLt917fOc33rrsWUKVN62C08//zzfPDBBwDU1dVRXl6OxdLzvTM9PZ3x48cDMHHiRKqrqy977vvuuy+45v333wdg586dwfMvXLiQ6OhLa5n37dvHnDlziI0NvO+tWLGCsrIyIGDXsWLFCpqamvB4PFe0irjedX1BiXxdhZbqTlx2H+FmPdGJxquuDYqvIVLzdX6eY2xqupIGGabUWO3805vBNpueA6fq+fZ//JQfvHeEGqs9kII8VwN2cnsDJ7fVKzYUg4Cz287bz/4Ka0Mpao2exV97WhFeCkMGk+764ikmbf/EXUymC56XW7duZePGjezZs4djx44xYcKEy9ox6HS64P9VKhU+3+WF4Pl1F6+53te4K73nff3rX+fpp5/mxIkT/PnPf76iXcT1rusLivi6Cucd7VMKzNcUMDrD0Eo7KsX2w5tgx9LBRrZFTKFbbSTcZaNiwz9Z+NvtbCltIW1MDFOWZgS6IHc1cWR9LfI1Ug4KvaejuZlVP/4F7c11qLVh3P3Nfyd9fH6ot6WgEGTZhBGor2HErBYFlk9IuuFzh4eH09V1maHw5+jo6CA6Ohqj0cipU6fYu3fvDV/jWsycOZNVq1YBsH79etraLp2VOnXqVLZu3YrVasXr9fLOO+/02GNSUuC+v/baa8Gff/a+XWldf6KIryvg8/hpKAs8sCnXSDnC0Es7KsX2w5fPdix5VDp2mW/DL6hIsdeQ2lYc7FhKHW3htvuyUKkFKg61sPejKvzXKLhVuHHqS4pY9bNf0NHaglZvYem3vk1KQf+nIhQU+sKXZ2WgucYcWI1K5MlZN/7ctVgszJgxg9GjR/Od73znkuMLFy7E5/MxduxYfvjDHzJt2rQbvsa1eOaZZ1i/fj2FhYV8+umnJCYmEh4e3mNNYmIizz77LNOnT+fOO++ksLAweOzZZ5/lgQceYNasWcTEXHAvWLp0KR988EGw4P5K6/oTYaimKiZNmiQfPHgwZNevLbKy959VWEaYuOOJa3+67bKe5Z+/+RmmqGiWf++ZQdjhlXF0tPP+L59FrdWx4pn/RhAVjT2c+MGHJ1i5v+6SwtlkZz232QKfJg+bJzJ97hx+umw0AC01nex8pxyfRyIhM5Lblmf2GP6u0DtkWebklo3seud9PE4fxsgMFj/9JCOylTpKhcGlpKSEvLy8a67bUtrCU28cxuuXeryGqEUBjUrkj48VMjc3biC3OmC43W5UKhVqtZo9e/bw1a9+laNHj4ZkL5d7PARBOCTL8nXVISjvylegpuhcyvEKjvafZShFvi6OeinCa/hxpY6lekMyh6MmAFBoO8zenRfC+nGpEcx5dBQ6o5ozlR1sW1mGx9k/RbW3Kj6Phx1vvsauVQHhFRk/mXu//ZQivBSGNHNz41j7zVk8PCWFMJ0aQYAwnZqHp6Sw9puzhq3wAqitrWXy5MmMGzeOb3zjG7z88suh3lKvUbodL4Pb4eVMVSeCACPzzNd1G41OjyCK+Dxu/D4vKnXohmufF1+xaUpaZDhytY6lClMmWsnN6M5ixrfsoaliFolZuQCYE03M/fwotq8sw1rfzea/lzDjgWzCzfrB2vpNQ+fZFra/8Sp1JVX4vSKxaYtY8K8LsIwIC/XWFBSuSarFxE+XjQ5Gxm8WsrOzOXLkSKi30S8oYZHLUFfShizJxGdceZzQZxEE4YLXV4g7Hi8M01aK7Ycj1+pYKg7LozwsC7Ugs+31v3C2tjp4LMJiYN7n84iMM9BpdbHptWJaa69cJKtwKVVHDrD697+hrrgKWQojPvtzLPzKXYrwUlBQ6Df6RXwJgrBQEIRSQRAqBEH4/mWO6wRBePvc8X2CIKT1x3UHimDKMf/6ol7nMUZEAmDvuPoMq4HE7XDQ3tyEqFIRMzIlZPtQ6D3X7FgSBE5EjScsczQ+j4fNr75E25kLZqvGCC3zHs8jMSsSj9PPtrdKOX2sdRB2Przxul3sWvUPdr71Bm1NHah0aYzIe4AFT04nOsF07RMoKCgoXCd9Fl+CIKiAF4FFQD7wsCAIn61Q/xLQJstyFvBb4Fd9ve5AYW93Y63vRq0RScq9sWHUYeZAfVi37VJX38Gi9VzKMWZkakhTnwq957o6ltQqHvnKkyTnFeBxOtjw0gtBF3wAjVbFzPuzyZkSj+SXObC6mmOb6xQriitga6xnzR/+h/L9++iyeTFZZpFcsJg7vzieyNire/wpKCgo3Cj9EfmaAlTIslwly7IHWAnc+5k19wLnzTLeBe4Qhqjz5/mo14icKDQ32C02FMRXS835Ynsl5ThcSbWY+ONjhRg0qksiYGpRwKBR8cfHCkmPi2DWI08EBdjGv/wfjWUlwbWCKDD+zhQmLkpFEAVK955h57vluJVC/CCS38/xTetY++L/b+/eo6K+zoWPfzfDdbiKgDfUjIriDZFbQAMJVYwxOabWWqNN6+rNrJd4XtpzPD2mb2Ny8mKTrJOX2Jy2x5VG60n1AIlWrU2aGG8nBi8EFA0BEVBRLoqKKMh1mP3+MUASQUG5zCDPZ62wGI7EAAAgAElEQVRZMDP7t3+P81N82Hv/nv0mVWWXaLhlZMiobzNmegRzV0zBY4hL150IYW+qzsLf/gl+Ewgv+1i//u2frK8Lu9Abydco4OLXnpe2vtZpG621GbgBdO82wn6ktebCl93bTqgzdpF8tVW2l/peA1p371gyODoR9+yPGRcWSUtzEwf/6x3Onzz+jb7Gzwwg7pmJOLsZqCi6wScbv+RamX0UA+5LJddu8eudXzDtpY8xrfmAaS99zK93fkHJNeuazKryUv7++xRO7f07dTUNWPR4hox6mvHhE4hbNhFnN7kfSQxAhZ/Af86G4+9CUw2grV+Pv2t9vfCT++q2urqaP/zhD/cd1vr166nrRjWAgwcP8tRTT921TU5ODh9++OF9x2IPeuOnS2cjWLfPbXSnDUqplcBKgDFj+n+90o3Kem5ebcDF6Mgwk9c9H+8xxLbJV3NTI9fKLqCUImCs3Ok40HX3jiUHBwMxi5fhYnQn/7ODfJb2Z5rq65gY/Uh7m2EPeZHw46kc3VHMtfJb7P/zaULiA5kYNeyB3H6qs1pHtY1m0jIvsiPrAi9NqqE5/xgWiwVzsxuuXvE4G0cyNXYkUx4Z+UB+JmIQqDoL7/0QmjtJcizN1sd7P4T/lQG+9/YLelvylZiYeF+hrV+/nmeffRajsefT+Dk5OWRlZbFgwYIe92UrvTHyVQqM/trzQKD8Tm2UUo6AN1B1e0da67e11hFa64i2TTH708V8a0ijJg3BoYs1N51pH/m6bpvk69qFErTFwpARo3BylfICg4lycCBswdOEznsS0GTu2saJj/+GtnxV7d7d24XHfhDMxKhhaIvm5L6LZGwreuDqgd2+Q8DX+dZfJrZ8D8c/+Zj6BjPKMAm3IQsxegfy8NPjmBo7ShIvMXAd/h20NN+9TUszHPn9PXe9Zs0aiouLCQ0Nba9w/+///u9ERkYSEhLCSy9Zi4vfunWLJ598khkzZjBt2jTS09N56623KC8vJz4+nvj4+A59f/TRRwQHB/PII4+0b6INkJmZyaxZs5g5cyazZs2ioKCApqYm1q5dS3p6OqGhoaSnp3fazt71xsjX50CQUsoElAHPAMtva/NXYAVwBPgusF/bWWl9rTWlp63bCXW3ttft3H2GAIpb1dW0mM0YHPt32kK2FBrclFJMi0/AxehO5q73+fLgXq5dLGH2Mz/EzcO6BYfB4EDo3DH4j/Yk82/nKC+s5uN3viT8ibGMnOBj4z9B7/jjobM037bFktF8i9CbpwisLwPAYvDkpo7Fy/0hvP3diFk0Hi8/N1uEK0TvOfWedXTrbizNcCodnvx/99T1a6+9Rm5ubntF+T179lBYWEhmZiZaaxYuXMinn37KlStXGDlyJB988AFg3SfR29ublJQUDhw40GG7noaGBn72s5+xf/9+JkyYwNKlS9vfCw4O5tNPP8XR0ZG9e/fyq1/9iu3bt/PKK6+QlZXF7373OwBu3rzZaTt71uPsQGttVkqtAj4GDMAmrfWXSqlXgCyt9V+BjcCflVJFWEe8nunpeXvbjcp6aqqsU47+Yzy7PqATBkdH3H18uFV9nVvV1/Hy69/RO9lMWwAEPTwLz6F+fJb+Zy4VF/Lhf7xB7LIV30jKR00awrxhRo7uKuZa2S0+e6+QsdOGMjNhzIBf6/T1HQIMFjPBtWeYXHsaB23Bogw0uUzF22kKN5oNxM7wY+a8MTg6yVZM4gHQ1M21nN1tdxd79uxhz549zJxp3XWjtraWwsJCYmNjWb16Nf/6r//KU089RWxs7F37OX36NCaTiaCgIACeffZZ3n77bcCauK1YsYLCwkKUUjQ3d55YdredPemVOl9a6w+11hO11uO11utaX1vbmnihtW7QWi/RWk/QWkdpre3ulou2KcfA4CE4dLEr/N3Yat2XpaWFKxdKAFlsL2D4hIks+MfV+I8dR/3NG3zy9n+Qd+gAXx9wdvdx4Vs/mMyMOaMxOCpKcq/x0dtfUHq6w4qAAeVWoxmlLZhuneOJyj1MrcnDQVu44TIGg8c/4O08HbNy4JBrM5FPmiTxEg8O524WAu5uu7vQWvPCCy+Qk5NDTk4ORUVF/OQnP2HixIlkZ2czffp0XnjhBV555ZUu+7rTVP+LL75IfHw8ubm57N69m4aGhh61sydS4R7rX6Kvkq/7m3JsY6t1X1VlF2lpbsLLL6B9ikkMbkYvbxJ+9jxT4r6F1prjH+7iwOa3qam62t5GOSgmPTyceT+dhv9oDxpumTn8l2IythVSU2X/P8Bupy0WJjWX8kTlHiKrszG21FHr6E2NxxwCXGJxVu5UGix84N5Mpbv8+BMPmJDvgUMX9R0dnCBk6d3bdMLT05Oamq92y3j88cfZtGkTtbXWUbSysjIqKyspLy/HaDTy7LPPsnr1ao4fP97p8W2Cg4M5d+4cxcXWmZvU1NT2927cuMGoUdbiCZs3b75jLHdqZ8/kpw/WKcfa6424ut//lGMbW5WbkPVeojMOBgNhTyzk0Wd/jLOrG+Vn8tn95mt8cWAPLeavhuY9fV157PvB1ik4ZwfKzlTz8R9zOfHJhQFRF0xbLJw/dYK/rX+d+LrjeLbUUuvoQZlnNN5uC/BzGE4TmqMuzXzs1kydIyyaeXtFHCEGuFmrwNBF8mVwgpjn77nroUOHMnv2bKZNm8a//Mu/MG/ePJYvX05MTAzTp0/nu9/9LjU1NXzxxRdERUURGhrKunXr+PWvfw3AypUreeKJJzosuHd1deXtt9/mySef5JFHHmHs2LHt7/3yl7/khRdeYPbs2bS0tLS/Hh8fT15eXvuC+zu1s2fKzta9t4uIiNBZWVn9cq4vDpaSf7iC8WH+hM9/qEd9ncvJJiP9z4yZNoO47/+odwLshoPv/pHS/C+ZteT7jAuL7LfzioGjvuYmxz/cxbmcbAC8/AN4+NvfY9i4Cd9oV3eziS8PlXH+1FW0BicXA1Nmj2BCxDAMjvb1+1pzQwNFWUc5ffhTbl23jl47GL358PII/LQJT6xTiiWOLXzuYqa+NXw3JwMf/TyWsUNl2yAxMOTn5zN58uSuGxZ+Yi0n0dL8zcX3Dk7WxOt770JQQt8FOkh0dj2UUtla64juHD+wV9b2gq9POd7vXY5fZ4uRL22xUHneurWMjHyJO3Hz9GL20h8wLjyKzJ3buHmlkk/++DvGTJvBtPh5+I60jgQZvZyJfNJEUMQwTu67yOXzNzm5v5Qzn1cSFBnAuFB/nF17+KOj6qz1tvhT71kX/zp7WKdMZq3qVv2hmmtXKTj8KUVZxzA3NQLW9ZbDxkdRfWUY8c11XKiq44bSZLmYKXW03v3o6KBwMjjwh2fDJPESD6agBGsdryO/t97V2P7va6l1xOse63uJvjHok6/qy3VfTTmO7vlaKVskXzcqL9NUX4fRyxv3IT1PIMWDbcSESTz181/y5f/sJ/fgJ1zIPcmF3JOMnDiZqY/NZVjr3bI+w4zELZvIpbM3OXXgIjcq6zm1v5S8z8oxhfgTFDns/rbf6ew387YK3CdT7/ibeXNDAxe+PMm5E1lcKi6irU6z/9jxDBkZyvVKHy4WNADNDB/uwfTHx7C78jrVOeWoJgvuzo4smjmKn8aaJPESDzbfcdZSEvdYTkL0n0GffF3Mt9b2GjVpCKoHdzm2cXX3wODkTFNDPY11dbj0QjXfrlw+VwRYS0xIgUjRHQZHJ0LmPM6EiIfJO3SAwswjlJ/Jp/xMPv5jTQTPepRRk6fi6OTEiPHeDDd5cencTc4cu8Tl8zcpzLpMUfZlRkzwYcwUX0YEdXMv1HuswN1ibuZSUSFnT3xOaV5u+zo1B4Mjw8ZNw8k4mauljtzItQANuHk6M+WRETwU4ofB4EAUgfzfRdN798MTQogeGtTJl7Wwau9NOYL1llkP36HcuFxB7fVr/ZJ8XSouBOiwdkeIrhi9fYh4ahHT4hMoOHyIgiOHuFJyjisl53B0diZw8jTGhsxk5MRgRoz3ZsR4b6ov13Em8xIX8qooL6ymvLAag6Ni+HgfRgcPuXsi1o0K3DUNivLUVyn3iOJScSEtzU0AaA3ew8bgPmQSTY0juHapbWGtBb/RHoyb4c/oKb52ty5NCCFuN6iTr96ecmzj6etrTb6qrjF01OiuD+gBbbFw+ax15Gv4hIl9ei7x4HJ192BGwhNMiYunKOsY509kca3sIudPHuf8yeM4ubgyImgSfqPHMnT0WMIeH01I/Ggu5ldx8XQV10prKSu4TlnBdZQC7wAjviPdGTrSnaGjPPD0dbWOLN9WgbvZ4kB1kyvXmoxUNRq50uBOjdkFLhZjCTBiabHg4uGHi8d4zM2jaKj3oKEeoAVnN0cemj6UcaH+Up1eCDGgDOrk6+u1vXpjyrFNfxZarSovo6m+Dvchvnj6+nV9gBB34eTiyuTZjzJ59qPUVF2l5FQOJadOcL2irH1tGFhHeH2Gj8Rn+Ah8/LzwHW7kVrXi+mULN6+0cKXkKldKrGuytNY4OJgxODbiUPoUWG6hLfU0WZqpN2s0qrWdQuMAyhlHR3+cVCQuXoEYnNxpq4rhNdSVAJMXw03eDDN5ySiXEGJAGrTJl/UuR+t6r8DgIb3ad38uur9UfAaA4eOC+vxcYnDx9PVj2mNzmfbYXG5evULl+bNcu1jC1YslVF8q53pFGdcryjocp7WmpdmCucmCudmCubkFS0trSZvmeqwL5V0AFxQKR4M7jgYPnA0eODp64mTwRBkccR42BQ9fVzx9XQkY68kwkzdGL+f+/AiEEHfw8ssv4+HhwerVq20dSgenT5/mRz/6EcePH2fdunV2GeOgTb6uX6rjVnXvTzlC/1a5b1vvJVOOoi95+fnj5efPhIiHAWhuaqSqrJSaa1epr7lB/c2b1NdYH+bGRpSDA0op68PBAZQBZ1dPHEszcagswkG54ODggoeTBUeHGhwwY1BmDDTj4VSNZ3gCzk//WG4gEeIB1tLSgsHQ+9t7+fr68tZbb7Fz585e77u3DNrky8nFwPgwf1yMTr065Qjg0Tr919cjXy1mc3tl++HjZeRL9B8nZxeGmca3l6XotqpH4T9nd363Y3vnRoh7DiTxEsKurFu3jnfffZfRo0fj7+9PeHg4xcXFPP/881y5cgWj0cgf//hHgoODKS4u5vvf/z4tLS088cQTpKSkUFtby8GDB/m3f/s3RowYQU5ODnl5eWzZsoW33nqLpqYmHn74Yf7whz9gMBjYs2cPL730Eo2NjYwfP54//elPeHh0vS9lQEAAAQEBfPDBB/3wqdyfQZt8efq69ria/Z14tNbaqr1ehbZYrL/594GrF0toaW7CO2A4bp5efXIOIXqV7zhrHa+uKnBLIUgh7mjp3+59b8buSH8q/Y7vZWdnk5aWxokTJzCbzYSFhREeHs7KlSvZsGEDQUFBHDt2jMTERPbv309SUhJJSUksW7aMDRs2fKOvzMxMcnNzMZlM5Ofnk56eTkZGBk5OTiQmJrJ161YWLFhAcnIye/fuxd3dnddff52UlBTWrl3LL37xCw4cONAhxmeeeYY1a9b0+ufSFwZt8tWXHJ2dcfXwpKG2hls3qtuTsd52uW3KUUa9xEAiFbiFGHAOHTrEokWLMLaWT1q4cCENDQ0cPnyYJUuWtLdrbLTuOHHkyJH2ab/ly5d/Y91VVFQUJpMJgH379pGdnU1kpHVbvPr6egICAjh69Ch5eXnMnj0bgKamJmJiYgB48803+/hP2/ck+eojHr5DrcnX9ao+S74qiloX28t6LzHQSAVuIe7b3Uao+tLtazAtFgs+Pj7k5OTcUz/u7l/tMKG1ZsWKFbz66qvfaLN7924SEhJITU3tcPyDMPIl92n3Ec+h1nVfNX207qu5qZGrF8+jlLr3dTdCCCHEPYiLi2PHjh3U19dTU1PD7t27MRqNmEwm3n//fcCaSJ08aS1HEx0dzfbt2wFIS0u7Y79z5sxh27ZtVFZWAlBVVUVJSQnR0dFkZGRQVGStY1lXV8eZM9YBhzfffJOcnJwOj4GSeIEkX32mfd3Xtat90n/lubNoiwXfkYE4u/V9FX0hhBCDV1hYGEuXLiU0NJTFixcTGxsLwNatW9m4cSMzZsxg6tSp7Nq1C4D169eTkpJCVFQUFRUVeHt7d9rvlClTSE5OZt68eYSEhJCQkEBFRQX+/v5s3ryZZcuWERISQnR0NKdPn+5WrJcuXSIwMJCUlBSSk5MJDAzk5s2bvfNB9BKltbZ1DJ2KiIjQWVlZtg7jvhVnH+PItlQemhHGI8/8sNf7P/7hLvIOHWDqo3OYOf8fer1/IYQQ9iM/P5/JkyfbOoxuq6urw83NDaUUaWlppKamtidmD4LOrodSKltrHdGd42XNVx/5qsp9VZ/0317fa7ys9xJCCGFfsrOzWbVqFVprfHx82LRpk61DsiuSfPWRviy02lhXR1V5GQ4GA/5jTb3evxBCCNETsbGx7eu/REey5quPGL28cTAYaKitobmxoVf7vny2END4jzHh6CzbrQghhBADiSRffUQ5OODuY110f+v69V7tu23KcZjU9xJCCCEGHEm++lBbuYne3mboUmt9rxFS30sIIYQYcCT56kPureUmaqp6r9xE3Y1qbl6txNHZhaGBY3qtXyGEEEL0D0m++pBn26L7Xhz5aptyDDCNw6EPdoMXQgghuvLyyy/zxhtv2DqMTu3atYuQkBBCQ0OJiIjgs88+s3VIHcjdjn3oqzsee6/cxMW8LwAYERTca30KIYQQ/a2lpQVDHwwizJkzh4ULF6KU4tSpU3zve9/rdoHW/iIjX33Iw7d313w1NzVSXpAPwJipIb3SpxBCCNEd69atY9KkScydO5eCggIAiouLmT9/PuHh4cTGxrYnOcXFxURHRxMZGcnatWvx8PAA4ODBg8THx7N8+XKmT58OwJYtW4iKiiI0NJTnnnuOlpYWAPbs2UNMTAxhYWEsWbKE2trabsXp4eHRvg/lrVu3OuxJaQ9k5KsPefh+teZLa93jvwDlBfm0mJvxGz0Wd58hvRGiEEKIAebc4u/2Sb+m7dvu+F52djZpaWmcOHECs9lMWFgY4eHhrFy5kg0bNhAUFMSxY8dITExk//79JCUlkZSUxLJly9iwYcM3+srMzCQ3NxeTyUR+fj7p6elkZGTg5OREYmIiW7duZcGCBSQnJ7N3717c3d15/fXXSUlJYe3atd3aWHvHjh288MILVFZW8sEHH/TuB9ULJPnqQ86ubji7GWmqr6O+5iZGr873tuquC7nWgnVjps3ojfCEEEKIbjl06BCLFi3CaLTuJbxw4UIaGho4fPgwS5YsaW/X2NgIwJEjR9i5cycAy5cvZ/Xq1e1toqKiMJmsBcL37dtHdnY2kZGRANTX1xMQEMDRo0fJy8tj9uzZADQ1NRETEwNYN9buyqJFi1i0aBGffvopL774Inv37u3pR9CrJPnqY15+/ly9WML1ivIeJV/m5mbKTucBknwJIcRgdrcRqr50++yNxWLBx8eHnJyce+rH3d29/XutNStWrODVV1/9Rpvdu3eTkJBAampqh+O7M/LVJi4ujuLiYq5evYqfn989xdmXerTmSynlq5T6RClV2Pq107kwpVSLUiqn9fHXnpxzoGkrhFpxJr9H/VQUnsbc1IjvqNHtC/mFEEKI/hAXF8eOHTuor6+npqaG3bt3YzQaMZlMvP/++4A1kWrbUig6Oprt27cDkJaWdsd+58yZw7Zt26isrASgqqqKkpISoqOjycjIoKioCLBu1H3mjLXG5ZtvvklOTk6HR1viVVRUhNYagOPHj9PU1MTQofb1/2ZPF9yvAfZprYOAfa3PO1OvtQ5tfSzs4TkHlFGTpgBQVtCz5EumHIUQQthKWFgYS5cuJTQ0lMWLFxMbGwvA1q1b2bhxIzNmzGDq1Kns2rULgPXr15OSkkJUVBQVFRV4e3c+8zNlyhSSk5OZN28eISEhJCQkUFFRgb+/P5s3b2bZsmWEhIQQHR3d7TsWt2/fzrRp0wgNDeX5558nPT3d7hbdq7bs8L4OVqoAeExrXaGUGgEc1FpP6qRdrdba4176joiI0FlZWfcdm72wWFrYlvwiTfV1LPznX+HlF3DPfbSYm9mW/CLNjQ333YcQQoiBKz8/n8mTJ9s6jG6rq6vDzc0NpRRpaWmkpqa2J2YPgs6uh1IqW2sd0Z3je7rma5jWugKgNQG7U1bgqpTKAszAa1rrnT0874Dh4GBgRNAkSk6doKwg/74Sp4qiMzQ3NjBkxChJvIQQQti97OxsVq1ahdYaHx8fNm3aZOuQ7EqXyZdSai8wvJO3/s89nGeM1rpcKTUO2K+U+kJrXdzJuVYCKwHGjHlwts4ZFTyFklMnKC/IY/LsR+/5+AtfyJSjEEKIgSM2NrZ9/ZfoqMvkS2s9907vKaUuK6VGfG3asfIOfZS3fj2rlDoIzAQ6JF9a67eBt8E67ditP8EAMDJoMqC4fLaI5qZGnJxdun1si9lMaWtVe0m+hBBCiIGvpwvu/wqsaP1+BdBhQlcpNUQp5dL6vR8wG8jr4XkHFFcPD/xGj8HS0sLlosJ7Ovby2UKaGurxDhiOd8CwPopQCCGEEP2lp8nXa0CCUqoQSGh9jlIqQin1TmubyUCWUuokcADrmq9BlXwBjJxkXZhXVnBvf/QLuacAGfUSQgghHhQ9WnCvtb4GzOnk9Szgp63fHwam9+Q8D4JRk6Zwau9HlBXkdXurIYulhYtftiZf0yX5EkIIIR4EsrF2P/EdGYirhyd1N6q5cflSt46pPFtMY90tPIf64zNsRB9HKIQQQnTPyy+/zBtvvGHrMO7o4MGDhIaGMnXqVB599N5vdOtrsr1QP1EODoycOJmzxzMpK8jDZ3jXyVTB0c8A66iXvRWIE0IIIXqipaUFg8HQ6/1WV1eTmJjIRx99xJgxY9qr59sTGfnqR19Vu+963Vf5mdNc/PIUjs7OTIp+pK9DE0IIIe5q3bp1TJo0iblz51JQUABAcXEx8+fPJzw8nNjY2PYq9MXFxURHRxMZGcnatWvx8LDWWT948CDx8fEsX76c6dOtK5K2bNlCVFQUoaGhPPfcc7S0tACwZ88eYmJiCAsLY8mSJdTW1nYrzv/+7//mO9/5TnvJqoAA+6uPKSNf/WhE0ESUUlw5f5am+jqc3Yydtmsxm/n8r9Y9saZ/63GM3j79GaYQQgg79t5vPu+Tfr/3q8g7vpednU1aWhonTpzAbDYTFhZGeHg4K1euZMOGDQQFBXHs2DESExPZv38/SUlJJCUlsWzZMjZs2PCNvjIzM8nNzcVkMpGfn096ejoZGRk4OTmRmJjI1q1bWbBgAcnJyezduxd3d3def/11UlJSWLt2bZcba585c4bm5mYee+wxampqSEpK4oc//GGvf149IclXP3J2M+L/0DgqzxVTUXiGsSGhnbY7nfE/1Fy7gudQf4LvoyirEEII0ZsOHTrEokWLMBqtgwYLFy6koaGBw4cPs2TJkvZ2jY2NABw5coSdO62b2SxfvpzVq1e3t4mKisJkMgGwb98+srOziYy0Jn719fUEBARw9OhR8vLymD17NgBNTU3ExMQA1o2178ZsNpOdnc2+ffuor68nJiaG6OhoJk6c2BsfRa+Q5KufjZo0hcpzxZQV5HWafNXdqOaL/R8DELlwMQZHuURCCCG+crcRqr50+9pji8WCj48POTk599SPu7t7+/daa1asWMGrr776jTa7d+8mISGB1NTUDsd3NfIVGBiIn58f7u7uuLu7ExcXx8mTJ+0q+ZI1X/2sbd1X+Zl8tMXS4f3sD3dhbmpi9NQQRk4M7u/whBBCiA7i4uLYsWMH9fX11NTUsHv3boxGIyaTiffffx+wJlJtWwpFR0ezfbt1+UxaWtod+50zZw7btm1rXxRfVVVFSUkJ0dHRZGRkUFRUBFg36j5z5gxgHfnKycnp8FizZg0ATz/9NIcOHcJsNlNXV8exY8fsblNySb76mfew4Ri9fWiorSHjvS1cK73Q/t7ls0WUnDqBg6Mj4U9+24ZRCiGEEF8JCwtj6dKlhIaGsnjxYmJjYwHYunUrGzduZMaMGUydOpVdu6wb3axfv56UlBSioqKoqKjA29u7036nTJlCcnIy8+bNIyQkhISEBCoqKvD392fz5s0sW7aMkJAQoqOj2xfzd2Xy5MnMnz+fkJAQoqKi+OlPf8q0adN654PoJUpr+9xCMSIiQmdlZdk6jD5ReOwwx3a+D1g/e7/RDzFpViy5Bz7hRuUlQuY+Qcicx20bpBBCCLuRn59vd6M3d1NXV4ebmxtKKdLS0khNTW1PzB4EnV0PpVS21jqiO8fLgiIbCHp4FiOCJnHm6GcUfX6UqxfPczX9PAAevkOZ+ui3bBugEEII0QPZ2dmsWrUKrTU+Pj5s2rTJ1iHZFUm+bMTDdyhhC55m+tz5nD+Rzekjh6i5UknU09/F4Ohk6/CEEEKI+xYbG9u+/kt0JMmXjTk5uxD08CwmRMXQYjbj6CSJlxBCCPEgkwX3dkIpJYmXEEIIMQhI8iWEEEII0Y8k+RJCCCGE6EeSfAkhhBDinrz88su88cYbtg6jU9evX2fRokXtdb5yc3NtHVIHknwJIYQQot+1tLT0Sb+/+c1vCA0N5dSpU7z77rskJSX1yXl6QpIvIYQQQnRp3bp1TJo0iblz51JQUABAcXEx8+fPJzw8nNjY2PYq9MXFxURHRxMZGcnatWvx8PAA4ODBg8THx7N8+XKmT58OwJYtW4iKiiI0NJTnnnuuPSnbs2cPMTExhIWFsWTJEmpra7sVZ15eHnPmzAEgODiY8+fPc/ny5V79LHpKSk0IIYQQA8iWF37eJ/0+++r6O76XnZ1NWloaJ06cwGw2ExYWRnh4OCtXrmTDhg0EBQVx7NgxEhMT2b9/P0lJSSQlJbFs2TI2bE89TZAAAAeCSURBVNjwjb4yMzPJzc3FZDKRn59Peno6GRkZODk5kZiYyNatW1mwYAHJycns3bsXd3d3Xn/9dVJSUli7dm2XG2vPmDGDv/zlLzzyyCNkZmZSUlJCaWkpw4YN6/XP7H5J8iWEEEKIuzp06BCLFi3CaDQCsHDhQhoaGjh8+DBLlixpb9fY2AjAkSNH2LlzJwDLly9n9erV7W2ioqIwmUwA7Nu3j+zsbCIjIwGor68nICCAo0ePkpeXx+zZswFoamoiJiYGsG6sfTdr1qwhKSmJ0NBQpk+fzsyZM3F0tK90x76iEUIIIcRd3W2Eqi8ppb7x3GKx4OPjQ05Ozj314+7u3v691poVK1bw6quvfqPN7t27SUhIIDU1tcPxXY18eXl58ac//am9f5PJ1J7s2QtZ8yWEEEKIu4qLi2PHjh3U19dTU1PD7t27MRqNmEwm3n//fcCa6LRtKRQdHc327dsBSEtLu2O/c+bMYdu2bVRWVgJQVVVFSUkJ0dHRZGRkUFRUBFg36j5z5gxgHfnKycnp8FizZg0A1dXVNDU1AfDOO+8QFxeHl5dXH3wq90+SLyGEEELcVVhYGEuXLiU0NJTFixcTGxsLwNatW9m4cSMzZsxg6tSp7Nq1C4D169eTkpJCVFQUFRUVeHt7d9rvlClTSE5OZt68eYSEhJCQkEBFRQX+/v5s3ryZZcuWERISQnR0dPti/q7k5+czdepUgoOD+fvf/85vf/vb3vkQepHSWts6hk5FRETorKwsW4chhBBC2Fx+fj6TJ0+2dRjdVldXh5ubG0op0tLSSE1NbU/MHgSdXQ+lVLbWOqI7x8uaLyGEEEL0quzsbFatWoXWGh8fHzZt2mTrkOyKJF9CCCGE6FWxsbHt679ER7LmSwghhBCiH0nyJYQQQgwA9rpGe7DpjesgyZcQQghh51xdXbl27ZokYDamtebatWu4urr2qB9Z8yWEEELYucDAQEpLS7ly5YqtQxn0XF1dCQwM7FEfPUq+lFJLgJeByUCU1rrT2hBKqfnAbwED8I7W+rWenFcIIYQYTJycnOyuSru4fz2ddswFvgN8eqcGSikD8HvgCWAKsEwpNaWH5xVCCCGEGJB6NPKltc6Hjvs93SYKKNJan21tmwY8DeT15NxCCCGEEANRfyy4HwVc/Nrz0tbXhBBCCCEGnS5HvpRSe4Hhnbz1f7TW3dkroLNhsU5v11BKrQRWtj6tVUoVdKP/nvIDrvbDeUT3yPWwP3JN7ItcD/sj18T+2OKajO1uwy6TL6313J7FQikw+mvPA4HyO5zrbeDtHp7vniilsrq7F5Poe3I97I9cE/si18P+yDWxP/Z+Tfpj2vFzIEgpZVJKOQPPAH/th/MKIYQQQtidHiVfSqlFSqlSIAb4QCn1cevrI5VSHwJorc3AKuBjIB94T2v9Zc/CFkIIIYQYmHp6t+MOYEcnr5cDC772/EPgw56cqw/16zSn6JJcD/sj18S+yPWwP3JN7I9dXxMlWxUIIYQQQvQf2dtRCCGEEKIfDdrkSyk1XylVoJQqUkqtsXU8g51SarRS6oBSKl8p9aVSKsnWMQnrDhVKqRNKqb/ZOhYBSikfpdQ2pdTp1n8rMbaOabBTSv2i9WdWrlIqVSnVsx2XxT1RSm1SSlUqpXK/9pqvUuoTpVRh69chtoyxM4My+ZItj+ySGfhnrfVkIBp4Xq6JXUjCeqOMsA+/BT7SWgcDM5BrY1NKqVHA/wYitNbTsO5f/Ixtoxp0NgPzb3ttDbBPax0E7Gt9blcGZfLF17Y80lo3AW1bHgkb0VpXaK2Pt35fg/U/FdkJwYaUUoHAk8A7to5FgFLKC4gDNgJorZu01tW2jUpgvXHNTSnlCBi5Qx1L0Te01p8CVbe9/DTwX63f/xfw7X4NqhsGa/IlWx7ZMaXUQ8BM4JhtIxn01gO/BCy2DkQAMA64AvypdSr4HaWUu62DGsy01mXAG8AFoAK4obXeY9uoBDBMa10B1l/sgQAbx9PBYE2+ur3lkehfSikPYDvwc631TVvHM1gppZ4CKrXW2baORbRzBMKA/9RazwRuYYfTKYNJ61qipwETMBJwV0o9a9uoxEAwWJOvbm95JPqPUsoJa+K1VWv9F1vHM8jNBhYqpc5jnZb/llJqi21DGvRKgVKtdduI8DasyZiwnbnAOa31Fa11M/AXYJaNYxJwWSk1AqD1a6WN4+lgsCZfsuWRnVFKKaxrWfK11im2jmew01q/oLUO1Fo/hPXfx36ttfxGb0Na60vARaXUpNaX5gB5NgxJWKcbo5VSxtafYXOQmyDswV+BFa3frwB22TCWTvWowv1ApbU2K6XatjwyAJtkyyObmw38APhCKZXT+tqvWndHEEJY/SOwtfWXxrPAj2wcz6CmtT6mlNoGHMd6x/YJ7Lyy+oNGKZUKPAb4tW53+BLwGvCeUuonWBPkJbaLsHNS4V4IIYQQoh8N1mlHIYQQQgibkORLCCGEEKIfSfIlhBBCCNGPJPkSQgghhOhHknwJIYQQQvQjSb6EEEIIIfqRJF9CCCGEEP1Iki8hhBBCiH70/wEVlCGyIFOVyQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faf930ce828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "clf = LinearRegression()\n",
    "preds = np.zeros((4, 100))\n",
    "X_input = np.linspace(0, 10, 100)\n",
    "powers = [1, 3, 6, 9]\n",
    "\n",
    "for i in range(len(powers)):\n",
    "    poly = PolynomialFeatures(powers[i])\n",
    "    X_train_poly = poly.fit_transform(X_train[None].T)\n",
    "    X_input_poly = poly.fit_transform(X_input[None].T)\n",
    "    clf.fit(X_train_poly, y_train)\n",
    "    preds[i,:] = clf.predict(X_input_poly)\n",
    "    \n",
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(X_train, y_train, 'o', label='training data', markersize=10)\n",
    "plt.plot(X_test, y_test, 'o', label='test data', markersize=10)\n",
    "for i,degree in enumerate([1,3,6,9]):\n",
    "    plt.plot(np.linspace(0,10,100), preds[i], alpha=0.8, lw=2, label='degree={}'.format(degree))\n",
    "plt.ylim(-1,2.5)\n",
    "plt.legend(loc=4);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-regularized Linear Regression model\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Non-regularized linear regression:  -4.311965180969886\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.regression import r2_score\n",
    "\n",
    "poly = PolynomialFeatures(12)\n",
    "lr_clf = LinearRegression()\n",
    "    \n",
    "X_train_poly = poly.fit_transform(X_train[None].T)\n",
    "X_test_poly = poly.fit_transform(X_test[None].T)\n",
    "    \n",
    "lr_clf.fit(X_train_poly, y_train)\n",
    " \n",
    "lr_score = r2_score(y_test, lr_clf.predict(X_test_poly))   \n",
    "\n",
    "print('Non-regularized linear regression: ', lr_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lasso Regression model (with parameters alpha=0.01, max_iter=10000)\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lasso regression score:  0.8406625614750349\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "lasso_clf = Lasso(alpha=0.01, max_iter=10000)\n",
    "    \n",
    "lasso_clf.fit(X_train_poly, y_train)\n",
    "    \n",
    "lasso_score = r2_score(y_test, lasso_clf.predict(X_test_poly))\n",
    "\n",
    "print('Lasso regression score: ', lasso_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification\n",
    "----\n",
    "\n",
    "For this part of the notebook, we will be working with the UCI Mushroom Data Set. The data will be used to train a model to predict whether or not a mushroom is poisonous. The following attributes are provided:\n",
    "* **cap-shape**: bell=b, conical=c, convex=x, flat=f, knobbed=k, sunken=s\n",
    "* **cap-surface**: fibrous=f, grooves=g, scaly=y, smooth=s\n",
    "* **cap-color**: brown=n, buff=b, cinnamon=c, gray=g, green=r, pink=p, purple=u, red=e, white=w, yellow=y\n",
    "* **bruises?**: bruises=t, no=f\n",
    "* **odor**: almond=a, anise=l, creosote=c, fishy=y, foul=f, musty=m, none=n, pungent=p, spicy=s\n",
    "* **gill-attachment**: attached=a, descending=d, free=f, notched=n\n",
    "* **gill-spacing**: close=c, crowded=w, distant=d\n",
    "* **gill-size**: broad=b, narrow=n\n",
    "* **gill-color**: black=k, brown=n, buff=b, chocolate=h, gray=g, green=r, orange=o, pink=p, purple=u, red=e, white=w, yellow=y\n",
    "* **stalk-shape**: enlarging=e, tapering=t\n",
    "* **stalk-root**: bulbous=b, club=c, cup=u, equal=e, rhizomorphs=z, rooted=r, missing=?\n",
    "* **stalk-surface-above-ring**: fibrous=f, scaly=y, silky=k, smooth=s\n",
    "* **stalk-surface-below-ring**: fibrous=f, scaly=y, silky=k, smooth=s\n",
    "* **stalk-color-above-ring**: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y\n",
    "* **stalk-color-below-ring**: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y\n",
    "* **veil-type**: partial=p, universal=u\n",
    "* **veil-color**: brown=n, orange=o, white=w, yellow=y\n",
    "* **ring-number**: none=n, one=o, two=t\n",
    "* **ring-type**: cobwebby=c, evanescent=e, flaring=f, large=l, none=n, pendant=p, sheathing=s, zone=z\n",
    "* **spore-print-color**: black=k, brown=n, buff=b, chocolate=h, green=r, orange=o, purple=u, white=w, yellow=y\n",
    "* **population**: abundant=a, clustered=c, numerous=n, scattered=s, several=v, solitary=y\n",
    "* **habitat**: grasses=g, leaves=l, meadows=m, paths=p, urban=u, waste=w, woods=d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "mush_df = pd.read_csv('datasets/mushrooms.csv')\n",
    "mush_df2 = pd.get_dummies(mush_df)\n",
    "\n",
    "X_mush = mush_df2.iloc[:,2:]\n",
    "y_mush = mush_df2.iloc[:,1]\n",
    "\n",
    "# use the following (X_train2, X_test2, y_train2, y_test2) for questions 5 through 7:\n",
    "X_train2, X_test2, y_train2, y_test2 = train_test_split(X_mush, y_mush, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision tree classifier (default parameters)\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 5 most important features:  ['odor_n', 'stalk-root_c', 'stalk-root_r', 'spore-print-color_r', 'odor_l']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "clf = DecisionTreeClassifier(random_state=0)\n",
    "clf.fit(X_train2, y_train2)\n",
    "    \n",
    "idx_features = np.argsort(clf.feature_importances_)[-5:]\n",
    "\n",
    "print('The 5 most important features: ', list(X_train2.columns[idx_features][::-1].values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support vector machines\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import validation_curve\n",
    "\n",
    "clf = SVC(kernel='rbf', C=1, random_state=0)\n",
    "param_range = np.logspace(-4,1,6)\n",
    "train_scores, test_scores = validation_curve(clf, X_test2, y_test2, 'gamma', param_range, \n",
    "                                             cv=3, scoring='accuracy', n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl4VPXZxvHvkwRMQPZFhKhRlB0SQkBZBFFERIGiFkjVUmURBRdsqyhYkNoWaqtVoVAqIvoqiKUqVhAFFURUQBZlEQiIGNHKjgUjBH7vHzMZJyEhk5BZc3+uiyuZmZM5T1huTs495zfmnENERGJLXLgHEBGRsqdwFxGJQQp3EZEYpHAXEYlBCncRkRikcBcRiUEKdxGRGKRwFxGJQQp3EZEYlBCuHdeuXdulpKSEa/ciIlHpk08+2eOcq1PcdmEL95SUFFatWhWu3YuIRCUz+zKQ7XRaRkQkBincRURikMJdRCQGKdxFRGKQwl1EJAYVG+5m9oyZfWdm64t43MzsSTPLMrNPzSy97McUEZGSCOTI/Vmgxykevxq4yPtrKDDl9McSEZHTUezr3J1zS80s5RSb9AGec5736/vIzKqb2dnOuW/KaMYi7f1uF58vmMqPJJCTUI0fEqqRU8H7MaEaR+Mrg1mwxxARKZErmp5F6jnVg7qPsriIqQHwld/tbO99J4W7mQ3Fc3TPueeee1o7PXH8BF89fSMdj64ucpujLp6DnMl+dyYHqOL9eCb7XRXvxzPZz5kccFXYRxUOeB8/TvxpzSYicip1qyZGRbgXdmhc6LtuO+emAdMAMjIyTuuduZfOncRlR1ezpvkoWvccAkf2wQ/78n2s+MM+6hzZR50f9sGR/d77sz0fjx8t+snPqAaVakBSTahU86ePlWpDy+uh5gWnM7qISNCVRbhnA+f43U4GdpXB8xZp2xdfkLphIlmJzUi7/j6Ii4fKtQN/Aufg6OGT/jPgh/1wZG/++47shT1bPY/9eAiWPQ5XT4TWN+mUj4hErLII93nACDObDVwMHAzm+fbc4yfYOetuzrUcyPwHFleKUyhmcMaZnl/VS3B66GA2vDIM5o2ArEXQ62+QVKPk+xcRCbJAXgo5C/gQaGxm2WY2yMyGmdkw7ybzge1AFvBP4I6gTQtk79hC16NL+Pz8gdRIaRXMXZ2sWjL88jW4Yix8/h+Y0hF2LAvtDCIiAQjk1TKZxTzugOFlNlExTvx4CIBjdVqEapf5xcXDpffCBV1g7hB49lroNBK6PgjxFcIzk4hIAVF3heqJ4ycAsLgwj96gDdy2FFrfCMseg+ndYe+28M4kIuIVdeHOieOej3FhW4r+J2ecCX0mw89nwr7tMPVSWP28p7AVEQmjqAv3E8c94R5XmiI1WJr/DG7/ABqke8rWl3/leXWNiEiYRF+4n8j1fBIfQeEOKltFJKJEXbg772mZOIvA0fPK1kFvQUKip2xd9DAcPxbuyUSknInAhDw1d9xz5G6RcM69KCeVrVeqbBWRkIq6cD/hPK+WIdyvlilOvrL1C5WtIhJSEZ6QJ3O+QjWCj9z9qWwVkTCIvnDPO+ceaYXqqahsFZEQi9pwt2gKd1DZKiIhFYXh7ilU4yzKwj2Pr2y9SWWriARNFIa7d/mBaDty93fGmdBnEvR7TmWriARFFIZ7lBWqp9KsT4GydaDKVhEpE9Eb7tF85O4vr2ztNg4+f8NTtn7xfrinEpEop3CPBHHxnmWD88rWmb1g0TiVrSJSalEb7hF9hWpp5StbH1fZKiKlFsXhHkNH7v5UtopIGYi6cMdF4JK/waCyVUROQ/SFeyyecy+KylYRKaWoC/e817nHxcfgOffC+MrWt1W2ikjAojDcy9GRu78G6SpbRSRgURfuvtMysX7OvTAqW0UkQNEX7t713GPiCtXSatYHbl+ev2w9si/cU4lIBInCcC/HR+7+qjXIX7ZO7aSyVUR8oi/cvZxZuEcIP5WtIlKEqA138aOyVUQKULjHCpWtIuJH4R5rVLaKCAr32KSyVaTcU7jHqqLK1tyj4Z5MREJA4R7r8srW9Js9Zesz3VW2ipQDCvfy4IwzofdTKltFypGAwt3MepjZZjPLMrNRhTx+npktNrNPzew9M0su+1HltBUsW+f8UmWrSIwqNtzNLB6YDFwNNAMyzaxZgc3+AjznnGsFjAf+VNaDShnxL1s3z1fZKhKjAjlybwdkOee2O+eOArOBPgW2aQYs9n7+biGPSyTJK1sHL4IKSSpbRWJQIOHeAPjK73a29z5/64DrvZ/3BaqYWa3TH0+Cqn5rla0iMSqQcC9sEZeCTdxvgC5mtgboAnwN5J70RGZDzWyVma3avXt3iYeVIKhYWWWrSAwKJNyzgXP8bicDu/w3cM7tcs5d55xrDYz23new4BM556Y55zKccxl16tQ5jbGlzKlsFYkpgYT7SuAiMzvfzCoCA4B5/huYWW0zy3uuB4BnynZMCQmVrSIxo9hwd87lAiOAhcAmYI5zboOZjTez3t7NLgM2m9kW4CzgD0GaV4JNZatITAjo7Yycc/OB+QXu+53f5/8C/lW2o0lY5ZWtb47ylK3b34PrnobaF4Z7MhEJgK5QlaIVLFv/cSmsfk5lq0gUULhL8XxlaxuYd6fKVpEooHCXwPjK1odVtopEgSgMd50SCJu4eOh0j8pWkSgQheHupTfIDp/CrmzdkxXuqUTET/SGu4SXr2x9XmWrSARSuMvpadbbU7YmZ6hsFYkgCnc5fdUawM1+ZeuUjvDF0nBPJVKuKdylbMTF/VS2VqwEM3vD22NVtoqEicJdypZ/2frB32D6lSpbRcJA4S5lz79s3b9DZatIGCjcJXhUtoqEjcJdgktlq0hYKNwl+FS2ioScwl1Cx1e2/lJlq0iQKdwltCpWht5PesrWA1+qbBUJEoW7hIfKVpGgUrhL+FSt7ylbrxwPmxeobBUpQwp3Ca+4OOh4Nwx+W2WrSBlSuEtkUNkqUqYU7hI5VLaKlBmFu0Qela0ip03hLpFJZavIaVG4S+RS2SpSalEX7jr9Wg6pbBUpsagLd8OT7np/7HImr2zt/38/la2fzNT/9iJFiLpw/4nSvVxq2uunsvX1u1S2ihQhisNdyi2VrSLFUrhLdFLZKnJKCneJbipbRQqlcJfop7JV5CQKd4kdKltFfAIKdzPrYWabzSzLzEYV8vi5Zvauma0xs0/NrGfZjyoSAJWtIkAA4W5m8cBk4GqgGZBpZs0KbDYGmOOcaw0MAP5e1oOKBExlq0hAR+7tgCzn3Hbn3FFgNtCnwDYOqOr9vBqwq+xGFCklla1SjgUS7g2Ar/xuZ3vv8zcOuMnMsoH5wJ1lMp3I6VLZKuVUIOFe2KWgBf9lZALPOueSgZ7A82Z20nOb2VAzW2Vmq3bv3l3yaUVK66Sy9WaVrRLTAgn3bOAcv9vJnHzaZRAwB8A59yGQCNQu+ETOuWnOuQznXEadOnVKN7FIaeUrW9/0lK3bl4R7KpGgCCTcVwIXmdn5ZlYRT2E6r8A2O4ErAMysKZ5w16G5RB5f2brIU7Y+1wfe/p3KVok5xYa7cy4XGAEsBDbheVXMBjMbb2a9vZv9GhhiZuuAWcCvnNNJTYlg9dM8ZWubgfDBEypbJeYkBLKRc24+nqLU/77f+X2+EehYtqOJBFnFytDrCbiwm+ft/P5xKfSY4Hl1jdaUliinK1RFVLZKDFK4i4DKVok5CneRPCpbJYYo3EUKKrRs3RruqURKROEuUpi8stV3ZWtnXdkqUSUKw13/uCSEfGVrW5WtElWiMNw9TC9Vk1CpWh9uflVlq0SVqA13kZDKV7ZWVtkqEU/hLlIS9dPgtiUqWyXiKdxFSkplq0QBhbtIaTXtBbd/qLJVIpLCXeR0VD1bZatEJIW7yOlS2SoRSOEuUlZUtkoEUbiLlKVCy9ZnVbZKyCncRYIhX9l6N7x0k8pWCSmFu0iw+MrW38OWhSpbJaQU7iLBFBcHHe9S2Sohp3AXCQVf2forla0SEgp3kVCpWBl6/Q36v6CyVYJO4S4Sak2vVdkqQadwFwkHla0SZAp3kXAprGx96yGVrVImFO4i4eZfti5/EqZ3U9kqp03hLhIJ8pWtO1W2ymlTuItEEpWtUkYU7iKRRmWrlIHoC3f9mCrlgcpWOU3RF+5eZhbuEUSCr34a3LZUZauUWNSGu0i5UbGSX9n6lcpWCYjCXSRaNL0Wbl+uslUConAXiSZ5ZWv3R7xlawfY/l64p5IIpHAXiTZxcdDhThiyGM6oAs/9TGWrnCSgcDezHma22cyyzGxUIY8/bmZrvb+2mNmBsh9VRPI5OxWG6spWKVyx4W5m8cBk4GqgGZBpZs38t3HOjXTOpTnn0oCngH8HY1gRKUBlqxQhkCP3dkCWc267c+4oMBvoc4rtM4FZZTGciARIZasUEEi4NwC+8rud7b3vJGZ2HnA+8E4Rjw81s1Vmtmr37t0lnVVETkVlq/gJJNwLu1qoqJ/5BgD/cs4dL+xB59w051yGcy6jTp06gc4oIoFS2SpegYR7NnCO3+1kYFcR2w5Ap2REwk9la7kXSLivBC4ys/PNrCKeAJ9XcCMzawzUAD4s2xFFpFRUtpZrxYa7cy4XGAEsBDYBc5xzG8xsvJn19ts0E5jtnP7miESUvLL1nHYqW8uRhEA2cs7NB+YXuO93BW6PK7uxRKRMVT0bbnoFPpoMix72lK19p8IFl4V7MgkSXaEqUl6obC1XFO4i5Y3K1nJB4S5SHqlsjXkKd5HyTGVrzFK4i5R3eWWrrmyNKQp3EVHZGoMU7iLyk8LK1t1bwj2VlEIUhru38NEbZIsER17ZOuDFn8rWVTNUtkaZKAx3D0W7SJA1ucZTtp57MfznHpWtUSZqw11EQkBla9RSuIvIqalsjUoKdxEJTF7ZmnGLytYooHAXkcBVrATXPq6yNQoo3EWk5FS2RjyFu4iUjsrWiKZwF5HSU9kasRTuInL6CpatT1+hsjXMFO4iUjb8y9aD2Spbw0zhLiJly1e2XqKyNYwU7iJS9qqeDTf9G7r/4aeyddu74Z6qXFG4i0hwxMVBhxEw5B1P2fr8z+CtMSpbQ0ThLiLBdXYrb9l6Kyx/SmVriCjcRST4VLaGnMJdREKnyTVwx4cqW0NA4S4ioVWlnsrWEFC4i0joqWwNOoW7iISPytagUbiLSHgVWrY+o7L1NEVfuOsPXCQ25StbR3rK1sN7wz1V1Iq+cPcy01tki8Qc/7J161sqW09D1Ia7iMSovLJ18GJIrKqytZQU7iISmXxl6yCVraUQULibWQ8z22xmWWY2qoht+pnZRjPbYGYvlu2YIlIuVawE1z4GA2apbC2hYsPdzOKBycDVQDMg08yaFdjmIuABoKNzrjlwTxBmFZHyqklPla0lFMiRezsgyzm33Tl3FJgN9CmwzRBgsnNuP4Bz7ruyHVNEyj2VrSUSSLg3AL7yu53tvc9fI6CRmX1gZh+ZWY+yGlBExKfIsvXHcE8WcQIJ98Jec1jwhFcCcBFwGZAJPG1m1U96IrOhZrbKzFbt3r27pLOKiHicVLZ2U9laQCDhng2c43c7GdhVyDavOeeOOee+ADbjCft8nHPTnHMZzrmMOnXqlHZmERGVrcUIJNxXAheZ2flmVhEYAMwrsM2rQFcAM6uN5zTN9rIcVESkUAXL1tk3qmwlgHB3zuUCI4CFwCZgjnNug5mNN7Pe3s0WAnvNbCPwLvBb55x+d0UkNPzL1qy3VbbiOVdeLOfcfGB+gft+5/e5A+71/pIIcOzYMbKzs8nJyQn3KFJOJCYmkpycTIUKFcIzQF7Zen5nmDvYU7Z2uBMufwgSzgjPTGEUULhL9MnOzqZKlSqkpKRoHR4JOucce/fuJTs7m/PPPz+8w5zdCoa+53kVzfKnYPsSuH461GkU3rlCTMsPxKicnBxq1aqlYJeQMDNq1aoVOT8p+peth74ul2Wrwj2GKdgllCLy71uTnnD78nJZtircJSj27t1LWloaaWlp1KtXjwYNGvhuHz0a2Op+t9xyC5s3by7Rft944w3atGlDs2bNaNKkCffffz+LFi3i0ksvzbfdsWPHqFu3Lt99p4upY15e2XrVH8tV2apwl6CoVasWa9euZe3atQwbNoyRI0f6blesWBHwnKc9ceJEkc8xY8YMGjduHPA+161bxz333MOsWbPYuHEj69evJyUlha5du7Jt2zays7N92y5cuJDWrVtTt27d0n+TUSw3NzfcI4RWXBy0H+69srWap2xdODqmr2xVuEtIZWVl0aJFC4YNG0Z6ejrffPMNQ4cOJSMjg+bNmzN+/Hjftp06dWLt2rXk5uZSvXp1Ro0aRWpqKu3bty/0iHvixIk89NBDNGrkKc4SEhK4/fbbiY+P54YbbuCll17ybTt79mwyMzNPeo6bbrqJ4cOH07VrVxo2bMjSpUsZOHAgTZo0YdCgQb7tFixYQPv27UlPT6d///4cPnwYgLFjx9K2bVvf9+i853g7derEqFGjaNeuHY0bN2b58uUn7fvQoUNcfvnlpKen06pVK/7zn//4HpsxYwatWrUiNTWVW265BYBvv/2WPn36+O7/+OOPycrKIi0tzfd1EyZM4JFHHvHNMHr0aDp37sykSZN47bXXuPjii2ndujXdu3f3/Z5+//33DBw4kJYtW9KqVSteffVV/vGPf/Db3/7W97xTpkzhvvvuK/LPOWLlla0Zg+DDSTF9ZateLVMOPPz6BjbuOlSmz9msflXG9mpeqq/duHEjM2bMYOrUqYAngGrWrElubi5du3blhhtuoFmzfAuPcvDgQbp06cKECRO49957eeaZZxg1Kv/q0+vXr2f06NGF7jMzM5O77rqLX//61+Tk5LBw4UImT55c6LYHDx7k3XffZe7cufTq1YsPP/yQJk2akJ6ezvr166lbty4TJkxg8eLFVKpUiT/84Q888cQTPPjgg9x99908/PDDOOf4xS9+wZtvvsnVV18NeH5SWbFiBfPmzWP8+PG8+eab+fablJTEa6+9RpUqVfjuu+/o2LEj1157LevWrWPixIksX76cmjVrsm/fPgCGDx/OlVdeyYgRI8jNzeXIkSPFnmY6dOgQS5cuBWD//v307t0bM2Pq1Kn89a9/ZeLEiYwbN446derw2Wef4ZzjwIEDJCQkkJaWxp/+9CcSEhKYMWMGzz777Cn3FbHyytYLu8G8EZ6ytccfoc0tEIm9QSkp3CXkGjZsSNu2bX23Z82axfTp08nNzWXXrl1s3LjxpHBPSkryhWSbNm14//33S7TP9u3bs3fvXrZt28aaNWu49NJLqVatWqHb9urVC4CWLVtSv3593yzNmjVjx44dZGVlsXHjRjp06ADA0aNH6dSpEwCLFy/m0UcfJScnhz179tCmTRvf3Nddd51v/h07dpy0X+cc999/P8uWLSMuLo6vvvqKPXv28M4779C/f39q1qwJ4Pv43nvvMXv2bMDzU0rVqlWLDfcBAwb4Pt+5cyf9+vXj22+/5ccff/T9xLNo0SJeffVVwFOS1qhRA4DOnTuzYMECLrjgAuLj40/6M4o6TXpCg+XwyjBP2bp1EfR+CirXCvdkZSLqwj037gz2uzMpf5cklF5pj7CDpXLlyr7Pt27dyhNPPMGKFSuoXr06N910U6Evp8s7Tw8QHx9f6Dnj5s2b88knn9C8eeHfb//+/Zk9ezZr1qwp9JRMnjPO8PztiouL832edzs3NxfnHD169OD555/P93VHjhxhxIgRrF69mgYNGjBmzJh830vecxU1/3PPPcfBgwdZvXo1CQkJJCcnk5OTg3OuyFeiFLw/ISEhX4+Rk5NDQsJP/8z9f++HDx/Ogw8+SM+ePVm0aBETJkwAKHJ/gwcP5rHHHiMlJcV3aijq5ZWtH0+BReM8ZWvfqdCwa7gnO21Rd869+mUjmHfVB1RIqhLuUaQMHDp0iCpVqlC1alW++eYbFi5cWOrnuu+++3jkkUfIysoC4Pjx4zz22GO+xzMzM3nuuedYunQp1157ban306FDB5YsWcL27Z7lkw4fPszWrVv54YcfiIuLo3bt2nz//ffMnTu3RM978OBB6tatS0JCAm+//TZff/01AN26dWP27Nm+0zF5H7t27eo7tXX8+HEOHTpEvXr12LVrF/v37ycnJ4c33njjlPtr0KABzjlmzpzpu7979+5MmjQJ8AT9/v37AejYsSPbtm3j5Zdfpn///iX63iJajJatURfuLRpUY2CHFCrER93oUoj09HSaNWtGixYtGDJkCB07diz1c7Vu3Zq//OUv9OvXj6ZNm9KyZUv8l5Zu1aoVFSpUoHv37iQlJZV6P2eddRbTp0+nf//+pKam0qFDB7Zs2UKtWrUYOHAgLVq0oG/fvlx88cUlet6bb76Z5cuXk5GRwcsvv8xFF13km/u+++6jc+fOpKWl+YrNSZMmsXDhQlq2bElGRgaff/45iYmJPPjgg7Rt25bevXuf8tTJuHHj6Nu3L126dOGss87y3T927Fj++9//0qJFC9LS0vKdArvhhhvo3Llzkae0olqMla3mwnTFVkZGhlu1alVY9l0ebNq0iaZNm4Z7DIkxPXr04IEHHqBLly6FPh4zf+8+n+8pW48eibiy1cw+cc5lFLedDn9FpFh79+6lUaNG1KhRo8hgjyl5V7ae1z5qr2yNukJVREKvVq1abNkSvacoSqVKPbhxbtSWrTpyFxEpShSXrQp3EZHi5JWtbQd7y9YrIr5sVbiLiASiYiW45q/eZYR3Rfwywgp3EZGSiJKyVeEuQVEWS/4CPPPMM3z77beFPuac489//jONGzf2vSb7hRdeYMyYMTz00EP5tl21ahWtWrU6re9JxCevbI3gZYQV7hIUgSz5G4hThfvkyZN59913WbVqFevXr+e9997j+PHjZGZm5lsBEopeBbK8OH78eLhHiD0RXrYq3CXkZs6cSbt27UhLS+OOO+7gxIkT5ObmcvPNN9OyZUtatGjBk08+yUsvvcTatWvp379/oUf8f/zjH5k6dSpVqniWoqhevTq//OUvad68OYmJiXzyySeA5wj/5ZdfzrdoVp7k5GRGjx7NJZdcQtu2bVm9ejXdu3enYcOG/POf//RtN2HCBNq1a0erVq3yLUvcq1cv2rRpQ/PmzXn66acBAl6i+KOPPqJ9+/a0bt2ajh07snXrVt/Xjxw5khYtWtCqVSv+/ve/A/Dxxx/Tvn17UlNTufjiizly5AhPP/0099xzj+85e/TowbJly3wzjBkzhnbt2rFixYoilyPesmULl19+OampqaSnp7Njxw4yMzPzLV3Qv39/5s+fX4I/5XIkQstWvc69PFgwCr79rGyfs15LuHpCib9s/fr1vPLKKyxfvpyEhASGDh3K7NmzadiwIXv27OGzzzxzHjhwgOrVq/PUU08xadKkfGuUg2e52mPHjnHeeecVup/MzExmz55NmzZt+OCDD6hfv36Rb9yckpLCRx99xJ133smgQYNYtmwZ//vf/0hNTWXIkCHMnz+fnTt38vHHH+Oco2fPnixfvpwOHTowc+ZMatasyZEjR8jIyOD666+nSpUqAS1R3LRpU5YtW0Z8fDxvvvkmY8aM4aWXXmLKlCns2rWLdevWER8fz759+8jJyWHAgAHMnTuX9PR0Dh48mG9Rs8IcPHiQ9PR033rujRs3LnQ54szMTMaNG0evXr3IycnhxIkTDB48mClTpnDNNdewf/9+Vq5cyYsvvhjQn3G5lFe2XtgNXhseEcsI68hdQmrRokWsXLmSjIwM0tLSWLJkCdu2bePCCy9k8+bN3H333SxcuLDYtUuKWzYjMzOTOXPm4Jwr9pRM7969Ac8Sv5dccgmVK1fmrLPOIi4ujv/973+89dZbLFiwgNatW5Oenk5WVpbvgp7HH3/cd3SenZ3Ntm3bgJOXKC5sid8DBw5w3XXX0aJFC37zm9+wYcMG3+/RsGHDiI+PBzxL/G7atIlzzz2X9PR0AKpVq+Z7vCgVK1akb9++vtuLFy+mXbt2pKamsmTJEjZs2MD+/fvZs2ePb5njxMREKlWqxOWXX87GjRvZu3cvL7zwAv369St2fwI0vjpiylYduZcHpTjCDhbnHLfeeiu///3vT3rs008/ZcGCBTz55JPMnTuXadOmFfk8NWvWpEKFCuzcuZNzzz33pMdTUlKoX78+77//Pq+88orvFE1hAlnid8yYMfneiQk8Ibx06VI++ugjkpKS6NSpk2+J30CWKB49ejRXXXUVd9xxB1lZWfTo0cP3e1Rwyd2iluEtbInfPElJSb6vOdVyxIU9r5lx44038uKLL/Lss8/qqL0kfFe2ToVFY71Xtk6BhpeHdAwduUtIdevWjTlz5rBnzx7A86qanTt3snv3bpxz/PznP+fhhx9m9erVAFSpUoXvv/++0OcaNWoUd9xxh+/xAwcO5DtPnvfuS02bNqVevXqlnvmqq65i+vTpvrfSy87OZs+ePRw8eJCaNWuSlJTEhg0bWLlyZYmeN2/JXSDfuxp1796dKVOm+ErQffv20bx5c7788kvf78uhQ4c4fvw4KSkprFmzBuccO3bsKPI/saKWI65Rowa1a9fm9ddfBzz/ORw5cgTwvEH5o48+SmJiYoney1bwlq13wJB3IKk6PN835GWrjtwlpFq2bMnYsWPp1q0bJ06coEKFCkydOpX4+HgGDRrkO0KdOHEi4AmYwYMHk5SUxIoVK/IdEd95550cPnyYNm3aULFiRSpUqJDvfT379evHvffe61vzvLR69uzJ559/ziWXXAJ4/sN58cUXueaaa5g2bRqpqak0adKkxEv83n///dx66638+c9/pmvXn9Yrue2229i6dSutWrXyvQ/ssGHDmDVrFrfffjs5OTkkJSXxzjvv0KVLFxo0aOArogt2E3n8lyM+77zz8s36wgsvcNtttzF69GgqVqzI3LlzOe+886hfvz6NGjUqtIiWANVrCUPehbcf8pStXyyB65+BOo2Cvmst+RujYmbpVQmbw4cP07JlS9atW+d7RVJx9PfuFDYv8JStR4/Az/4OLa4r1dNoyV8RKbWFCxfStGlTRo4cGXCwSzHyytZW/UK2AAAGlklEQVQLLoNaFwZ9dzotIyInueqqq9i5c2e4x4g9VerBL2aHZFc6chcRiUEK9xgWrj5Fyif9fYssCvcYlZiYyN69e/UPTkLCOcfevXtJTEwM9yjipXPuMSo5OZns7Gx2794d7lGknEhMTCQ5OTncY4hXQOFuZj2AJ4B44Gnn3IQCj/8KeBT42nvXJOfc02U4p5RQhQoVilxLRURiX7HhbmbxwGTgSiAbWGlm85xzGwts+pJzbkQQZhQRkRIK5Jx7OyDLObfdOXcUmA30Ce5YIiJyOgIJ9wbAV363s733FXS9mX1qZv8ys3MKeyIzG2pmq8xslc4Fi4gETyDn3AtbjLjgSzBeB2Y55340s2HATOCkJdCcc9OAaQBmttvMvizhvHlqA3tK+bXRSt9z+aDvuXw4ne+58DcxKCCQcM8G/I/Ek4Fd/hs45/wXLP4nMLG4J3XO1QlkwMKY2apA1laIJfqeywd9z+VDKL7nQE7LrAQuMrPzzawiMACY57+BmZ3td7M3sKnsRhQRkZIq9sjdOZdrZiOAhXheCvmMc26DmY0HVjnn5gF3mVlvIBfYB/wqiDOLiEgxAnqdu3NuPjC/wH2/8/v8AeCBsh3tlIp+i57Ype+5fND3XD4E/XsO23ruIiISPFpbRkQkBkVduJtZDzPbbGZZZjYq3PMEm5mdY2bvmtkmM9tgZneHe6ZQMLN4M1tjZv8J9yyhYGbVvdeIfO79s24f7pmCzcxGev9OrzezWWYWc6uOmdkzZvadma33u6+mmb1tZlu9H2sEY99RFe5+SyFcDTQDMs2sWXinCrpc4NfOuabAJcDwcvA9A9xN+XrV1RPAm865JkAqMf69m1kD4C4gwznXAs+LNWLxzVqfBXoUuG8UsNg5dxGw2Hu7zEVVuFMOl0Jwzn3jnFvt/fx7PP/oC7tCOGaYWTJwDVAuFp8zs6pAZ2A6gHPuqHPuQHinCokEIMnMEoBKFLh+JhY455bieQWhvz54LvTE+/Fnwdh3tIV7oEshxCQzSwFaAx+Hd5Kg+xtwH3Ai3IOEyAXAbmCG91TU02ZWOdxDBZNz7mvgL8BO4BvgoHPurfBOFTJnOee+Ac/BG1A3GDuJtnAPZCmEmGRmZwJzgXucc4fCPU+wmNm1wHfOuU/CPUsIJQDpwBTnXGvgMEH6UT1SeM8z9wHOB+oDlc3spvBOFVuiLdyLXQohFplZBTzB/oJz7t/hnifIOgK9zWwHntNul5vZ/4V3pKDLBrKdc3k/kf0LT9jHsm7AF8653c65Y8C/gQ5hnilU/pt3Vb/343fB2Em0hXuxSyHEGjMzPOdiNznnHgv3PMHmnHvAOZfsnEvB8+f7jnMupo/onHPfAl+ZWWPvXVcABd8vIdbsBC4xs0rev+NXEOMlsp95wEDv5wOB14Kxk6h6m72ilkII81jB1hG4GfjMzNZ673vQe9WwxI47gRe8By3bgVvCPE9QOec+NrN/AavxvCJsDTF4paqZzQIuA2qbWTYwFpgAzDGzQXj+k/t5UPatK1RFRGJPtJ2WERGRACjcRURikMJdRCQGKdxFRGKQwl1EJAYp3CVqmdlZZvaimW03s0/M7EMz6xvuuUQigcJdopL3wpdXgaXOuQucc23wXPSUHN7JRCKDwl2i1eXAUefc1Lw7nHNfOueeMrMUM3vfzFZ7f3UAMLPLzGyJmc0xsy1mNsHMbjSzFWb2mZk19G73rJlN8a6jv93MunjX5d5kZs/m7c/M/mdmE70/NSwys3Zm9p73a3p7tyl0FpFgU7hLtGqO5+rGwnwHXOmcSwf6A0/6PZaKZ634lniu/G3knGuHZ3nhO/22q4HnP5CRwOvA4959tjSzNO82lYH3vD81fA88AlwJ9AXGBzCLSNBE1fIDIkUxs8lAJ+AonkWpJnlD+DjQyG/TlXnLrZrZNiBvmdnPgK5+273unHNm9hnwX+fcZ96v2QCkAGu9+3rT7+t/dM4d835Nivf+CqeYRSRoFO4SrTYA1+fdcM4NN7PawCo8R9v/xXOUHgfk+H3dj36fn/C7fYL8/x5+LGSbgtsdcz+t3+Hbzjl3wvsGFBQzi0jQ6LSMRKt3gEQzu93vvkrej9WAb5xzJ/CceokP9XB+ImkWKUcU7hKVvEfMPwO6mNkXZrYCz1uW3Q/8HRhoZh/hOQ1yOHyTRtQsUo5oVUgRkRikI3cRkRikcBcRiUEKdxGRGKRwFxGJQQp3EZEYpHAXEYlBCncRkRikcBcRiUH/D+4XL3EFj69UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faf91f93278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(param_range, train_scores.mean(axis=1), label='Train CV mean accuracy');\n",
    "plt.plot(param_range, test_scores.mean(axis=1), label='Test CV mean accuracy');\n",
    "plt.xlabel('Gammma');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 3\n",
    "----\n",
    "\n",
    "In this part of the notebook, we will train several models and evaluate how effectively they predict instances of fraud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Use X_train, X_test, y_train, y_test for all of the following questions\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_csv('datasets/fraud_data.csv')\n",
    "\n",
    "X = df.iloc[:,:-1]\n",
    "y = df.iloc[:,-1]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What percentage of the observations in the dataset are instances of fraud?\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of fraud: 1.64%\n"
     ]
    }
   ],
   "source": [
    "print('Percentage of fraud: %.2f%%' %(100*df.Class.value_counts()[1]/df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the accuracy of a dummy classifier? What is the recall?\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy dummy classifier:  0.9852507374631269\n",
      "Recall dummy classifier:  0.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.metrics import recall_score\n",
    "    \n",
    "clf = DummyClassifier(strategy='most_frequent')\n",
    "clf.fit(X_train, y_train)\n",
    "accuracy = clf.score(X_test, y_test)\n",
    "recall = recall_score(y_test, clf.predict(X_test))\n",
    "\n",
    "print('Accuracy dummy classifier: ', accuracy)\n",
    "print('Recall dummy classifier: ', recall)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the accuracy, recall, and precision of a Support Vector Machines classifier (SVC)?\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision SVC classifier:  1.0\n",
      "Recall SVC classifier:  0.375\n",
      "Accuracy SVC classifier:  0.9907817109144543\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import recall_score, precision_score\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "clf = SVC()\n",
    "clf.fit(X_train, y_train)\n",
    "accuracy = clf.score(X_test, y_test)\n",
    "recall = recall_score(y_test, clf.predict(X_test))\n",
    "precision = precision_score(y_test, clf.predict(X_test))\n",
    "\n",
    "print('Precision SVC classifier: ', precision)\n",
    "print('Recall SVC classifier: ', recall)\n",
    "print('Accuracy SVC classifier: ', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the confusion matrix when using a threshold of -220 on the decision function?\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix: \n",
      " [[5320   24]\n",
      " [  14   66]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "clf = SVC(C=1e9, gamma=1e-7)\n",
    "clf.fit(X_train, y_train)\n",
    "preds = clf.decision_function(X_test)\n",
    "preds[preds>=-220] = 1 \n",
    "preds[preds<-220] = 0\n",
    "conf = confusion_matrix(y_test, preds) \n",
    "\n",
    "print('Confusion matrix: \\n', conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a logistic regression classifier\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XuYXXV97/H3Z+4zyeQ6kxBy5ZJIUkDAEbD2AC3IAY6CejgIFS2KoLbUWqutfeqF2lqrPepplVZpQdAKCB7LSS3IsYiiFiRBLppIbAiXTBLMJOQyM5n7fPvHWrPYTCYze5JZs+fyeT3Pfmattdes/f3NJOsz6/dbF0UEZmZmAGWlLsDMzCYOh4KZmWUcCmZmlnEomJlZxqFgZmYZh4KZmWUcCjapSNog6ZwR1lkmqU1S+TiVlTtJz0o6L52+XtI/l7omm5ocCjYm0p1WR7oz/pWkr0iaOdafExG/FhHfH2Gd5yNiZkT0jfXnpzvknrSdeyX9h6TXjPXnmJWKQ8HG0hsiYiZwGvBq4CODV1Bisv+7+0bazgbgAeCuEtcz5iRVlLoGK43J/p/TJqCI2AbcC5wIIOn7kj4p6cfAAeBYSbMl3SRph6Rtkv6ysLtH0jWSfiGpVdJGSaelywu7UU6XtF7S/vTo5HPp8hWSYmDHJuloSWslvShps6RrCj7nekl3Svpq+lkbJDUV2c5e4OvAYkmNBdt8vaTHC44kTi54b6mkb0lqkbRb0hfT5cdJ+l66bJekr0uaczg/f0mXpJ+/X9LTki4Y/LMraPs/D/qZXS3peeB7kr4j6bpB235C0pvT6RMkfTf9uW6SdNnh1GsTi0PBxpykpcBFwGMFi98GXAvUA88BtwK9wPHAqcD5wLvS7/9fwPXA24FZwMXA7iE+6m+Bv42IWcBxwJ2HKOl2oBk4GrgU+CtJ5xa8fzFwBzAHWAt8sch2VqU17gb2pMtOA24G3g3MB74MrJVUnYbet9P2rwAWp58LIOBTaY2rgaXpz2BUJJ0OfBX4UNqes4BnR7GJs9PP/+/AbcAVBdteAywH/k3SDOC76ToL0vX+XtKvjbZmm1gcCjaW7pa0F/gR8APgrwreuyUiNqR/Xc8DLgTeHxHtEbET+Dxwebruu4DPRMS6SGyOiOeG+Lwe4HhJDRHRFhEPD14hDajfAP4kIjoj4nHgn0hCasCPIuKedAzia8ArR2jnZWk7O4BrgEvTdpHOfzkifhIRfRFxK9AFnAmcTrLT/1Da7s6I+BFA2sbvRkRXRLQAnyPZQY/W1cDN6bb6I2JbRDw1iu+/Pq2tA/gX4BRJy9P33gp8KyK6gNcDz0bEVyKiNyJ+CvxfktC1ScyhYGPpjRExJyKWR8TvpjuWAVsLppcDlcCOtItlL8lf1AvS95cCTxfxeVcDq4CnJK2T9Poh1jkaeDEiWguWPUfyV/qAFwqmDwA1kiokvTUdUG6TdG/BOndGxBxgIfBz4FWD2vZHA+1K27Y0rWMp8FxBgGQkLZB0R9qVth/4Z5Ixi9Eq9md3KNnvKf2Z/RsvhfXlJN1lkLTzjEHtfCtw1BF8tk0AHkyy8VJ4O96tJH89Nwy1g0zfP27EDUb8J3BFOnD9ZuCbkuYPWm07ME9SfUEwLAO2FbH9r/PSTnCo93dJejewTtJtEbEjrf2TEfHJweunZyktk1QxRLs/RfIzOjkidkt6I0V2Yw0y3M+uHagrmB9qBz74tsm3Ax+X9CBQSzKwPvA5P4iI1x1GjTaB+UjBxl268/z/wGclzZJUlg60DnSX/BPwQUmvSk5W0vEFXRgZSVdKaoyIfmBvuvhlp6FGxFbgP4BPSapJB32vZpid/Sjb8hRwH/DH6aJ/BN4j6Yy09hmS/oekeuARYAfw1+nyGkmvTb+vHmgD9kpaTDImcDhuAt4h6dz057pY0gnpe48Dl0uqTAfTi+nquYfkqOATJGdd9afLvw2skvS2dHuVkl4tafVh1m0ThEPBSuXtQBWwkWSQ9pvAIoCIuAv4JMkgZitwN8k4xGAXABsktZEMOl8eEZ1DrHcFycDudpJ+8o9HxHfHsC1/A1wraUFErCcZV/hi2q7NwFUA6ZjFG0gG158nGfx+S7qNPyc5lXcfSZfNtw6nkIh4BHgHyRjNPpKxnYFA/SjJUcSe9PNuK2J7XWkt5xWunx51nU/SpbSdpAvu00D14dRtE4f8kB0zMxvgIwUzM8s4FMzMLONQMDOzjEPBzMwyk+46hYaGhlixYkWpyzAzm1QeffTRXRHRONJ6ky4UVqxYwfr160tdhpnZpCJpqFvFHMTdR2ZmlnEomJlZxqFgZmYZh4KZmWUcCmZmlsktFCTdLGmnpJ8f4n1J+jslj0d8Mn1ilZmZlVCeRwq3kNzF8lAuBFamr2uBf8ixFjMzK0Ju1ylExIOSVgyzyiXAVyO5TevDkuZIWpTea3/MrXv2RX74y5Y8Nl1SNVXlXPXrK6irmnSXnJjZBFTKPcliXv6IxuZ02UGhIOlakqMJli1bdlgf9tPn9vCFBzYf1vdOVAN3PX/FwnrOXb2wtMWY2ZRQylDQEMuGfLhDRNwI3AjQ1NR0WA+AePfZx/Hus0d8wuOk8vNt+3j9F35EX7+fiWFmY6OUZx81kzxkfMASkic4mZlZiZQyFNYCb0/PQjoT2JfXeIKZmRUnt+4jSbcD5wANkpqBjwOVABHxJZIHgl9E8gzbAyTPlTUzsxLK8+yjK0Z4P4Dfy+vzzWxyiAi6+/rp7k1fRU539b40X14mLn3VEmZU+yy8I+WfoJVEb18/ezt62HugmxfbezimYQaN9dWlLmtaigi6evvp6umnq7ePzp5+Onv76OxJp3uS6a7edLq3n66el78/8F7XEDvwrmy6L1ve0xcv28GPhQX11Vx40qIx2dZ05lCwMdHT18+uti5aWrvYub+LlrYuXmzvZk97Ny8e6GbvgR5ebO9OQ6Cb/Z29L/v+s1Y18tV3nl6i6ie2/v6go6eP9u5eOrr7aO/qo6Onl/auPg50vzTd0V2wTndv8l53H+3d6U493YkP3vF39fZnpzePVpmgprI8eVWUUV1ZTlV5GVUV6au8jDlVldn0y75WlB16+aB1qsvLqBxineqKMrbu6eB//sN/0Ouz8MaEQ8GG1d8ftLR10byngx37OpKdfrrj39naSUtrEgQvHugecsdSV1XO3Loq5s6oZG5dFcvm1TG3rpI5dVXMm1HFnLpK/uH7T3Ogq/fgb57Eevv6aevqpbWzt+BrD62dvdlrYL6ts5fWrl4OdA+9c+/sGd1f0tUVZdRVlVNXVZF+Lae6spzZdVUsrChLd+JlB+3MB5ZXV6TvV5QPWnfgvZemK8uFNNTZ5eNnf2cPALf95Hm+99ROOnv6qK+p4JNvOonKct/ebbQcCtNcRPBiezfP7GrnmV3tNO/pYPveDralrx17Ow86vK8oE4311Syor2bJ3DpOWz6XBfXV6bKabHrejCpqKstHrOHO9VvpGuWOL28RQVtXL3sP9LCvo4e9B3rY29Gdze9Lu76ynXxXL62dPckOvrOXjp6+ET+jvEzU11Qws/qlV31NBUfNqkl25tXJjr22spwZ1eXUVlUwI93JD0zXpjv/wunystLupMdbY30NxzTM4OmWNpr3HqCrp5+drV2895zjOaZhRqnLm3QcCtNEd28/T7e0sXlnWxYAW3a180xL20FdOQvqq1k8t5aTFs/mghOPYsmcWhbPrWXR7FoWzqphTm0lZZNsx3Ogu5fdbd20tHWxu62b3W1d7GrrYnd7N/sO9GTjG3s7erL54S4KrK0sZ3ZtJbNqk535nNpKlsytpT7dsc+srky+1lQwa9B8fU0F9dWV1FSWlfyv7Klgdm0lD3zwnGz+7se28f5vPF66giY5h8IUtPdANxt37Gfj9v38YkcrG3fsZ/POVnr6XtrJLZ5Ty4qGOi4+5WiOaZjJsQ0zWNEwg6Pn1FBdMfJf9xNBW1cvL+zryMYwdrd1s7u9i12t6deC+UP95T6zuoI5dZXJq7aKRXNqmVP70vzsusp0vipdVsms2sqijoDMJiOHwhTwix2tPLu7nZ8+t5cnm/eyfV9n9l5jfTWrF83i7FWNrF5Uz6qF9ayYP4Paqom7Uxvo0nphfycv7Otkx75OfrV/0Nd9nbQOMQ5RXibmzahi/owqGuurWTG/joaZ1cyfWc38mVU0pl/nz6xmfpHdW2bTiUNhEhvoO/78v/8SgGXz6mhaMY9fO3oWqxclr8lymueWXe287aafsG1PMpbR1fvyMYYyJQF31OxajmucwW8c38DCWTUsml3DglnV6c6+elJ2bZlNJA6FSewVC+v5yzeeyIL6ak5dNnfSBMBgJy6ezaYXWtnf0cMJi+o5d/UCjp5Ty1GzajhqdvJqnFlNhc8kMcudQ2ESKysTV565vNRlHLE/vXA1f3rh6lKXYWb4Gc1mZlbAoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZplcQ0HSBZI2Sdos6cNDvL9M0gOSHpP0pKSL8qzHzMyGl1soSCoHbgAuBNYAV0haM2i1jwB3RsSpwOXA3+dVj5mZjSzPI4XTgc0RsSUiuoE7gEsGrRPArHR6NrA9x3rMzGwEeYbCYmBrwXxzuqzQ9cCVkpqBe4DfH2pDkq6VtF7S+paWljxqNTMz8g0FDbEsBs1fAdwSEUuAi4CvSTqopoi4MSKaIqKpsbExh1LNzAzyDYVmYGnB/BIO7h66GrgTICIeAmqAhhxrMjOzYeQZCuuAlZKOkVRFMpC8dtA6zwPnAkhaTRIK7h8yMyuR3EIhInqB64D7gF+QnGW0QdInJF2crvZHwDWSngBuB66KiMFdTGZmNk4q8tx4RNxDMoBcuOxjBdMbgdfmWYOZmRXPVzSbmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWyfWGeGZmpXLfhhfY19HDf/6qjcVzavjzS04sdUmTgkPBzKaUmsqkA+Sv732KijJRU1lORDgUiuRQMLMp5bdOWMhX33k6i2bXsKJhBp++9yluf+T5Upc1aTgUzGxKqaoo46xVfpb74fJAs5mZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVnGoWBmZhmHgpmZZRwKZmaWcSiYmVkm11CQdIGkTZI2S/rwIda5TNJGSRsk3ZZnPWZmNryiH7IjaTGwvPB7IuLBYdYvB24AXgc0A+skrY2IjQXrrAT+FHhtROyRtGD0TTAzs7FSVChI+jTwFmAj0JcuDuCQoQCcDmyOiC3pNu4ALkm3MeAa4IaI2AMQETtHVb2ZmY2pYo8U3gi8IiK6RrHtxcDWgvlm4IxB66wCkPRjoBy4PiK+M3hDkq4FrgVYtmzZKEowM7PRKHZMYQtQOcpta4hlMWi+AlgJnANcAfyTpDkHfVPEjRHRFBFNjY1+9qqZWV6KPVI4ADwu6X4gO1qIiPcN8z3NwNKC+SXA9iHWeTgieoBnJG0iCYl1RdZlZmZjqNhQWJu+RmMdsFLSMcA24HLgtwetczfJEcItkhpIupO2jPJzzMxsjBQVChFxq6Qq0jEAYFP61/1w39Mr6TrgPpLxgpsjYoOkTwDrI2Jt+t75kgYGsD8UEbsPtzFmZnZkij376BzgVuBZkrGCpZJ+Z7hTUgEi4h7gnkHLPlYwHcAH0peZmZVYsd1HnwXOj4hNAJJWAbcDr8qrMDMzG3/Fnn1UORAIABHxS0Z/NpKZmU1wxR4prJd0E/C1dP6twKP5lGRmZqVSbCi8F/g94H0kYwoPAn+fV1FmZlYaxZ591AV8Ln2ZmdkUNWwoSLozIi6T9DMOvhqZiDg5t8rMzGzcjXSk8Afp19fnXYiZmZXesGcfRcSOdHIXsDUingOqgVdy8C0rzMxskiv2lNQHgZr0mQr3A+8AbsmrKDMzK41iQ0ERcQB4M/CFiHgTsCa/sszMrBSKDgVJryG5PuHf0mVFP7XNzMwmh2JD4f0kj838l/SmdscCD+RXlpmZlUKx1yn8APhBwfwWkgvZzMxsChnpOoX/ExHvl/SvDH2dwsW5VWZmZuNupCOFgXsd/e+8CzEzs9IbNhQiYuCmd+uBjojoB5BUTnK9gpmZTSHFDjTfD9QVzNcC/z725ZiZWSkVGwo1EdE2MJNO1w2zvpmZTULFhkK7pNMGZiS9CujIpyQzMyuVYi9Aez9wl6SB+x0tAt6ST0lmZlYqxV6nsE7SCcArSB6y81RE9ORamZmZjbuiQkFSHfABYHlEXCNppaRXRMS38y3PzOzIBbB5Zxsbtu9jw/b9nL5iHuetWVjqsiakYruPvkLyTObXpPPNwF2AQ8HMJjQJDnT3cd7nspsy8PjWvQ6FQyg2FI6LiLdIugIgIjokKce6zMzGxJtPW4IkVi2s56TFs/nI3T8rdUkTWrGh0C2plvRWF5KOA7pyq8rMbIysXjSL1YtmZfPlZaL/oJv22IBiQ+HjwHeApZK+DrwWuCqvoszMrDRGDIW0m+gpkgfsnEly9tEfRMSunGszM7NxNmIoRERIujsiXsVLD9gxM7MpqNgrmh+W9OpcKzEzs5IrdkzhN4H3SHoWaCfpQoqIODmvwszMbPwVGwoX5lqFmZlNCCM9ea0GeA9wPPAz4KaI6B2PwszMbPyNNKZwK9BEEggXAp8dzcYlXSBpk6TNkj48zHqXSgpJTaPZvpmZja2Ruo/WRMRJAJJuAh4pdsPp09luAF5HcluMdZLWRsTGQevVA+8DfjKaws3MbOyNdKSQ3Qn1MLqNTgc2R8SWiOgG7gAuGWK9vwA+A3SOcvtmZjbGRgqFV0ran75agZMHpiXtH+F7FwNbC+ab02UZSacCS0e626qkayWtl7S+paVlhI81M7PDNWz3UUSUH8G2h7phXnbHEUllwOcp4nYZEXEjcCNAU1OT71piZpaTYi9eOxzNwNKC+SXA9oL5euBE4Pvp9Q9nAms92GxmVjp5hsI6YKWkYyRVAZcDawfejIh9EdEQESsiYgXwMHBxRKzPsSYzMxtGbqGQDkxfB9wH/AK4MyI2SPqEpIvz+lwzMzt8xV7RfFgi4h7gnkHLPnaIdc/JsxYzMxtZnt1HZmY2yTgUzMws41AwM7OMQ8HMzDIOBTMzyzgUzMws41AwM7OMQ8HMzDK5XrxmZjYR9fUHv/xVK5teaOXZXe1cdPIijmucWeqyJgSHgplNK0I8+tyLnP/5B7Nl+zp6+Mjr15SwqonDoWBm08p7zjmOU5fNYdXCelYunMllX3qIft+QP+NQMLNp5exVjZy9qjGbL9NQj36ZvjzQbGZmGYeCmZllHApmZpZxKJiZWcahYGZmGYeCmZllHApmZpZxKJiZWcahYGZmGYeCmZllHApmZpZxKJiZWcahYGZmGYeCmZllfOtsM5v2Wjt7ePS5F3lu9wF2t3Vz2auXMru2stRllYRDwcymtbIycdejzdz1aHO2bMGsai45ZXEJqyodh4KZTWt//eaT+NX+TpbPn4EEV31lHX3T+FFsDgUzm9YuPGlRNv3c7vYSVjIxeKDZzMwyuYaCpAskbZK0WdKHh3j/A5I2SnpS0v2SludZj5mZDS+3UJBUDtwAXAisAa6QtGbQao8BTRFxMvBN4DN51WNmZiPL80jhdGBzRGyJiG7gDuCSwhUi4oGIOJDOPgwsybEeMzMbQZ6hsBjYWjDfnC47lKuBe4d6Q9K1ktZLWt/S0jKGJZqZWaE8Q0FDLBvyPC9JVwJNwN8M9X5E3BgRTRHR1NjYOIYlmplZoTxPSW0GlhbMLwG2D15J0nnAnwFnR0RXjvWYmdkI8jxSWAeslHSMpCrgcmBt4QqSTgW+DFwcETtzrMXMzIqQ25FCRPRKug64DygHbo6IDZI+AayPiLUk3UUzgbskATwfERfnVZOZWTFueGAzP31+Dy/s66Q/4IbfPo3aqvJSlzUucr2iOSLuAe4ZtOxjBdPn5fn5ZmajMaumkspy8XRLOy+2d1NdUc4L+ztp3nOAlQvrS13euPBtLszMUnNnVPHoR19HZVkZtVXlfPvJ7Vx322OlLmtcORTMzArMqpmet8we4HsfmZlZxkcKZmYjuPnHzzC7toqdrZ0snVvHH75uValLyo1DwczsEObWVQFw+yNbqSovo7xM9EU4FMzMpqNfP24+P/7wbzGjqpzZtZV8+jubuPnHz5S6rFw5FMzMDkESi+fUlrqMceWBZjMzyzgUzMws41AwM7OMQ8HMzDIOBTMzyzgUzMws41AwMzsMXb197GztpKevv9SljClfp2BmNgrdvf2c8NF76exJwuD8NQu58e1NJa5q7DgUzMyKdMkpR7O/s4eZ1RXMqqngWz/dxvc3tfDD/2yhtbOXtq5ezl7VyMJZNaUu9bA5FMzMirR60Sz+6k0nZfNPNO9jy6523nbTI9myVy6dw/vPW0lrZy+9ff1ccOJR1FVNnl2tIqLUNYxKU1NTrF+/vtRlmJnxwr5ONmzfR31NJfU1FVz1lUf41f6ul61z7gkLOG/NQtq7eqksL+NtZy6nrEzjXqukRyNixH6uyRNfZmYTzFGzazhq9ktdRV9/1xls39tJfU0Fff3BpV96iPuf2sn9T+3M1jnj2HmccNSsUpRbFIeCmdkYOX5BPccveOlZzj/849+ks6ePGdUVPLxlNx+48wn6+id274xDwcwsJ0vn1WXTM6onx+7W1ymYmVlmckSXmdkUcc2t6zmmcQYd3X007+ngTacupqOnj47uPs5a1cgbXnl0SetzKJiZjYNXLKznxMWz6OuHrp5+Ht+6l/6AWx96ltrKctq7+njqhVaHgpnZdLCiYQbf/v3/9rJlEYGUnJ76zlvW0dLaNdS3jiuPKZiZlchAIEwkDgUzM8s4FMzMLONQMDOzjAeazcwmAAE/27aP1R/9Dn39QXf6nIZl8+ro6w96+/v5kwtO4M2nLcm1DoeCmdkEcO1Zx7Jsfh0VZaKivIzHnt9Dw8xqKspEeVkZleXi6Dm1udfhUDAzmwDOOHY+Zxw7v9Rl5DumIOkCSZskbZb04SHer5b0jfT9n0hakWc9ZmY2vNxCQVI5cANwIbAGuELSmkGrXQ3siYjjgc8Dn86rHjMzG1meRwqnA5sjYktEdAN3AJcMWucS4NZ0+pvAuZqIV3OYmU0TeYbCYmBrwXxzumzIdSKiF9gHHNSpJulaSeslrW9pacmpXDMzyzMUhvqLf/DTJYpZh4i4MSKaIqKpsbFxTIozM7OD5RkKzcDSgvklwPZDrSOpApgNvJhjTWZmNow8Q2EdsFLSMZKqgMuBtYPWWQv8Tjp9KfC9iJjYz6ozM5vCcrtOISJ6JV0H3AeUAzdHxAZJnwDWR8Ra4Cbga5I2kxwhXJ5XPWZmNjJNtj/MJbUAzx3mtzcAu8awnMnAbZ4e3Obp4UjavDwiRhyUnXShcCQkrY+IplLXMZ7c5unBbZ4exqPNvkuqmZllHApmZpaZbqFwY6kLKAG3eXpwm6eH3Ns8rcYUzMxseNPtSMHMzIbhUDAzs8yUDIXp+ByHItr8AUkbJT0p6X5Jy0tR51gaqc0F610qKSRN+tMXi2mzpMvS3/UGSbeNd41jrYh/28skPSDpsfTf90WlqHOsSLpZ0k5JPz/E+5L0d+nP40lJp41pARExpV4kV08/DRwLVAFPAGsGrfO7wJfS6cuBb5S67nFo828Cden0e6dDm9P16oEHgYeBplLXPQ6/55XAY8DcdH5BqesehzbfCLw3nV4DPFvquo+wzWcBpwE/P8T7FwH3ktxQ9EzgJ2P5+VPxSGE6PsdhxDZHxAMRcSCdfZjkBoWTWTG/Z4C/AD4DdI5ncTkpps3XADdExB6AiNg5zjWOtWLaHMCsdHo2B994c1KJiAcZ/saglwBfjcTDwBxJi8bq86diKIzZcxwmkWLaXOhqkr80JrMR2yzpVGBpRHx7PAvLUTG/51XAKkk/lvSwpAvGrbp8FNPm64ErJTUD9wC/Pz6llcxo/7+PSm43xCuhMXuOwyRSdHskXQk0AWfnWlH+hm2zpDKSR7xeNV4FjYNifs8VJF1I55AcDf5Q0okRsTfn2vJSTJuvAG6JiM9Keg3JTTZPjIj+/MsriVz3X1PxSGE6PsehmDYj6Tzgz4CLI6JrnGrLy0htrgdOBL4v6VmSvte1k3ywudh/2/8vInoi4hlgE0lITFbFtPlq4E6AiHgIqCG5cdxUVdT/98M1FUNhOj7HYcQ2p10pXyYJhMnezwwjtDki9kVEQ0SsiIgVJOMoF0fE+tKUOyaK+bd9N8lJBUhqIOlO2jKuVY6tYtr8PHAugKTVJKEwlZ/buxZ4e3oW0pnAvojYMVYbn3LdRzENn+NQZJv/BpgJ3JWOqT8fEReXrOgjVGSbp5Qi23wfcL6kjUAf8KGI2F26qo9MkW3+I+AfJf0hSTfKVZP5jzxJt5N0/zWk4yQfByoBIuJLJOMmFwGbgQPAO8b08yfxz87MzMbYVOw+MjOzw+RQMDOzjEPBzMwyDgUzM8s4FMzMLONQMBtEUp+kxyX9XNK/Spozxtu/StIX0+nrJX1wLLdvdiQcCmYH64iIUyLiRJLrWH6v1AWZjReHgtnwHqLgZmOSPiRpXXof+z8vWP72dNkTkr6WLntD+ryOxyT9u6SFJajfbFSm3BXNZmNFUjnJ7RNuSufPJ7mP0OkkNyVbK+ksYDfJPaVeGxG7JM1LN/Ej4MyICEnvAv6Y5OpbswnLoWB2sFpJjwMrgEeB76bLz09fj6XzM0lC4pXANyNiF0BEDNxccQnwjfRe91XAM+NSvdkRcPeR2cE6IuIUYDnJznxgTEHAp9LxhlMi4viIuCldPtT9Yr4AfDEiTgLeTXKjNrMJzaFgdggRsQ94H/BBSZUkN2V7p6SZAJIWS1oA3A9cJml+unyg+2g2sC2d/h3MJgF3H5k5ad8UAAAAZUlEQVQNIyIek/QEcHlEfC29NfND6Z1m24Ar07t2fhL4gaQ+ku6lq0ieCHaXpG0kt+4+phRtMBsN3yXVzMwy7j4yM7OMQ8HMzDIOBTMzyzgUzMws41AwM7OMQ8HMzDIOBTMzy/wXqUgV9WAgUT8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faf91e34358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import precision_recall_curve\n",
    "from sklearn.metrics import roc_curve\n",
    "lr_clf = LogisticRegression()\n",
    "lr_clf.fit(X_train, y_train)\n",
    "probs = lr_clf.predict_proba(X_test)[:,1]\n",
    "precision, recall, thresholds = precision_recall_curve(y_test, probs)\n",
    "\n",
    "plt.plot(recall, precision);\n",
    "plt.title('Precision-Recall curve')\n",
    "plt.xlabel('Recall');\n",
    "plt.ylabel('Precision');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid search with a Logistic Regression classifier\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_scores():    \n",
    "    from sklearn.model_selection import GridSearchCV\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    \n",
    "    lr_clf = LogisticRegression()\n",
    "    params = {'penalty': ['l1', 'l2'],\n",
    "              'C': [0.01, 0.1, 1, 10, 100]}\n",
    "    grid = GridSearchCV(lr_clf, params, scoring='recall', cv=3)\n",
    "    grid.fit(X_train, y_train)\n",
    "    cv_results = grid.cv_results_\n",
    "    return cv_results['mean_test_score'].reshape((5,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFOpJREFUeJzt3X+MXeWd3/H3BxMn2R9JILArAiSY1kFJNyppU7JSpChZCvFG7UJVQuyqiqlQvakCf9AqCkhpduUKKfsX0qoojZsQIGXxEq9o3F2yJAV2s61g107rJNgrwDERTE2WpoQ0NOGHZ779455xLuOZuWc813PvnPN+oSPfe85zznkGWd/5+nue5zmpKiRJ3XHapDsgSRovA7skdYyBXZI6xsAuSR1jYJekjjGwS1LHGNglqWMM7JLUMQZ2SeqY0yfdgYVO33iuU2F1gh/+07dPuguaQm+6+6Gs9hqv/PBI65jzmrMuXPX91oIZuyR1zNRl7JK0puZmJ92DsTOwS+q32WOT7sHYGdgl9VrV3KS7MHYGdkn9Nmdgl6RuMWOXpI7x4akkdYwZuyR1SzkqRpI6xoenktQxlmIkqWN8eCpJHdPBjN1FwCT12+yx9tsISbYkeSzJ4SQ3LnL8rUkeSvI/k3wnyYeHjt3UnPdYkg+1veZizNgl9duYHp4m2QDcClwGzAD7kuytqkNDzT4N3FNVn0vyTuA+4ILm81bg7wBvAf5rkvm1qkdd8wQGdkm9VjW2GvslwOGqOgKQZDdwBTAchAt4Q/P5jcDR5vMVwO6qegl4Msnh5nq0uOYJDOyS+m18NfZzgaeHvs8A713Q5neBrye5HvhF4B8OnfvIgnPPbT6PuuYJWtXYW9SNXpvkD5vjf5nkgmb/m5t60gtJ/n2be0nSmpqba70l2ZFk/9C2Y+hKi71daeHbmbYBt1fVecCHgS8nOW2Zc9tc8wQjM/aWdaNrgR9V1d9OshX4PeCjwIvAvwV+rdkkabqsIGOvql3AriUOzwDnD30/j5+XWuZdC2xprvVwktcBZ404d9Q1T9AmYz9eN6qql4H5Gs+wK4A7ms97gEuTpKr+X1X9NwYBXpKmz+wr7bfl7QM2J9mUZCODh6F7F7R5CrgUIMk7gNcB/7tpt7WpfmwCNgN/1fKaJ2hTY29TNzrepqqOJfkx8Gbghy2uL0mTM6ZRMU3suw64H9gA3FZVB5PsBPZX1V7g3wD/MckNDEoq11RVAQeT3MPgoegx4BPVPNVd7Jqj+tImsLep8ZxUHej4yYM61Q6AbHgjp532i21PlaTVGeMEpaq6j8EQxuF9nxn6fAh43xLn3gzc3Oaao7QpxbSpGx1vk+R0BsN4nmvbiaraVVXvqar3GNQlrakVPDxdL9oE9jY1nr3A9ubzVcCDzT8vJGm6dTCwjyzFtKwbfZHBsJ3DDDL1rfPnJ/k+gwH5G5NcCVw+ataUJK2VGv1QdN1pNUGpRd3oReAjS5x7wSr6J0mnVgcXAXPmqaR+W0cllrYM7JL6zYxdkjrGjF2SOsaMXZI65tjoF2isNwZ2Sf1mxi5JHWONXZI6xoxdkjrGjF2SOsaMXZI6xlExktQxHVyI1sAuqd+ssUtSxxjYJaljfHgqSR0zOzvpHozd1AX2F/7q85PugqZQfuFNk+6CuspSjCR1jIFdkjrGGrskdUvNOY5dkrrFUowkdYyjYiSpY8zYJaljDOyS1DEdXATstEl3QJImam6u/TZCki1JHktyOMmNixy/JcmBZns8yfPN/g8O7T+Q5MUkVzbHbk/y5NCxi0f1w4xdUr+Nabhjkg3ArcBlwAywL8neqjo036aqbhhqfz3w7mb/Q8DFzf4zgcPA14cu/8mq2tO2L2bskvptdrb9trxLgMNVdaSqXgZ2A1cs034bcPci+68CvlZVPz2pnwcDu6Seq7m51luSHUn2D207hi51LvD00PeZZt8JkrwN2AQ8uMjhrZwY8G9O8p2mlPPaUT+TpRhJ/baCUkxV7QJ2LXE4i52yRNutwJ6qetU/A5KcA7wLuH9o903AD4CNzb0/Bexcrp9m7JL6rebab8ubAc4f+n4ecHSJtotl5QBXA/dW1SvHu1f1TA28BHyJQclnWQZ2Sf02V+235e0DNifZlGQjg+C9d2GjJBcBZwAPL3KNE+ruTRZPkgBXAo+O6oilGEn9dmw8SwpU1bEk1zEoo2wAbquqg0l2Avuraj7IbwN2V716AH2SCxhk/H++4NJ3JTmbQannAPDxUX0xsEvqtzEu21tV9wH3Ldj3mQXff3eJc7/PIg9bq+o3VtoPA7ukfuvgsr2rqrG3mGX1/iT/I8mxJFet5l6SdCqsZLjjenHSgX1oltVvAu8EtiV554JmTwHXAH9wsveRpFNqfA9Pp8ZqSjHHZ1kBJJmfZTU8ffb7zbH186tOUr+so4Dd1moC+2KzrN67uu5I0hrr4Is2VlNjX8ksq+UvNDRN94t/9Ker6JIkrUzNVettvVhNxr6SWVbLGp6m++KBP14///ckrX/rKGC3tZrAfnyWFfC/GMyy+mdj6ZUkrZV1NNqlrZMuxVTVMWB+ltVfA/fMz7JK8lsASf5BkhngI8DnkxwcR6claWwcFfNqo2ZZVdU+BiUaSZpO6yhgt+XMU0m9VrPdK8UY2CX1mxm7JHXLehrG2JaBXVK/GdglqWO6V2I3sEvqtzrWvchuYJfUb92L6wZ2Sf3mw1NJ6hozdknqFjN2SeoaM3ZJ6pY6NukejJ+BXVKvlRm7JHWMgV2SusWMXZI6xsC+Bjac985Jd0FSj9RsJt2FsZu6wC5Ja6mLGftJv/NUkrqg5tJ6GyXJliSPJTmc5MZFjt+S5ECzPZ7k+aFjs0PH9g7t35TkL5M8keQPk2wc1Q8zdkm9Nq6MPckG4FbgMmAG2Jdkb1UdOn6vqhuG2l8PvHvoEj+rqosXufTvAbdU1e4k/wG4Fvjccn0xY5fUa1VpvY1wCXC4qo5U1cvAbuCKZdpvA+5e7oJJAvwGsKfZdQdw5aiOGNgl9VrNtd9GOBd4euj7TLPvBEneBmwCHhza/bok+5M8kmQ+eL8ZeL7q+PzYJa85zFKMpF6bW8GomCQ7gB1Du3ZV1a75w4ucstQKY1uBPVU1O7TvrVV1NMmFwINJvgv83xVc8zgDu6Rea/NQ9HjbQRDftcThGeD8oe/nAUeXaLsV+MSCax9t/jyS5M8Y1N//CHhTktObrH25ax5nKUZSr41xVMw+YHMzimUjg+C9d2GjJBcBZwAPD+07I8lrm89nAe8DDlVVAQ8BVzVNtwNfHdURA7ukXqtqvy1/nToGXAfcD/w1cE9VHUyyM8lvDTXdBuxugva8dwD7k3ybQSD/7NBomk8B/zrJYQY19y+O+plSo3q7xl754ZHp6pCkqfWasy5c9bTRI++6vHXMufC7X18X01StsUvqtRbDGNcdA7ukXpt1rRhJ6hYzdknqmJUMd1wvDOySem3Kxo+MhYFdUq+ZsUtSx8zOdW86zyn/iZLcluTZJI+e6ntJ0kqNa4LSNFmLX1W3A1vW4D6StGJzldbbenHKSzFV9c0kF5zq+0jSyXC4oyR1zHoqsbQ1FU8NkuxoFpjf/4U7l32hiCSNlaWYU2R4jWMXAZO0lro4KmYqArskTUoXM8m1GO54N4MF5S9KMpPk2lN9T0lqy1LMSaiqbaf6HpJ0shwVI0kdMzfpDpwCBnZJvVaYsUtSpxyzFCNJ3WLGLkkdY41dkjrGjF2SOsaMXZI6ZtaMXZK6pYNvxjOwS+q3OTN2SeqWLi4CZmCX1Gs+PJWkjplL90ox3VthXpJWYHYF2yhJtiR5LMnhJDcucvyWJAea7fEkzzf7L07ycJKDSb6T5KND59ye5Mmh8y4e1Q8zdkm9Nq5RMUk2ALcClwEzwL4ke6vq0HybqrphqP31wLubrz8FPlZVTyR5C/CtJPdX1fPN8U9W1Z62fTFjl9Rrc6T1NsIlwOGqOlJVLwO7gSuWab8NuBugqh6vqieaz0eBZ4GzT/ZnmrqMvX72k0l3QVMor//lSXdBHbWSUTFJdgA7hnbtat7ZDHAu8PTQsRngvUtc523AJuDBRY5dAmwEvje0++YknwEeAG6sqpeW6+fUBXZJWksrKcU0QXzXEocXu9JSvze2Anuq6lWl+yTnAF8GtlfV/ICdm4AfMAj2u4BPATuX66elGEm9NreCbYQZ4Pyh7+cBR5dou5WmDDMvyRuAPwE+XVWPzO+vqmdq4CXgSwxKPssysEvqtdm030bYB2xOsinJRgbBe+/CRkkuAs4AHh7atxG4F7izqr6yoP05zZ8BrgQeHdURSzGSem1cE5Sq6liS64D7gQ3AbVV1MMlOYH9VzQf5bcDuqhou01wNvB94c5Jrmn3XVNUB4K4kZzMo9RwAPj6qL3n1tSfv5ae/PV0d0lTw4akW85qzLlz1YMXPn/fPW8ec3575T+tiNpMZu6Re6+ArTw3skvrNtWIkqWPaLBWw3hjYJfWaL9qQpI6xFCNJHWNgl6SO6eL4agO7pF6zxi5JHeOoGEnqmLkOFmMM7JJ6zYenktQx3cvXDeySeq6LGfvY1mNPcluSZ5M8OrTvzCTfSPJE8+cZ47qfJI3DsVTrbb0Y54s2bge2LNh3I/BAVW2meVffGO8nSatWK9jWi7EF9qr6JvDcgt1XAHc0n+9g8PYPSZoaY3w13tQ41TX2X62qZ2Dw3r4kv3KK7ydJK9LF4Y5T8c7TJDuS7E+y/wt37Zl0dyT1SBdLMac6Y/+bJOc02fo5wLOLNaqqXcAu8NV4ktbWeiqxtHWqM/a9wPbm83bgq6f4fpK0IrNU6229GFvGnuRu4APAWUlmgN8BPgvck+Ra4CngI+O6nySNQxcz9rEF9qratsShS8d1D0kat1pHmXhbzjyV1Gtm7JLUMV0c7mhgl9Rr3QvrBnZJPXesg6HdwC6p13x4Kkkd08WHp1OxpIAkTUqt4L9RkmxJ8liSw0lOWM02yS1JDjTb40meHzq2vVni/Ikk24f2//0k322u+ftJRr5+24xdUq+NK2NPsgG4FbgMmAH2JdlbVYfm21TVDUPtrwfe3Xw+k8GkzvcweJ77rebcHwGfA3YAjwD3MVge/WvL9cWMXVKvzVa13ka4BDhcVUeq6mVgN4Oly5eyDbi7+fwh4BtV9VwTzL8BbGnW2HpDVT1cVQXcSYvlz83YJfXaGMexnws8PfR9BnjvYg2TvA3YBDy4zLnnNtvMIvuXZcYuqddWUmMfXmK82XYMXWqx2vdSvzW2AnuqanbEuSu55nFm7JJ6bSU19uElxhcxA5w/9P084OgSbbcCn1hw7gcWnPtnzf7zWl7zODN2Sb02R7XeRtgHbE6yKclGBsF778JGSS4CzgAeHtp9P3B5kjOSnAFcDtzfvIHuJ0l+vRkN8zFaLH9uxi6p18Y1QamqjiW5jkGQ3gDcVlUHk+wE9lfVfJDfBuxuHobOn/tckn/H4JcDwM6qmn+H9L8Cbgdez2A0zLIjYgBSo5/0rinfoKTF5PW/POkuaAq95qwLR47pHuWfvPUft4459z71X1Z9v7Vgxi6p11zdUZqQ+tlPJt0FdVQXlxQwsEvqNRcBk6SOsRQjSR0zbQNIxsHALqnXZs3YJalbLMVIUsdYipGkjjFjl6SOcbijJHVMixdorDsGdkm9ZilGkjrGwC5JHeOoGEnqGDN2SeoYR8VIUsfMVvcW7jWwS+o1a+yS1DFdrLGfttITktyW5Nkkjw7tOzPJN5I80fx5RrM/SX4/yeEk30ny98bZeUlarVrBf+vFigM7g7dlb1mw70bggaraDDzQfAf4TWBzs+0APndy3ZSkU2OuqvW2Xqw4sFfVN4HnFuy+Arij+XwHcOXQ/jtr4BHgTUnOOdnOStK4dTFjH1eN/Ver6hmAqnomya80+88Fnh5qN9Pse2ZM95WkVeniqJiTKcWsRBbZd8KvvSQ7kuxPsv8Ld+05xV2SpJ/rYilmXBn73yQ5p8nWzwGebfbPAOcPtTsPOLrw5KraBewCePnpb6+f/3uS1r31VGJpa1wZ+15ge/N5O/DVof0fa0bH/Drw4/mSjSRNAzN2IMndwAeAs5LMAL8DfBa4J8m1wFPAR5rm9wEfBg4DPwX+xRj6LElj08WMfcWBvaq2LXHo0kXaFvCJld5DktbKbM1Ougtj58xTSb3WxSUFTvWoGEmaanNU622UJFuSPNbMtr9xiTZXJzmU5GCSP2j2fTDJgaHtxSRXNsduT/Lk0LGLR/XDjF1Sr40rY0+yAbgVuIzBiMB9SfZW1aGhNpuBm4D3VdWP5uf8VNVDwMVNmzMZPJf8+tDlP1lVrceCG9gl9doYR7tcAhyuqiMASXYzmH1/aKjNvwRuraofAVTVsydcBa4CvlZVPz3ZjliKkdRrK1lSYHgyZbPtGLrUUjPth70deHuS/57kkSQL190C2ArcvWDfzc1Circkee2on8mMXVKvrWRJgeHJlItoM9P+dAaLIn6AwYTNv0jya1X1PEAzwfNdwP1D59wE/ADY2Nz7U8DO5fppxi6p16qq9TZCm5n2M8BXq+qVqnoSeIxBoJ93NXBvVb0y1L9nmoUUXwK+xKDksywDu6ReG+PM033A5iSbkmxkUFLZu6DNfwY+CJDkLAalmSNDx7exoAwzvyJukjBYOfdRRrAUI6nXxjUqpqqOJbmOQRllA3BbVR1MshPYX1V7m2OXJzkEzDIY7fJ/AJJcwCDj//MFl74rydkMSj0HgI+P6kumbXC+i4BJamvj+X93sbr2irzxl/5W65jz4xe+t+r7rQUzdkm9Nm3J7TgY2CX1WhdftGFgl9Rr62k53rYM7JJ6zVKMJHWM67FLUseYsUtSx3Sxxj5149j1c0l2NGtTSMf590KjuKTAdNsxuol6yL8XWpaBXZI6xsAuSR1jYJ9u1lG1GP9eaFk+PJWkjjFjl6SOMbBPmSQvDH3+0yTPJ/njSfZJkzf/9yLJxUkeTnKweQfmRyfdN00fSzFTJskLVfVLzedLgV8Afruq/tFke6ZJmv97keTtQFXVE0neAnwLeMf8OzMlMGOfalX1APCTSfdD06OqHq+qJ5rPR4FngbMn2ytNGwO7tE4luYTBm+u/N+m+aLq4Voy0DjUvOP4ysL2qg2+K0KqYsUvrTJI3AH8CfLqqHpl0fzR9DOzSOpJkI3AvcGdVfWXS/dF0MrBPsSR/AXwFuDTJTJIPTbpPmrirgfcD1yQ50GwXT7pTmi4Od5SkjjFjl6SOMbBLUscY2CWpYwzsktQxBnZJ6hgDuyR1jIFdkjrGwC5JHfP/Ae0IfiSlH1X0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faf91e34b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "def GridSearch_Heatmap(scores):\n",
    "    plt.figure()\n",
    "    sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n",
    "    plt.yticks(rotation=0);\n",
    "\n",
    "GridSearch_Heatmap(get_scores())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 4\n",
    "----\n",
    "\n",
    "This assignment is based on a data challenge from the Michigan Data Science Team ([MDST](http://midas.umich.edu/mdst/)). \n",
    "\n",
    "The Michigan Data Science Team ([MDST](http://midas.umich.edu/mdst/)) and the Michigan Student Symposium for Interdisciplinary Statistical Sciences ([MSSISS](https://sites.lsa.umich.edu/mssiss/)) have partnered with the City of Detroit to help solve one of the most pressing problems facing Detroit - blight. [Blight violations](http://www.detroitmi.gov/How-Do-I/Report/Blight-Complaint-FAQs) are issued by the city to individuals who allow their properties to remain in a deteriorated condition. Every year, the city of Detroit issues millions of dollars in fines to residents and every year, many of these fines remain unpaid. Enforcing unpaid blight fines is a costly and tedious process, so the city wants to know: how can we increase blight ticket compliance?\n",
    "\n",
    "The first step in answering this question is understanding when and why a resident might fail to comply with a blight ticket. This is where predictive modeling comes in. For this assignment, our task is to predict whether a given blight ticket will be paid on time.\n",
    "\n",
    "All data for this assignment has been provided to through the [Detroit Open Data Portal](https://data.detroitmi.gov/).\n",
    "\n",
    "We will use the following files for training and validating our models: train.csv and test.csv. Each row in these two files corresponds to a single blight ticket, and includes information about when, why, and to whom each ticket was issued. The target variable is compliance, which is True if the ticket was paid early, on time, or within one month of the hearing data, False if the ticket was paid after the hearing date or not at all, and Null if the violator was found not responsible. Compliance, as well as a handful of other variables that will not be available at test-time, are only included in train.csv.\n",
    "\n",
    "Note: All tickets where the violators were found not responsible are not considered during evaluation. They are included in the training set as an additional source of data for visualization, and to enable unsupervised and semi-supervised approaches. However, they are not included in the test set.\n",
    "\n",
    "### Data fields\n",
    "---\n",
    "\n",
    "** train.csv & test.csv **\n",
    "\n",
    "    ticket_id - unique identifier for tickets\n",
    "    agency_name - Agency that issued the ticket\n",
    "    inspector_name - Name of inspector that issued the ticket\n",
    "    violator_name - Name of the person/organization that the ticket was issued to\n",
    "    violation_street_number, violation_street_name, violation_zip_code - Address where the violation occurred\n",
    "    mailing_address_str_number, mailing_address_str_name, city, state, zip_code, non_us_str_code, country - Mailing address of the violator\n",
    "    ticket_issued_date - Date and time the ticket was issued\n",
    "    hearing_date - Date and time the violator's hearing was scheduled\n",
    "    violation_code, violation_description - Type of violation\n",
    "    disposition - Judgment and judgement type\n",
    "    fine_amount - Violation fine amount, excluding fees\n",
    "    admin_fee - $20 fee assigned to responsible judgments\n",
    "state_fee - $10 fee assigned to responsible judgments\n",
    "    late_fee - 10% fee assigned to responsible judgments\n",
    "    discount_amount - discount applied, if any\n",
    "    clean_up_cost - DPW clean-up or graffiti removal cost\n",
    "    judgment_amount - Sum of all fines and fees\n",
    "    grafitti_status - Flag for graffiti violations\n",
    "    \n",
    "** train.csv only **\n",
    "\n",
    "    payment_amount - Amount paid, if any\n",
    "    payment_date - Date payment was made, if it was received\n",
    "    payment_status - Current payment status as of Feb 1 2017\n",
    "    balance_due - Fines and fees still owed\n",
    "    collection_status - Flag for payments in collections\n",
    "    compliance [target variable for prediction] \n",
    "     Null = Not responsible\n",
    "     0 = Responsible, non-compliant\n",
    "     1 = Responsible, compliant\n",
    "    compliance_detail - More information on why each ticket was marked compliant or non-compliant\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "def blight_model():\n",
    "    \n",
    "    train_data = pd.read_csv('datasets/train.csv', encoding='latin-1')\n",
    "    test_data = pd.read_csv('datasets/test.csv')\n",
    "    addresses = pd.read_csv('datasets/addresses.csv')\n",
    "    latlons = pd.read_csv('datasets/latlons.csv')\n",
    "    addresses = addresses.merge(latlons, how='left', on='address')\n",
    "    addresses.drop(['address'], axis=1, inplace=True)\n",
    "    train_data = train_data.merge(addresses, on='ticket_id', how='left')\n",
    "    train_data = train_data[~train_data.compliance.isnull()]\n",
    "    test_data = test_data.merge(addresses, on='ticket_id', how='left')\n",
    "    features = ['agency_name', 'state', 'country', 'violation_code', 'disposition',\n",
    "                'fine_amount', 'admin_fee', 'state_fee', 'late_fee', 'discount_amount', \n",
    "                'clean_up_cost', 'judgment_amount', 'grafitti_status', 'lat', 'lon']\n",
    "    X_train = train_data[features]\n",
    "    y_train = train_data['compliance']\n",
    "    X_test = test_data[features]\n",
    "\n",
    "    X_merged = pd.concat([X_train, X_test], axis=0)\n",
    "    X_merged = pd.get_dummies(X_merged).reset_index(drop=True)\n",
    "\n",
    "    X_train = X_merged.ix[0:X_train.shape[0]-1,:]\n",
    "    X_test = X_merged.ix[X_train.shape[0]:,:]\n",
    "\n",
    "    # Impute missing data\n",
    "    from sklearn.preprocessing import Imputer\n",
    "    imp = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
    "    imp.fit(X_train)\n",
    "    X_train = imp.transform(X_train)\n",
    "    X_test = imp.transform(X_test)\n",
    "\n",
    "    from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "    clf = GradientBoostingClassifier(n_estimators=100)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    preds = pd.Series(clf.predict_proba(X_test)[:,1])\n",
    "    preds.index = test_data['ticket_id']\n",
    "    \n",
    "    return preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "preds = blight_model()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}