2_1_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning with Python "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Classification\n",
    "\n",
    "Now we will move on to supervised methods, starting with some different approaches to classification. Let's use the `moons` data from our clustering examples but increase the noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = make_moons(n_samples=100, noise=0.25, random_state=1)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "\n",
    "\n",
    "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train)\n",
    "plt.xlabel(\"Feature 0\")\n",
    "plt.ylabel(\"Feature 1\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [k-Nearest Neighbours](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier)\n",
    "\n",
    "In many respects this is the simplest supervised algorithm:\n",
    "\n",
    "* Choose an integer value for *k*.\n",
    "* Choose a [distance metric](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html#sklearn.neighbors.DistanceMetric). The default is `minkowski`, which corresponds to Euclidean distance when the power parameter `p=2`(also the default setting).\n",
    "* Given an input feature vector **x**, rank the training data by increasing distance from **x**.\n",
    "* Predict the class label by majority vote from the *k* nearest instances to **x**.\n",
    "\n",
    "This simplicity makes k-NN a good baseline for comparing with the performance of other methods.\n",
    "\n",
    "\"Training the model\" consists only of storing the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "k = 1\n",
    "knn = KNeighborsClassifier(n_neighbors=k, metric='minkowski', p=2)\n",
    "knn.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make a prediction for any new data point. By considering a grid of points, we can construct the *decision boundary*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_boundary(model,X_p,y_p,title):\n",
    "    # Plot the decision boundary. For that, we will assign a color to each\n",
    "    # point in the mesh [x_min, x_max]x[y_min, y_max].\n",
    "    h = 0.05\n",
    "    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "\n",
    "    # Put the result into a color plot\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    plt.figure(figsize=(6, 6))\n",
    "    plt.contourf(xx, yy, Z, alpha=0.2)\n",
    "\n",
    "    # Plot also the training points\n",
    "    plt.scatter(X_p[:, 0], X_p[:, 1], c=y_p)\n",
    "\n",
    "    plt.xlim(xx.min(), xx.max())\n",
    "    plt.ylim(yy.min(), yy.max())\n",
    "    plt.title(title)\n",
    "    plt.xlabel(\"Feature 0\")\n",
    "    plt.ylabel(\"Feature 1\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(knn,X_train,y_train,\"k-NN(k=\" + str(k) + \") + training set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite its simplicity, k-NN performs very well on our test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(knn,X_test,y_test,\"k-NN(k=\" + str(k) + \") + test set\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = knn.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the *accuracy* as a basic performance metric:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred == y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( \"accuracy =\", np.mean(y_pred == y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can investigate what happens to accuracy as we increase the value of *k*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = np.zeros(50)\n",
    "for k in range(1,51):\n",
    "    knn = KNeighborsClassifier(n_neighbors=k, metric='minkowski', p=2)\n",
    "    knn.fit(X_train,y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    acc[k-1] = np.mean(y_pred == y_test)\n",
    "    \n",
    "plt.scatter(np.arange(1,51),acc, c='c')\n",
    "plt.title( \"Test accuracy\")\n",
    "plt.xlabel(\"k\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When *k=1*, the model is *overfitting* to the training data. However, if we set *k* too high then the local information will be too diluted, resulting in *underfitting*.\n",
    "\n",
    "Looking at the plot of accuracy calculated for the training data helps to illustrate this point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = np.zeros(50)\n",
    "for k in range(1,51):\n",
    "    knn = KNeighborsClassifier(n_neighbors=k, metric='minkowski', p=2)\n",
    "    knn.fit(X_train,y_train)\n",
    "    y_pred = knn.predict(X_train)\n",
    "    acc[k-1] = np.mean(y_pred == y_train)\n",
    "    \n",
    "plt.scatter(np.arange(1,51),acc, c='m')\n",
    "plt.title( \"Training accuracy\")\n",
    "plt.xlabel(\"k\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Logistic regression](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression)\n",
    "\n",
    "Logistic regression is a simple method for linear classification. Because it is a linear method, the decision boundary will be a *hyperplane* in the feature space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "lr = LogisticRegression(penalty='none')\n",
    "lr.fit(X_train,y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(lr,X_train,y_train,\"Logistic regression + training set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a nonlinear classification task, so logistic regression is not able to capture the detailed shape of the training data. However, performance on this particular test data set is still reasonably good:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(lr,X_test,y_test,\"Logistic regression + test set\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = lr.predict(X_test)\n",
    "print( \"accuracy =\", np.mean(y_pred == y_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Decision Tree](https://scikit-learn.org/stable/modules/tree.html)\n",
    "\n",
    "A decision tree is one way to implement a nonlinear decision boundary:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(random_state=0)\n",
    "tree.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(tree,X_train,y_train,\"Decision Tree + training set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the decision tree's decision boundary has no underlying parametrisation, it may overfit when the training data is sparse (i.e. does not have good coverage of the feature space)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ensemble method [*random forest*](https://scikit-learn.org/stable/modules/ensemble.html#random-forests) mitigates this problem by resampling the training data to generate variation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rf = RandomForestClassifier(n_estimators=100,random_state=0)\n",
    "rf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(rf,X_train,y_train,\"Random Forest + training set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, the random forest is less prone to overfitting than the single tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = tree.predict(X_test)\n",
    "print( \"accuracy =\", np.mean(y_pred == y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = rf.predict(X_test)\n",
    "print( \"accuracy =\", np.mean(y_pred == y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the tree and the forest can report *feature importances*, which can be helpful to gain insight into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importances = rf.feature_importances_\n",
    "print(importances)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows us that *if removed from the model*, feature 1 will impact performance more than feature 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Support Vector Machines](https://scikit-learn.org/stable/modules/svm.html)\n",
    "\n",
    "Another example of a nonlinear classification algorithm. \n",
    "\n",
    "Conceptually, the SVM deals with nonlinearity by projecting into a *higher*-dimensional space in which the training data are linearly separable. The particular form of the transformation is called the [*kernel function*](https://scikit-learn.org/stable/modules/svm.html#kernel-functions). In practice, the so-called [*kernel trick*](https://towardsdatascience.com/the-kernel-trick-c98cdbcaeb3f) means that we can compute the separating hyperplane without actually needing to perform any expensive high-dimensional transformations.\n",
    "\n",
    "Different tasks will require different choices of kernel function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "svc = SVC(kernel='rbf')\n",
    "svc.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(svc,X_train,y_train,\"Support Vector Classifier + training set\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = svc.predict(X_test)\n",
    "print( \"accuracy =\", np.mean(y_pred == y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural network\n",
    "\n",
    "A neural network can be an extremely flexible way to learn a nonlinear decision boundary.\n",
    "\n",
    "More expressive power is gained from multiple hidden layers, at the expense of adding many parameters to the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "nn = MLPClassifier(hidden_layer_sizes=(10,50),max_iter=10000,random_state=0)\n",
    "nn.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.coefs_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.intercepts_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_boundary(nn,X_train,y_train,\"Neural Network + training set\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = nn.predict(X_test)\n",
    "print( \"accuracy =\", np.mean(y_pred == y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For difficult unstructured inputs such as image data, a neural network may be the best option as it has the potential to extract meaningful features despite rotations, translations and scaling. However, tuning the model metaparameters to the specific problem can be challenging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Train a classifier of your choice on the `digits` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the accuracy of your model, evaluated on the test data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does your model do better than random guessing?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}