machine_learning_lecture1_2020.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ml_lecture1_2020.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ch_WRVLhRVDy",
        "colab_type": "text"
      },
      "source": [
        "# Introduction to Machine Learning in Python, May 2020\n",
        "By Dr. Anders Christensen `anders.christensen @ unibas.ch`\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVYtPM6QyYfJ",
        "colab_type": "text"
      },
      "source": [
        "## Part 1: Exploratory Data Analysis and Classificaton: Fisher's *Iris* Data Set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q70_UgxPyoIQ",
        "colab_type": "text"
      },
      "source": [
        "Our data is in the file `iris.csv`: https://www.dropbox.com/s/g3njhhml16kvaci/iris.csv\n",
        "\n",
        "The file contains 151 lines, the first 9 are here:\n",
        "\n",
        "\n",
        "```\n",
        "Petal length,Petal Width,Sepal Length,Sepal Width,Species\n",
        "5.1,3.5,1.4,0.2,setosa\n",
        "4.9,3.0,1.4,0.2,setosa\n",
        "4.7,3.2,1.3,0.2,setosa\n",
        "4.6,3.1,1.5,0.2,setosa\n",
        "5.0,3.6,1.4,0.2,setosa\n",
        "5.4,3.9,1.7,0.4,setosa\n",
        "4.6,3.4,1.4,0.3,setosa\n",
        "5.0,3.4,1.5,0.2,setosa\n",
        "```\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdfcVa3gE4mO",
        "colab_type": "text"
      },
      "source": [
        "Download the file using Linux's `wget` command:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X8PkleJY0BNn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget -O iris.csv https://www.dropbox.com/s/g3njhhml16kvaci/iris.csv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ujpu-lGD1Adn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Pandas DataFrames\n",
        "import pandas as pd\n",
        "\n",
        "# Read in the CSV as a Pandas DataFrame\n",
        "data = pd.read_csv(\"iris.csv\")\n",
        "\n",
        "type(data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9cpB10cW2Sop",
        "colab_type": "text"
      },
      "source": [
        "Can also read Excel and a bunch of other formats."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SsKUsCHv1H8a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.head(n=8)    \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTEHlkWf2aii",
        "colab_type": "text"
      },
      "source": [
        "Let's get an overview of the data?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D7lQq3ct2jEm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.describe()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfCwwdOT7_Bi",
        "colab_type": "text"
      },
      "source": [
        "#Part 1: Plotting Pandas DataFrames with Seaborn"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rfMe2XLV5Aqe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "grid = sns.pairplot(data)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LfOjRUTY6Wza",
        "colab_type": "text"
      },
      "source": [
        "Much room for customization: https://seaborn.pydata.org/generated/seaborn.pairplot.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BV7s1KrP5wM4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "grid = sns.pairplot(data, kind=\"reg\", diag_kind=\"kde\",  \n",
        "                    hue=\"Species\", corner=True)\n",
        "grid.fig.suptitle(\"Fisher's Iris Data Set\", size=32)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xIuqn1r6M0pP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.figure(figsize=(10,11))\n",
        "sns.heatmap(data.corr(),annot=True)\n",
        "plt.plot()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_Zv2rUS_FB-",
        "colab_type": "text"
      },
      "source": [
        "## Code demonstration 1: K-NearestNeighbor (KNN) Classifier for the Iris Dataset.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9JSEFaNQJCFg",
        "colab_type": "text"
      },
      "source": [
        "Look at the label of the $K$ nearest neighbors. A given point is probably the same as its closest neighbors!\n",
        "\n",
        "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/e7/KnnClassification.svg/1280px-KnnClassification.svg.png\" width=\"250\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u0fdTlMrKyZ4",
        "colab_type": "text"
      },
      "source": [
        "Training and test data?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q0VdOCM9K4O2",
        "colab_type": "text"
      },
      "source": [
        "Splitting of dataset?\n",
        "\n",
        "Often a dataset is split into two sets:\n",
        "\n",
        "*   **Training set:** Data to train your model\n",
        "*   **Test set:** Data to evaluate/benchmark your model\n",
        "\n",
        "Important that the two do ***not*** overlap -- this is overfitting!\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1F482YkhF-Gg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# \"Cheating\" using function from Scikit-learn\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Make test set 70%/30% split\n",
        "train, test = train_test_split(data, test_size=0.30)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p9SU3muM4xVx",
        "colab_type": "text"
      },
      "source": [
        "Let's see what is in the training and test:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vf7Jaopd4fRY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XHof9E334fUW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OMeNgz9y4cmm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_features = train[[\"Sepal Length\",\"Sepal Width\",\"Petal length\",\"Petal Width\"]]\n",
        "train_labels = train[\"Species\"]\n",
        "\n",
        "test_features = test[[\"Sepal Length\",\"Sepal Width\",\"Petal length\",\"Petal Width\"]]\n",
        "test_labels = test[\"Species\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUo-9B252MY4",
        "colab_type": "text"
      },
      "source": [
        "First step to a simple classifier:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rav2Vnga0Lyt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# L2 distance\n",
        "def distance(A, B):\n",
        "  \n",
        "  difference = A - B\n",
        "  d = np.linalg.norm(difference)\n",
        "  \n",
        "  return d\n",
        "\n",
        "\n",
        "query = np.asarray(test_features)[0]\n",
        "train_features = np.asarray(train_features)\n",
        "\n",
        "for i in range(len(train_features)):\n",
        "\n",
        "  dist = distance(query, train_features[i])\n",
        "\n",
        "  print(i, dist)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTWvgdFrzthr",
        "colab_type": "text"
      },
      "source": [
        "## Demonstration of Scikit-Learn:\n",
        "\n",
        "Fortunately we don't always have to implement a classifier, etc from scratch. Sklearn already has a KNN classifier we can use:\n",
        "\n",
        "https://scikit-learn.org/stable/index.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oti7fnfLz0OE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import the classifier\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "\n",
        "# Initialize the object\n",
        "classifier = KNeighborsClassifier(n_neighbors=15)\n",
        "\n",
        "# Fit the model\n",
        "classifier.fit(train_features, train_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ENJ4A94138FQ",
        "colab_type": "text"
      },
      "source": [
        "KNN documentation: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBLbspr13bfQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Make a prediction\n",
        "predicted_labels = classifier.predict(test_features)\n",
        "print(predicted_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oHm9eM9w7_1X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "from sklearn import metrics\n",
        "\n",
        "accuracy = metrics.accuracy_score(predicted_labels, test_labels)\n",
        "print(\"KNN Classifier Accuracy is {:4.1f}%\".format(accuracy * 100))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3hcxX8n16oEg",
        "colab_type": "text"
      },
      "source": [
        "How about another classifier?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2pNKvM006oJ1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.neighbors import NearestCentroid\n",
        "classifier = NearestCentroid()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XIFkOIQgIfX2",
        "colab_type": "text"
      },
      "source": [
        "## Part 2: Linear Least Squares Regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3NXEAa2MAlV",
        "colab_type": "text"
      },
      "source": [
        "In the exercieses today, you will implment a linear least squares regression model (often just called linear regression). Here is an introduction to linear least squares regression."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XTeUKBrqNvcH",
        "colab_type": "text"
      },
      "source": [
        "First, lets start with some data for a linear function:\n",
        "\\begin{equation}\n",
        "y = 1.2x + \\mathrm{noise}\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LoRp09TFMIO0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.random.seed(666)\n",
        "\n",
        "# X-values\n",
        "x = np.arange(0,20.0, 0.2)\n",
        "\n",
        "# Y-values: Y = 1.2*X + random noise \n",
        "y = 1.2 * x + np.random.normal(scale=2.0, size=len(x))\n",
        "\n",
        "print(x.shape)\n",
        "print(y.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XfbtW4rFN7Nu",
        "colab_type": "text"
      },
      "source": [
        "Our little \"Exploratory Data Analysis\":"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OLwAyw1Mj8p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.scatter(x,y, color=\"C0\", label=\"Training\")\n",
        "plt.plot(x, x*1.2, color=\"C2\", label=\"Truth\")\n",
        "plt.grid(True)\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NzPL2zoDOpOV",
        "colab_type": "text"
      },
      "source": [
        "Linear Regression approximates a linear function, e.g.:\n",
        "\n",
        "\\begin{equation}\n",
        "y(\\mathbf{x}) = x_1 \\alpha_1 + x_2 \\alpha_2 + \\dots + x_n \\alpha_n\n",
        "\\end{equation}\n",
        "or in vector notation:\n",
        "\\begin{equation}\n",
        "y(\\mathbf{x}) = \\mathbf{x} \\cdot \\mathbf{\\alpha}\n",
        "\\end{equation}\n",
        "\n",
        "Where $\\mathbf{x}$ is our feature vector/descriptor/representation for a given datapoint. $\\mathbf{\\alpha}$ is the vector of regression coefficients.\n",
        "\n",
        "\"Fitting\" is what you do to find the best set of $\\alpha$-values. This is done by finding the solution with the \"least squares\":\n",
        "\n",
        "\\begin{equation}\n",
        "\\mathbf{y} = \\mathbf{X}\\mathbf{\\alpha}\n",
        "\\end{equation}\n",
        "\n",
        "Minimze the error:\n",
        "\\begin{equation}\n",
        "\\mathbf{\\hat{\\alpha}} = \\text{arg min} || \\mathbf{y}^\\text{ref} - \\mathbf{X}\\mathbf{\\alpha}||^2\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g2xW6GboPRac",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Small hack for 1-d representations\n",
        "X_matrix = x.reshape((100,1))\n",
        "\n",
        "# Run the solver\n",
        "alpha, residual, rank, singular_values = np.linalg.lstsq(X_matrix, y)\n",
        "\n",
        "print(alpha)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xeIEDGTy64r",
        "colab_type": "text"
      },
      "source": [
        "How can we interpret $\\alpha$ for our 1-D function?\n",
        "\\begin{equation}\n",
        "y(\\mathbf{x}) = x_1 \\alpha_1\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YSuw4M2mx8T3",
        "colab_type": "text"
      },
      "source": [
        "Let's try some predictions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dbaCS4fqx_O_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Random x-values:\n",
        "x_test = np.random.random(size=(15,1)) * 20.0\n",
        "\n",
        "print(x_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "daj8nM5ByPh3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(alpha.shape)\n",
        "print(x_test.shape)\n",
        "\n",
        "y_test = np.matmul(x_test, alpha)\n",
        "\n",
        "plt.plot(x, x*1.2, color=\"C2\", label=\"Truth\", zorder=-1)\n",
        "plt.scatter(x,y, color=\"C0\", label=\"Training\")\n",
        "plt.scatter(x_test,y_test, color=\"C3\", label=\"Test\")\n",
        "\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}