Day 1 - Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "Logistic regression is a classification model which assumes that the probability of each class is proportional to a weighted sum of the input features.\n",
    "\n",
    "In terms of mathematics, it is basically the same as linear regression except we apply a sigmoid activation to our output to map it between 0 and 1 as it represents a probability.\n",
    "\n",
    "### $h = \\sigma(XW)$\n",
    "\n",
    "Even though Mean Squared Error Loss would work for this problem too, it has been shown that using Cross Entropy Loss leads to faster convergence.\n",
    "\n",
    "### $ J = \\sum_{i=1}^{m} - y^{(i)} \\cdot \\text{log}(h^{(i)}) + (1-y^{(i)}) \\cdot \\text{log}(1-h^{(i)})$\n",
    "\n",
    "### Multi-class case\n",
    "Consider the case where we are not performing binary classification but have multiple classes. How do we extend our regression model to work with this? We simply have multiple outputs where each output is a sigmoid applied to a linear combination of the inputs representing the probability of belonging to each class. This corresponds to adding an extra column in our weights vector so it is now a matrix. To calculate the cost we use the same cost function but sum the cost across all outputs.\n",
    "\n",
    "To represent the labels, we can't use a single scalar value representing the class number anymore since our output is a vector so we wouldn't be able to compare them. Instead, we use one-hot encode the label. This is when we have a vector of length K, where K is the number of classes, which has a value of 0 for all the numbers except for the label class number which has a value of 1. If you think about it intuitively, this makes sense as we want our model to predict a probability of 1 for belonging to the label class and 0 for all other classes.\n",
    "\n",
    "### Optimization\n",
    "As usual, we use gradient descent to optimize this. Calculating our outputs and using that to calculate the gradient of the cost w.r.t each parameter so we can update the parameters to reduce the cost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "We are going to be implementing a model that takes in 4 features which are measurements of a flower and it will predict which one of the three species it belongs to.\n",
    "\n",
    "We begin by importing the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the required dataset intro a pandas data frame. Map our class labels into numerical values and shuffle the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Id  SepalLengthCm  SepalWidthCm  PetalLengthCm  PetalWidthCm  Species\n",
      "34    35            4.9           3.1            1.5           0.1        0\n",
      "100  101            6.3           3.3            6.0           2.5        1\n",
      "0      1            5.1           3.5            1.4           0.2        0\n",
      "47    48            4.6           3.2            1.4           0.2        0\n",
      "145  146            6.7           3.0            5.2           2.3        1\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('Iris.csv')\n",
    "df[['Species']] = df['Species'].map({'Iris-setosa':0, 'Iris-virginica':1, 'Iris-versicolor':2}) #map text labels to numberical vaules\n",
    "df = df.sample(frac=1) #shuffle our dataset\n",
    "\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select the appropriate features and labels and convert them into torch tensors so we can use them in our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.Tensor(np.array(df[df.columns[1:-1]])) #pick our features from our dataset\n",
    "Y = torch.LongTensor(np.array(df[['Species']]).squeeze()) #select our label - squeeze() removes redundant dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data into training and test sets and convert to variables so we can use them with the torch autograd library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = 100 #size of training set\n",
    "\n",
    "#training set\n",
    "x_train = Variable(X[0:m])\n",
    "y_train = Variable(Y[0:m])\n",
    "\n",
    "#test set\n",
    "x_test = Variable(X[m:])\n",
    "y_test = Variable(Y[m:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the Logistic Model using PyTorch's class interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class logisticmodel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__() #call parent class initializer\n",
    "        self.linear = torch.nn.Linear(4, 3) #define linear combination function with 4 inputs and 3 outputs\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.linear(x) #linearly combine our inputs to give 3 outputs\n",
    "        x = torch.nn.functional.softmax(x, dim=1) #activate our output neurons to give probabilities of belonging to each of the three class\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the training hyperparameters, cost function and optimizer. Instantiate a model from the class we defined earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "no_epochs = 100\n",
    "lr = 0.1\n",
    "\n",
    "mymodel = logisticmodel() #create our model from defined class\n",
    "criterion = torch.nn.CrossEntropyLoss() #cross entropy cost function as it is a classification problem\n",
    "optimizer = torch.optim.Adam(mymodel.parameters(), lr = lr) #define our optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the axes which we will use to plot the costs. Define the function used to train the model and train it for the number of epochs specified earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "costs=[] #store the cost each epoch\n",
    "plt.ion()\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.set_xlabel('Epoch')\n",
    "ax.set_ylabel('Cost')\n",
    "ax.set_xlim(0, no_epochs)\n",
    "plt.show()\n",
    "\n",
    "#define train function\n",
    "def train(no_epochs):\n",
    "    for epoch in range(no_epochs):\n",
    "        h = mymodel.forward(x_train) #forward propagate - calulate our hypothesis\n",
    "\n",
    "        #calculate, plot and print cost\n",
    "        cost = criterion(h, y_train)\n",
    "        costs.append(cost.data[0])\n",
    "        ax.plot(costs, 'b')\n",
    "        fig.canvas.draw()\n",
    "        print('Epoch ', epoch, ' Cost: ', cost.data[0])\n",
    "\n",
    "        #calculate gradients + update weights using gradient descent step with our optimizer\n",
    "        optimizer.zero_grad()\n",
    "        cost.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "train(no_epochs) #train the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the accuracy of the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_h = mymodel.forward(x_test) #predict probabilities for test set\n",
    "_, test_h = test_h.data.max(1) #returns the output which had the highest probability\n",
    "test_y = y_test.data\n",
    "correct = torch.eq(test_h, test_y) #perform element-wise equality operation\n",
    "accuracy = torch.sum(correct)/correct.shape[0] #calculate accuracy\n",
    "print('Test accuracy: ', accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the optimized model parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(mynet.state_dict(), 'trained_logistic_model')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}