3.2_Logistic_Regression.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3.2%20Logistic%20Regression.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 2",
      "language": "python",
      "name": "python2"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "4SPV5rlnPqjb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Logistic regression learning algorithm example using TensorFlow library."
      ]
    },
    {
      "metadata": {
        "id": "bDTVXgf1Pqjc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Logistic regression model is one of the simplest classification models. The most basic form deals with classifing a given set of data points into two possible classes, usually labelled as  **0**  and **1**. The logistic regression model thus predicts an output y in {**0**,**1**}, given an input vector **x**. The probability is modeled using the logistic function $$ g(z)=1/(1+e^{-z})$$Namely, the probability of finding the output **y=1** is given by\n",
        "$$ q_{{y=1}}\\ =\\ {\\hat  {y}}\\ \\equiv \\ g({\\mathbf  {w}}\\cdot {\\mathbf  {x}} + b)\\,,$$\n",
        "while the probability of finding **y=0** is given by\n",
        "$$ q_{{y=0}} = 1 - q_{{y=1}}$$\n",
        "\n",
        "Weights **w** are usually learned in the training step by using some optimization algorithem like gradient descent.\n",
        "\n",
        "The typical loss function that one uses in logistic regression is computed by taking the average of all cross-entropies in the sample. For example, suppose we have N￼ samples the loss function is then given by:\n",
        "$$L(w)\\frac{1}{N}\\sum_{n=1}^{N}H(p_{n},q_{n})=-{\\frac  1N}\\sum_{{n=1}}^{N}\\ {\\bigg [}y_{n}\\log {\\hat  y}_{n}+(1-y_{n})\\log(1-{\\hat  y}_{n}){\\bigg ]}$$\n",
        "\n",
        "In this example we will use MNIST database of handwritten digits provided in the tensorflow package. The corresponding labels in MNIST are numbers between 0 and 9, describing which digit a given image is. In order to deal with this problem we are going to use label representation of \"one-hot vectors\". A one-hot vector representation is a vector which is 0 in most dimensions, and 1 in a single dimension. In this case, the nth digit will be represented as a vector which is 1 in the nth dimensions. For example, 3 would be [0,0,0,1,0,0,0,0,0,0]. \n",
        "\n",
        "In the case of multiclass the output is given by:\n",
        "$$ \\hat{y} = softmax(g(w⋅x + b))$$\n",
        "which can be simplified by:\n",
        "$$ \\hat{y} = softmax(w⋅x + b)$$ \n",
        "and the loss is defined as:\n",
        "$$ L(w) = \\frac{1}{N}\\sum_{n=1}^{N}H(p_{n},q_{n})=-\\frac{1}{N}\\sum_{n=1}^{N}y_{n}log(\\hat{y}_{n})$$"
      ]
    },
    {
      "metadata": {
        "id": "T5lJYt2PRPXV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<br>Let's create a Logistic Regression classifier with the classical MNIST Dataset.\n",
        "\n",
        "MNIST is a simple computer vision dataset. It consists of images of handwritten digits like these:\n",
        "![alt text](https://www.tensorflow.org/versions/r1.0/images/MNIST.png)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Vyq3jHU3Pqjc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 411
        },
        "outputId": "9df27a5b-713a-4559-f673-3e6955a3e697",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528878763734,
          "user_tz": -120,
          "elapsed": 5051,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Import MINST data\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "mnist = input_data.read_data_sets(\"tmp/data/\", one_hot=True)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-1-4de56eb9035a>:2: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
            "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please write your own downloading logic.\n",
            "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.data to implement this functionality.\n",
            "Extracting tmp/data/train-images-idx3-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.data to implement this functionality.\n",
            "Extracting tmp/data/train-labels-idx1-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.one_hot on tensors.\n",
            "Extracting tmp/data/t10k-images-idx3-ubyte.gz\n",
            "Extracting tmp/data/t10k-labels-idx1-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: __init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zN6hibGQPqjh",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "53e8ca5f-c2fd-4763-cbed-a84a0d904fb3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528878764273,
          "user_tz": -120,
          "elapsed": 523,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "#import tensorflow\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "# tf Graph Input\n",
        "X = tf.placeholder(\"float\", [None, 784]) # mnist data image of shape 28*28=784\n",
        "y = tf.placeholder(\"float\", [None, 10]) # 0-9 digits recognition => 10 classes\n",
        "\n",
        "# Create model\n",
        "# Set model weights\n",
        "W = tf.Variable(tf.zeros([784, 10]))\n",
        "b = tf.Variable(tf.zeros([10]))\n",
        "\n",
        "# Construct model\n",
        "y_pred = tf.nn.softmax(tf.add(tf.matmul(X, W),b)) # Softmax  "
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YcTPnA0ZPqjj",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "70efd742-18d4-406e-833b-058f0da944e7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528878764814,
          "user_tz": -120,
          "elapsed": 486,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Define Training Parameters\n",
        "learning_rate = 0.01\n",
        "training_epochs = 25\n",
        "batch_size = 100\n",
        "display_step = 1"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0L0nz5qCPqjl",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "faa239e0-7e26-4e6b-86ef-2b1cf62a906d",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528878765478,
          "user_tz": -120,
          "elapsed": 596,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Minimize error using cross entropy\n",
        "# Cross entropy\n",
        "cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(y_pred), reduction_indices=1))\n",
        "\n",
        "# Gradient Descent\n",
        "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) "
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8_ZjaxyNPqjn",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 476
        },
        "outputId": "243e8165-ba60-4900-b182-07c5d77116e1",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528878795152,
          "user_tz": -120,
          "elapsed": 29625,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Initializing the variables\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "# Launch the graph\n",
        "with tf.Session() as sess:\n",
        "    sess.run(init)\n",
        "\n",
        "    # Training cycle\n",
        "    for epoch in range(training_epochs):\n",
        "        avg_cost = 0.\n",
        "        total_batch = int(mnist.train.num_examples/batch_size)\n",
        "        # Loop over all batches\n",
        "        for i in range(total_batch):\n",
        "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
        "            # Fit training using batch data\n",
        "            sess.run([optimizer, cost ],  feed_dict={X: batch_xs, y: batch_ys})\n",
        "            # Compute average loss\n",
        "            avg_cost += sess.run(cost, feed_dict={X: batch_xs, y: batch_ys})/total_batch\n",
        "        # Display logs per epoch step\n",
        "        if epoch % display_step == 0:\n",
        "            print \"Epoch:\", '%04d' % (epoch+1), \"cost=\", \"{:.9f}\".format(avg_cost)\n",
        "\n",
        "    print \"Optimization Finished!\"\n",
        "\n",
        "    # Test model\n",
        "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y, 1))\n",
        "    # Calculate accuracy\n",
        "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
        "    print \"Accuracy:\", accuracy.eval({X: mnist.test.images, y: mnist.test.labels})"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0001 cost= 1.176532306\n",
            "Epoch: 0002 cost= 0.662508476\n",
            "Epoch: 0003 cost= 0.550652506\n",
            "Epoch: 0004 cost= 0.496730035\n",
            "Epoch: 0005 cost= 0.463776750\n",
            "Epoch: 0006 cost= 0.440948209\n",
            "Epoch: 0007 cost= 0.423931220\n",
            "Epoch: 0008 cost= 0.410649940\n",
            "Epoch: 0009 cost= 0.399874245\n",
            "Epoch: 0010 cost= 0.390927679\n",
            "Epoch: 0011 cost= 0.383369292\n",
            "Epoch: 0012 cost= 0.376792346\n",
            "Epoch: 0013 cost= 0.371000649\n",
            "Epoch: 0014 cost= 0.365940430\n",
            "Epoch: 0015 cost= 0.361393990\n",
            "Epoch: 0016 cost= 0.357267638\n",
            "Epoch: 0017 cost= 0.353549642\n",
            "Epoch: 0018 cost= 0.350162132\n",
            "Epoch: 0019 cost= 0.347016938\n",
            "Epoch: 0020 cost= 0.344117598\n",
            "Epoch: 0021 cost= 0.341454458\n",
            "Epoch: 0022 cost= 0.339016740\n",
            "Epoch: 0023 cost= 0.336675162\n",
            "Epoch: 0024 cost= 0.334488640\n",
            "Epoch: 0025 cost= 0.332450373\n",
            "Optimization Finished!\n",
            "Accuracy: 0.9139\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "L4Syjh6oPqjr",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "7f33def0-acdd-461c-8f97-37811bec69ea",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528878795728,
          "user_tz": -120,
          "elapsed": 565,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 5,
      "outputs": []
    }
  ]
}