BackProp-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Back Propagation Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do backprop in some concrete examples. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the package and set device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "from math import exp, log\n",
    "\n",
    "device = torch.device('cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Network\n",
    "\n",
    "The network is: \n",
    "\n",
    "$$ x = A^0 = \n",
    "\\begin{bmatrix}\n",
    "  0.35 & 0.9\n",
    "\\end{bmatrix}$$\n",
    "$$ W^1  = \n",
    "\\begin{bmatrix}\n",
    "  0.1 & 0.4 \\\\\n",
    "  0.8 & 0.6\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "$$ W^2  = \n",
    "\\begin{bmatrix}\n",
    "  0.3  \\\\\n",
    "  0.9 \n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "\n",
    "![Network 1](./BP_example.png \"Network 1\")\n",
    "\n",
    "\n",
    "Or simply, \n",
    "$$ \n",
    "A^0 \\stackrel{W^1}{\\Longrightarrow} Z^1 \\stackrel{\\sigma}{\\Longrightarrow} \n",
    "A^1 \\stackrel{W^2}{\\Longrightarrow} Z^2 \\stackrel{\\sigma}{\\Longrightarrow} \n",
    "A^2\n",
    "$$\n",
    "and $x = A^0$, $y = A^2$, ground truth $t = 0.5$ \n",
    "\n",
    "All activation functions are sigmoid. \n",
    "\n",
    "Note that \n",
    "1. there is no biases in the network. \n",
    "2. $x$ is a row vector.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation in Numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _acti(z): \n",
    "    return 1.0 / (1.0 + exp(-z))\n",
    "\n",
    "def acti(zs):\n",
    "    return 1.0/(1.0 + np.exp(-zs))\n",
    "\n",
    "def acti_old(zs):\n",
    "    s = zs.shape\n",
    "    dup = zs.reshape(-1)\n",
    "    return np.fromiter( (_acti(elem) for elem in dup), zs.dtype ).reshape(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _grad(z):\n",
    "    return _acti(z) * (1 - _acti(z))\n",
    "\n",
    "\n",
    "def grad(zs):\n",
    "    s = zs.size\n",
    "    return np.multiply(acti(zs),(-acti(zs) + 1.0))\n",
    "\n",
    "def grad_old(zs):\n",
    "    return np.fromiter( (_grad(elem) for elem in zs), zs.dtype )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def loss(y, t): \n",
    "    return 0.5*(y-t).T*(y-t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def p(*vals, prec = 8,):\n",
    "    for val in vals:\n",
    "        if isinstance(val, str):\n",
    "            print(val, end=' ')\n",
    "        else: # assume numeric\n",
    "            print(val.round(prec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 8]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "matrix([[ 0.19661193,  0.19661193],\n",
       "        [ 0.10499359,  0.04517666]])"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test the above functions\n",
    "a = np.array([1, 2])\n",
    "b = np.array([3, 4])\n",
    "print(a * b)\n",
    "a = np.matrix([[1, -1], [2, 3]])\n",
    "#acti(a)\n",
    "grad(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Z1, A1 =  [[ 0.755  0.68 ]]\n",
      "[[ 0.6802672  0.6637387]]\n",
      "Z2, A2 =  [[ 0.80144499]]\n",
      "[[ 0.69028349]]\n",
      "loss= [[ 0.0181039]]\n"
     ]
    }
   ],
   "source": [
    "# specify the model. We use numpy.matrix as numpy 1d array cannot do transpose()\n",
    "A0 = np.matrix([0.35, 0.9])\n",
    "W1 = np.matrix([[0.1, 0.4], [0.8, 0.6]])\n",
    "W2 = np.array([[0.3], [0.9]])\n",
    "t = np.array([0.5])\n",
    "\n",
    "# forward pass\n",
    "Z1 = np.dot(A0, W1)\n",
    "A1 = acti(Z1)\n",
    "Z2 = np.dot(A1, W2)\n",
    "A2 = acti(Z2)\n",
    "y = A2\n",
    "\n",
    "# print\n",
    "p('Z1, A1 = ', Z1, A1)\n",
    "p('Z2, A2 = ', Z2, A2)\n",
    "p('loss=', loss(y, t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compute the Gradients on Parameters\n",
    "\n",
    "Overview: \n",
    "1. Compute $\\delta_{Z_2}$\n",
    "   * From $\\delta_{Z_2}$, compute $\\delta_{W_2}$\n",
    "2. Compute $\\delta_{Z_1}$ from $\\delta_{Z_2}$\n",
    "   * From $\\delta_{Z_1}$, compute $\\delta_{W_1}$\n",
    "  \n",
    "The forward and backward passes are very clear on the vectorized version of the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.04068113]]\n"
     ]
    }
   ],
   "source": [
    "d_Z2 = np.dot(grad(Z2), (y - t))\n",
    "p(d_Z2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.02767403]\n",
      " [ 0.02700164]]\n"
     ]
    }
   ],
   "source": [
    "d_W2 = np.dot(A1.T, d_Z2)\n",
    "p(d_W2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.01220434,  0.03661301])"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(d_Z2, W2.T).A1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 0.00265449,  0.00817165]])"
      ]
     },
     "execution_count": 237,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_Z1 = np.multiply(np.dot(d_Z2, W2.T), grad(Z1))\n",
    "d_Z1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.000929  0.00286 ]\n",
      " [ 0.002389  0.007354]]\n"
     ]
    }
   ],
   "source": [
    "d_W1 = np.dot(A0.T, d_Z1)\n",
    "p(d_W1, prec = 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.099071  0.39714 ]\n",
      " [ 0.797611  0.592646]]\n"
     ]
    }
   ],
   "source": [
    "p(W1 - d_W1, prec = 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch implementation\n",
    "\n",
    "Now we use PyTorch's autograd to verify our computation above. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tT = torch.tensor(t, requires_grad=False) # ground truth\n",
    "tx = torch.tensor(A0, requires_grad=False) # input x\n",
    "tW1= torch.tensor(W1, requires_grad=True) \n",
    "tW2 = torch.tensor(W2, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.00000e-02 *\n",
      "       1.8104, dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "igma = torch.sigmoid\n",
    "# our network (and its output Y for input X)\n",
    "Z1 = tx.mm(tW1)\n",
    "A1 = sigma(Z1)\n",
    "Z2 = A1.mm(tW2)\n",
    "A2 = sigma(Z2)\n",
    "Y = A2\n",
    "loss2 = (Y-tT).pow(2).sum()/2.0 # just sum all\n",
    "print(loss2.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.00000e-03 *\n",
      "       [[ 0.9291,  2.8601],\n",
      "        [ 2.3890,  7.3545]], dtype=torch.float64)\n",
      "tensor(1.00000e-02 *\n",
      "       [[ 2.7674],\n",
      "        [ 2.7002]], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "loss2.backward()\n",
    "print(tW1.grad)\n",
    "print(tW2.grad)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Modify the example to support biases in the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}