Homework1p2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HW1, COMS 4995_005, Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Max Possible Score 120 Points  (100 + 20 Extra Credits)\n",
    "----\n",
    "\n",
    "# Part1: (Basic Neural Nework) (70 Points)\n",
    "\n",
    "### Part 1.1:\n",
    "\n",
    "- Divide the training data into 80% training set and 20% validation set. \n",
    "- Implement the functions in the ipython notebook so that you can train your network. \n",
    "- Your code should take network structure, training data, hyperparameters and generate validation set accuracy.\n",
    "- Use Relu activation for intermediate layers and use cross entropy loss after taking softmax on the output of the final layer.\n",
    "\n",
    "If you get all the things mentioned above working - **60 Points**\n",
    "\n",
    "### Part 1.2:\n",
    "\n",
    "Test your model accuracy on test set. If it is more than **47%**, you will get an additional score of **10 points**\n",
    "\n",
    "# Part 2: (Regularization) (30 Points)\n",
    "\n",
    "### Part 2.1 (15 Points) :\n",
    "\n",
    "Modify code to add L2 regularization. Report the validation accuracy.\n",
    "\n",
    "You should get a validation and test accuracy of more than the one reported in Part-1\n",
    "\n",
    "### Part 2.2 (15 Points):\n",
    "\n",
    "You should get a validation and test accuracy crossing **50%**\n",
    "\n",
    "\n",
    "# Extra Credit (20 Points)\n",
    "\n",
    "Show your excitement on deep learning! Top **3 scorers** will get these **20 points**\n",
    "\n",
    "(Hints) Boost your accuracy by trying out: \n",
    "- Dropout Regularization\n",
    "- Batch Normalization\n",
    "- Other optimizers like Adam\n",
    "- Learning Rate Decay\n",
    "- Data Augmentation \n",
    "- Different Initializations for weights like Xaviers etc.\n",
    "\n",
    "\n",
    "------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Guidelines:\n",
    "\n",
    "1. Write your code in **Python 3**.\n",
    "2. **DONOT** import any other packages.\n",
    "3. Click **https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz** -> download **cifar-10-python.tar.gz** -> extract as **cifar-10-python**\n",
    "4. Ensure that **this ipython notebook** and **cifar-10-python** folder are in the same folder.\n",
    "\n",
    "\n",
    "-------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Submission Guidelines:\n",
    "\n",
    "1. Run this ipython notebook once and submit this file. Ensure that the outputs are printed properly. We will first see the outputs, if there are no outputs, we may not run the notebook at all.\n",
    "2. Training on the **test data** is considered cheating. If we find any clue of that happening, then we will disqualify the submission and it will be reported accordingly.\n",
    "3. Each team member needs to separately submit the the file named uni.ipynb on courseworks.\n",
    "------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Team Information\n",
    "\n",
    "Team Member1 (Name,UNI): \n",
    "\n",
    "Team Member2 (Name, UNI):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pickle\n",
    "import copy\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "# Do not import other packages\n",
    "# ys3031 Simon Sun | pz2232 Ping Zhu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class FullyConnectedNetwork(object):\n",
    "    \"\"\"\n",
    "    Abstraction of a Fully Connected Network.\n",
    "    Stores parameters, activations, cached values. \n",
    "    You can add more functions in this class, and also modify inputs and outputs of each function\n",
    "    \n",
    "    \"\"\"\n",
    "    def __init__(self, layer_dim, lambd=0):\n",
    "        \"\"\"\n",
    "        layer_dim: List containing layer dimensions. \n",
    "        \n",
    "        Code:\n",
    "        Initialize weight and biases for each layer\n",
    "        \"\"\"\n",
    "\n",
    "#         np.random.seed(1)\n",
    "        \n",
    "        self.W = []\n",
    "        self.b = []\n",
    "        i = 0\n",
    "        while i<(len(layer_dim)-1):\n",
    "            self.W.append((np.random.random((layer_dim[i+1],layer_dim[i]))-0.5)*0.02)\n",
    "            self.b.append(np.zeros((layer_dim[i+1],1)))\n",
    "            i = i+1\n",
    "        self.num_layers = len(layer_dim)-1\n",
    "        \n",
    "        self.lambd = lambd\n",
    "        \n",
    "    def feedforward(self,X):\n",
    "        \"\"\"\n",
    "        Expected Functionality: \n",
    "        Returns output of the neural network for input X. Also returns cache, which contains outputs of \n",
    "        intermediate layers which would be useful during backprop.\n",
    "        \"\"\"       \n",
    "        cache_dict = {\"d\":[None]*(len(self.W)), \"r\":[None]*(len(self.b))}\n",
    "        cache_dict[\"d\"][0] = X\n",
    "        cache_dict[\"r\"][0] = X\n",
    "        j = 1\n",
    "        while j<len(self.W):\n",
    "            cache_dict[\"r\"][j] = self.relu_forward(self.affineForward(cache_dict[\"d\"][j-1], self.W[j-1], self.b[j-1]))\n",
    "            cache_dict[\"d\"][j] = cache_dict[\"r\"][j]\n",
    "            j = j+1\n",
    "            \n",
    "        At = self.affineForward(cache_dict[\"d\"][j-1], self.W[j-1], self.b[j-1])\n",
    "        return At, cache_dict\n",
    "    \n",
    "    def loss_function(self, At, Y):\n",
    "        \"\"\"\n",
    "        At is the output of the last layer, returned by feedforward.\n",
    "        Y contains true labels for this batch.\n",
    "        this function takes softmax the last layer's output and calculates loss.\n",
    "        the gradient of loss with respect to the activations of the last layer are also returned by this function.\n",
    "        \"\"\"\n",
    "        exp_output = np.exp(At)\n",
    "        probabilities = exp_output / np.sum(exp_output, axis=0, keepdims=True)\n",
    "\n",
    "        log_prob = []\n",
    "        for j in range(At.shape[1]):\n",
    "            curr_prob = 0\n",
    "            for i in range(At.shape[0]):\n",
    "                if i == Y[0][j]:\n",
    "                    curr_prob += -np.log(probabilities[i][j])\n",
    "                else:\n",
    "                    curr_prob += -np.log(1-probabilities[i][j])\n",
    "            log_prob.append(curr_prob)\n",
    "        log_probabilities = np.array(log_prob)\n",
    "        \n",
    "        loss = np.sum(log_probabilities) / At.shape[1]\n",
    "        if self.lambd > 0:\n",
    "            # add regularization\n",
    "            i = 0\n",
    "            while i < len(self.W):\n",
    "                loss = loss + 0.5*self.lambd*np.sum(self.W[i]*self.W[i])\n",
    "                i = i + 1\n",
    "        \n",
    "        dAt = probabilities\n",
    "        dAt[Y,range(At.shape[1])] -= 1\n",
    "        dAt /= At.shape[1]\n",
    "        return loss, dAt\n",
    "    \n",
    "    def train(self, X, Y, max_iters=5000, batch_size=100, learning_rate=0.01, validate_every=200, \n",
    "              loss_until=0.4, accuracy_until=0.52):\n",
    "        \"\"\"\n",
    "        X: (3072 dimensions, 50000 examples) (Cifar train data)\n",
    "        Y: (1 dimension, 50000 examples)\n",
    "        lambd: the hyperparameter corresponding to L2 regularization\n",
    "        \n",
    "        Divide X, Y into train(80%) and val(20%), during training do evaluation on val set\n",
    "        after every validate_every iterations and in the end use the parameters corresponding to the best\n",
    "        val set to test on the Cifar test set. Print the accuracy that is calculated on the val set during \n",
    "        training. Also print the final test accuracy. Ensure that these printed values can be seen in the .ipynb file you\n",
    "        submit.\n",
    "        \n",
    "        Expected Functionality: \n",
    "        This function will call functions feedforward, backprop and update_params. \n",
    "        Also, evaluate on the validation set for tuning the hyperparameters.\n",
    "        \"\"\"\n",
    "        x_train, x_test = X[:, :40000], X[:, 40000:]\n",
    "        y_train, y_test = Y[:, :40000], Y[:, 40000:]\n",
    "        i = 0\n",
    "        loss = 200\n",
    "        loss_list = []\n",
    "        acc_list = []\n",
    "        iter_list = []\n",
    "        accuracy = -1\n",
    "        \n",
    "        print(\"traning started\")\n",
    "        while i < max_iters+1 and loss > loss_until and accuracy < accuracy_until:\n",
    "            x_batch,y_batch = self.get_batch(x_train,y_train,batch_size)\n",
    "            y_batch = y_batch.astype(int)\n",
    "            At,cache = self.feedforward(x_batch)\n",
    "            loss,dAt = self.loss_function(At,y_batch)\n",
    "            gradients = self.backprop(loss=loss,cache=cache,dAct=dAt,)\n",
    "            self.updateParameters(gradients,learning_rate)\n",
    "            if i % validate_every == 0:\n",
    "                accuracy = self.evaluate(x_test, y_test, print_every=False)\n",
    "                print(\"Step is:\", i, \"| Loss is:\", loss, \"| Accuracy is:\", accuracy)\n",
    "                \n",
    "                acc_list.append(accuracy)\n",
    "                iter_list.append(i)\n",
    "                if i % 5000 == 0 and i != 0:\n",
    "                    learning_rate /= 2\n",
    "            loss_list.append(loss)\n",
    "            i += 1\n",
    "           \n",
    "        print(\"training ended\")\n",
    "        print(\"----------------------------\")\n",
    "        print(\"plotting started\")\n",
    "        plt.plot(loss_list)\n",
    "        plt.xlabel('iterations')\n",
    "        plt.ylabel('loss over iterations')\n",
    "        plt.show()\n",
    "        \n",
    "        plt.plot(iter_list, acc_list, 'bo')\n",
    "        plt.xlabel('iterations')\n",
    "        plt.ylabel('accuracy over iterations')\n",
    "        plt.show()\n",
    "        \n",
    "        print(\"plotting ended\")\n",
    "        \n",
    "\n",
    "    def affineForward(self, A, W, b):\n",
    "        \"\"\"\n",
    "        Expected Functionality:\n",
    "        Forward pass for the affine layer.\n",
    "        :param A: input matrix, shape (L, S), where L is the number of hidden units in the previous layer and S is\n",
    "        the number of samples\n",
    "        :returns: the affine product WA + b, along with the cache required for the backward pass\n",
    "        \"\"\"\n",
    "        return np.dot(W, A)+b\n",
    "        \n",
    "    \n",
    "    def affineBackward(self, dA_prev, cache):\n",
    "        \"\"\"\n",
    "        Expected Functionality:\n",
    "        Backward pass for the affine layer.\n",
    "        dA_prev: gradient from the next layer.\n",
    "        cache: cache returned in affineForward\n",
    "        :returns dA: gradient on the input to this layer\n",
    "                 dW: gradient on the weights\n",
    "                 db: gradient on the bias\n",
    "        \"\"\"\n",
    "        dA = np.dot(self.W[self.num_layers].T, dA_prev)\n",
    "        dW = np.dot(cache[\"d\"][self.num_layers], dA_prev.T,) + self.lambd*self.W[self.num_layers].T\n",
    "        db = np.sum(dA_prev, axis=1, keepdims=True)\n",
    "        return dA, dW, db\n",
    "        \n",
    "    def relu_forward(self, X):\n",
    "        \"\"\"\n",
    "        Expected Functionality:\n",
    "        Forward pass of relu activation\n",
    "        \"\"\"\n",
    "        return np.maximum(0, X)\n",
    "        \n",
    "    def relu_backward(self, dx, cached_x):\n",
    "        \"\"\"\n",
    "        Expected Functionality:\n",
    "        backward pass for relu activation\n",
    "        \"\"\"\n",
    "        dx[cached_x[\"r\"][self.num_layers] <= 0] = 0\n",
    "        return dx\n",
    "    \n",
    "    def get_batch(self, X, Y, batch_size):\n",
    "        \"\"\"\n",
    "        Expected Functionality: \n",
    "        given the full training data (X, Y), return batches for each iteration of forward and backward prop.\n",
    "        \"\"\"\n",
    "        choices = np.random.choice(range(40000), batch_size, replace=False)\n",
    "        x_batch = np.zeros((X.shape[0], batch_size))\n",
    "        y_batch = np.zeros((Y.shape[0], batch_size))\n",
    "        for j, choice in enumerate(choices):\n",
    "            x_batch[:, j] = X[:, choice]\n",
    "            y_batch[:, j] = Y[:, choice]      \n",
    "        return x_batch, y_batch \n",
    "        \n",
    "    def backprop(self, loss, cache, dAct):\n",
    "        \"\"\"\n",
    "        Expected Functionality: \n",
    "        returns gradients for all parameters in the network.\n",
    "        dAct is the gradient of loss with respect to the output of final layer of the network.\n",
    "        \"\"\"\n",
    "        gradients = {\"d_weight\":[None]*(len(self.W)),\"d_bias\":[None]*(len(self.b))}\n",
    "        i = len(self.W)-1\n",
    "        self.num_layers = i\n",
    "        temp_dA, temp_dW, temp_db = self.affineBackward(dAct, cache)\n",
    "        \n",
    "        if self.lambd > 0:\n",
    "            temp_dW += self.lambd*self.W[i].T\n",
    "        \n",
    "        gradients[\"d_weight\"][i], gradients[\"d_bias\"][i] = temp_dW, temp_db\n",
    "        temp_dA_next = self.relu_backward(temp_dA, cache)\n",
    "        i -= 1\n",
    "        while i>=0:\n",
    "            self.num_layers = i\n",
    "            temp_dA, temp_dW, temp_db = self.affineBackward(temp_dA_next, cache)\n",
    "            \n",
    "            if self.lambd > 0:\n",
    "                temp_dW += self.lambd*self.W[i].T\n",
    "            \n",
    "            gradients[\"d_weight\"][i] = temp_dW\n",
    "            gradients[\"d_bias\"][i] = temp_db\n",
    "            temp_dA_next = temp_dA\n",
    "            temp_dA_next = self.relu_backward(temp_dA_next, cache)      \n",
    "            i -= 1\n",
    "            \n",
    "        return gradients\n",
    "    \n",
    "    def updateParameters(self, gradients, learning_rate):\n",
    "        \"\"\"\n",
    "        Expected Functionality:\n",
    "        use gradients returned by backprop to update the parameters.\n",
    "        \"\"\"\n",
    "        i=0\n",
    "        while i<len(self.W):\n",
    "            self.W[i] += -learning_rate*gradients[\"d_weight\"][i].T\n",
    "            self.b[i] += -learning_rate*gradients[\"d_bias\"][i]\n",
    "            i += 1\n",
    "    \n",
    "    def evaluate(self, X_test, Y_test, print_every=50):\n",
    "        '''\n",
    "        X: X_test (3072 dimensions, 10000 examples)\n",
    "        Y: Y_test (1 dimension, 10000 examples)\n",
    "        \n",
    "        Expected Functionality: \n",
    "        print accuracy on test set\n",
    "        '''\n",
    "        if print_every:\n",
    "            print(\"----------------------------\")\n",
    "            print(\"evaluation started\")\n",
    "            \n",
    "        correct = 0.0\n",
    "        Y_test = Y_test.astype(int)\n",
    "        i = 0\n",
    "        while i < len(X_test[0]):\n",
    "            curr_x = X_test[:, i:i+1]\n",
    "            y_predicted, _ = self.feedforward(curr_x)\n",
    "            if Y_test[0][i] == y_predicted.argmax():\n",
    "                correct += 1\n",
    "            i += 1\n",
    "#             if print_every and i % print_every == 0:\n",
    "#                 print(\"Step:\", i, \"Correctness:\", correct/i)\n",
    "        \n",
    "        total = i\n",
    "        return correct / total\n",
    "            \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Loader:\n",
    "    \n",
    "    def unpickle(self, file):\n",
    "        with open(file, 'rb') as fo:\n",
    "            dict = pickle.load(fo, encoding='bytes')\n",
    "        return dict\n",
    "    \n",
    "    def load_train_data(self):\n",
    "        '''\n",
    "        loads training data: 50,000 examples with 3072 features\n",
    "        '''\n",
    "        X_train = None\n",
    "        Y_train = None\n",
    "        for i in range(1, 6):\n",
    "            pickleFile = self.unpickle('./datasets/data_batch_{}'.format(i))\n",
    "            dataX = pickleFile[b'data']\n",
    "            dataY = pickleFile[b'labels']\n",
    "            if type(X_train) is np.ndarray:\n",
    "                X_train = np.concatenate((X_train, dataX))\n",
    "                Y_train = np.concatenate((Y_train, dataY))\n",
    "            else:\n",
    "                X_train = dataX\n",
    "                Y_train = dataY\n",
    "\n",
    "        Y_train = Y_train.reshape(Y_train.shape[0], 1)\n",
    "\n",
    "        return X_train.T, Y_train.T\n",
    "\n",
    "    def load_test_data(self):\n",
    "        '''\n",
    "        loads testing data: 10,000 examples with 3072 features\n",
    "        '''\n",
    "        X_test = None\n",
    "        Y_test = None\n",
    "        pickleFile = self.unpickle('./datasets/test_batch')\n",
    "        dataX = pickleFile[b'data']\n",
    "        dataY = pickleFile[b'labels']\n",
    "        if type(X_test) is np.ndarray:\n",
    "            X_test = np.concatenate((X_test, dataX))\n",
    "            Y_test = np.concatenate((Y_test, dataY))\n",
    "        else:\n",
    "            X_test = np.array(dataX)\n",
    "            Y_test = np.array(dataY)\n",
    "\n",
    "        Y_test = Y_test.reshape(Y_test.shape[0], 1)\n",
    "\n",
    "        return X_test.T, Y_test.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_Train: (3072, 50000) -> 50000 examples, 3072 features\n",
      "Y_Train: (1, 50000) -> 50000 examples, 1 features\n",
      "X_Test: (3072, 10000) -> 10000 examples, 3072 features\n",
      "Y_Test: (1, 10000) -> 10000 examples, 1 features\n",
      "(3072, 10000)\n"
     ]
    }
   ],
   "source": [
    "X_train,Y_train = Loader().load_train_data()\n",
    "X_test, Y_test = Loader().load_test_data()\n",
    "print(\"X_Train: {} -> {} examples, {} features\".format(X_train.shape, X_train.shape[1], X_train.shape[0]))\n",
    "print(\"Y_Train: {} -> {} examples, {} features\".format(Y_train.shape, Y_train.shape[1], Y_train.shape[0]))\n",
    "print(\"X_Test: {} -> {} examples, {} features\".format(X_test.shape, X_test.shape[1], X_test.shape[0]))\n",
    "print(\"Y_Test: {} -> {} examples, {} features\".format(Y_test.shape, Y_test.shape[1], Y_test.shape[0]))\n",
    "\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "traning started\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-215bc1644e12>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# 3072 is the input feature size, 10 is the number of outputs in the final layer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mFCN\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mFullyConnectedNetwork\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlayer_dimensions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mFCN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_iters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10000\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m500\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlearning_rate\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.001\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_every\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0;31m# lambd, the L2 regularization penalty hyperparamter will be 0 for this part\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0macc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mFCN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY_test\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# print accuracy on test set\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-7-d0a9f5102e6c>\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, X, Y, max_iters, batch_size, learning_rate, validate_every, loss_until, accuracy_until)\u001b[0m\n\u001b[1;32m    114\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdateParameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgradients\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mlearning_rate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    115\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mvalidate_every\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 116\u001b[0;31m                 \u001b[0maccuracy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprint_every\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    117\u001b[0m                 \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Step is:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"| Loss is:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"| Accuracy is:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccuracy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    118\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-7-d0a9f5102e6c>\u001b[0m in \u001b[0;36mevaluate\u001b[0;34m(self, X_test, Y_test, print_every)\u001b[0m\n\u001b[1;32m    254\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    255\u001b[0m             \u001b[0mcurr_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 256\u001b[0;31m             \u001b[0my_predicted\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeedforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcurr_x\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    257\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mY_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0my_predicted\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    258\u001b[0m                 \u001b[0mcorrect\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-7-d0a9f5102e6c>\u001b[0m in \u001b[0;36mfeedforward\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0mj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0mj\u001b[0m\u001b[0;34m<\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mW\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 40\u001b[0;31m             \u001b[0mcache_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"r\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrelu_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maffineForward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcache_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"d\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mW\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     41\u001b[0m             \u001b[0mcache_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"d\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcache_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"r\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m             \u001b[0mj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mj\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-7-d0a9f5102e6c>\u001b[0m in \u001b[0;36maffineForward\u001b[0;34m(self, A, W, b)\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;34m:\u001b[0m\u001b[0mreturns\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mthe\u001b[0m \u001b[0maffine\u001b[0m \u001b[0mproduct\u001b[0m \u001b[0mWA\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malong\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mcache\u001b[0m \u001b[0mrequired\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mbackward\u001b[0m \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    149\u001b[0m         \"\"\"\n\u001b[0;32m--> 150\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mW\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mA\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    151\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    152\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "layer_dimensions = [3072,1000,100, 10]  # including the input and output layers  \n",
    "# 3072 is the input feature size, 10 is the number of outputs in the final layer\n",
    "FCN = FullyConnectedNetwork(layer_dimensions)\n",
    "FCN.train(X_train, Y_train, max_iters=10000, batch_size=500, learning_rate=0.001, validate_every=100)\n",
    "# lambd, the L2 regularization penalty hyperparamter will be 0 for this part\n",
    "acc = FCN.evaluate(X_test, Y_test)  # print accuracy on test set\n",
    "print(\"The accuracy over test dataset is\", acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "traning started\n",
      "Step is: 0 | Loss is: 3.26164226005 | Accuracy is: 0.1089\n",
      "Step is: 100 | Loss is: 2.85296767498 | Accuracy is: 0.2858\n",
      "Step is: 200 | Loss is: 2.85756119733 | Accuracy is: 0.313\n",
      "Step is: 300 | Loss is: 2.67487967192 | Accuracy is: 0.3533\n",
      "Step is: 400 | Loss is: 2.59857483441 | Accuracy is: 0.3676\n",
      "Step is: 500 | Loss is: 2.61594060234 | Accuracy is: 0.3811\n",
      "Step is: 600 | Loss is: 2.46849615701 | Accuracy is: 0.3921\n",
      "Step is: 700 | Loss is: 2.50973141577 | Accuracy is: 0.3907\n",
      "Step is: 800 | Loss is: 2.45373150621 | Accuracy is: 0.4087\n",
      "Step is: 900 | Loss is: 2.53709273969 | Accuracy is: 0.3939\n",
      "Step is: 1000 | Loss is: 2.37487859216 | Accuracy is: 0.4234\n",
      "Step is: 1100 | Loss is: 2.52976383019 | Accuracy is: 0.4015\n",
      "Step is: 1200 | Loss is: 2.39672465518 | Accuracy is: 0.4425\n",
      "Step is: 1300 | Loss is: 2.44792376873 | Accuracy is: 0.4179\n",
      "Step is: 1400 | Loss is: 2.27280146066 | Accuracy is: 0.4447\n",
      "Step is: 1500 | Loss is: 2.32679975695 | Accuracy is: 0.452\n",
      "Step is: 1600 | Loss is: 2.34508097895 | Accuracy is: 0.4432\n",
      "Step is: 1700 | Loss is: 2.2350306631 | Accuracy is: 0.4484\n",
      "Step is: 1800 | Loss is: 2.25986502884 | Accuracy is: 0.4402\n",
      "Step is: 1900 | Loss is: 2.27061692855 | Accuracy is: 0.4385\n",
      "Step is: 2000 | Loss is: 2.33148663332 | Accuracy is: 0.4501\n",
      "Step is: 2100 | Loss is: 2.32730826898 | Accuracy is: 0.4482\n",
      "Step is: 2200 | Loss is: 2.25217070804 | Accuracy is: 0.4608\n",
      "Step is: 2300 | Loss is: 2.1754141085 | Accuracy is: 0.4663\n",
      "Step is: 2400 | Loss is: 2.18460039522 | Accuracy is: 0.4725\n",
      "Step is: 2500 | Loss is: 2.15066222472 | Accuracy is: 0.4696\n",
      "Step is: 2600 | Loss is: 2.16306627243 | Accuracy is: 0.4747\n",
      "Step is: 2700 | Loss is: 2.22569320133 | Accuracy is: 0.4661\n",
      "Step is: 2800 | Loss is: 2.03466691771 | Accuracy is: 0.4982\n",
      "Step is: 2900 | Loss is: 2.08202326411 | Accuracy is: 0.488\n",
      "Step is: 3000 | Loss is: 2.05916198787 | Accuracy is: 0.4711\n",
      "Step is: 3100 | Loss is: 2.11619983438 | Accuracy is: 0.4959\n",
      "Step is: 3200 | Loss is: 1.99153594023 | Accuracy is: 0.4979\n",
      "Step is: 3300 | Loss is: 2.01248069984 | Accuracy is: 0.4754\n",
      "Step is: 3400 | Loss is: 2.01638647566 | Accuracy is: 0.4929\n",
      "Step is: 3500 | Loss is: 1.98356144198 | Accuracy is: 0.5016\n",
      "Step is: 3600 | Loss is: 2.03425446823 | Accuracy is: 0.4865\n",
      "Step is: 3700 | Loss is: 2.02514704386 | Accuracy is: 0.4753\n",
      "Step is: 3800 | Loss is: 1.9290978343 | Accuracy is: 0.4997\n",
      "Step is: 3900 | Loss is: 1.85232867142 | Accuracy is: 0.4996\n",
      "Step is: 4000 | Loss is: 2.05160615518 | Accuracy is: 0.494\n",
      "Step is: 4100 | Loss is: 1.89615489861 | Accuracy is: 0.4974\n",
      "Step is: 4200 | Loss is: 1.94152163172 | Accuracy is: 0.5047\n",
      "Step is: 4300 | Loss is: 1.90017406806 | Accuracy is: 0.4982\n",
      "Step is: 4400 | Loss is: 1.93964027209 | Accuracy is: 0.5039\n",
      "Step is: 4500 | Loss is: 1.8696785404 | Accuracy is: 0.5092\n",
      "Step is: 4600 | Loss is: 1.80180725694 | Accuracy is: 0.5035\n",
      "Step is: 4700 | Loss is: 1.8314290435 | Accuracy is: 0.5206\n",
      "training ended\n",
      "----------------------------\n",
      "plotting started\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XecVOX1+PHP2QLL0stSpLgqXUMT\nFEVRERHBFn9GE6Pxa4mxF4wJmhhjjIoajT2KGluMHaOChSJgQZEiRXpHOkiRDrt7fn/cO7NT7sze\n2d2Z2d0579drX8zcNs8Mu/fM084jqooxxhgDkJXuAhhjjKk6LCgYY4wJsqBgjDEmyIKCMcaYIAsK\nxhhjgiwoGGOMCbKgYIwxJsiCgjHGmCALCsYYY4Jy0l2ARDVr1kwLCwvTXQxjjKlWZsyYsUVVC8o6\nrtoFhcLCQqZPn57uYhhjTLUiIqv8HGfNR8YYY4IsKBhjjAmyoGCMMSbIgoIxxpggCwrGGGOCLCgY\nY4wJsqBgjDEmKGOCQlFxCW9N+4HiElt+1BhjYql2k9fK67kvVvDAJwtB4ILebdNdHGOMqZIypqaw\nfsdeAPbsL0pzSYwxpurKmKDQuWUDAJZs2pXmkhhjTNWVMUHhyEOcoLDLagrGGBNTxgSFLq2coLB7\nf3GaS2KMMVVXxgSFWjnOWx2/YGOaS2KMMVVX0oKCiOSJyLciMltE5onI3R7HDBOR+SIyR0QmiMih\nySqPMcaYsiWzprAfGKCq3YEewGAR6RtxzHdAb1XtBrwDPJjE8hhjjClD0oKCOgJDfXLdH404ZqKq\n7nGffgO0SVZ5AM7ufkgyL2+MMdVeUvsURCRbRGYBm4Bxqjo1zuFXAB/HuM5VIjJdRKZv3ry53OUp\nUScmHSgqKfc1jDGmJktqUFDVYlXtgVMDOEZEjvI6TkQuBnoDD8W4zkhV7a2qvQsKylxiNKZe7RoD\n8OPu/eW+hjHG1GQpGX2kqtuBScDgyH0iMhD4E3C2qib1bt22ST4Am3daUDDGGC/JHH1UICKN3Md1\ngIHAwohjegLP4gSETckqS0BB/dqABQVjjIklmQnxWgEvi0g2TvB5S1VHi8jfgOmq+gFOc1E94G0R\nAVitqmcnq0AWFIwxJr6kBQVVnQP09Nj+l5DHA5P1+l6a1asFWFAwxphYMmZGM0DtnGwARn23Ns0l\nMcaYqimjgkLAii27010EY4ypkjIyKBhjjPFmQcEYY0xQxgaFomKb1WyMMZEyNijsOWjrKhhjTKSM\nCwq/O+lwAHbsOZjmkhhjTNWTcUFh3DxnkZ1/f7UizSUxxpiqJ+OCwgV92gLw9bIf01wSY4ypejIu\nKLRqmAfAwg0701wSY4ypejIuKJxxVKt0F8EYY6qsjAsKtXJK3/L+IhuBZIwxoTIuKIT63asz0l0E\nY4ypUjI6KExaVP6lPY0xpibK6KBgjDEmnAUFY4wxQclcjjNPRL4VkdkiMk9E7vY4praIvCkiS0Vk\nqogUJqs8xhhjypbMmsJ+YICqdgd6AINFpG/EMVcA21S1PfBP4IEklscYY0wZkhYU1LHLfZrr/mjE\nYecAL7uP3wFOFXex5qqgpET55Pv1lJREFtsYY2qmpPYpiEi2iMwCNgHjVHVqxCGtgR8AVLUI2AE0\nTWaZEvHGtB+4+j8zeWPaD+kuijHGpERSg4KqFqtqD6ANcIyIHBVxiFetIOpruYhcJSLTRWT65s0V\nH0b6zMVH+zpu0859AGz4aV+FX9MYY6qDlIw+UtXtwCRgcMSuNUBbABHJARoCWz3OH6mqvVW1d0FB\nQYXLM6hri7DnG3/ax0GPRXey3JYsVWs+MsZkhmSOPioQkUbu4zrAQGBhxGEfAJe6j88HPtMU3IGz\nskorKDv3HeTY+ybw1w/mRR/nHlZiQcEYkyGSWVNoBUwUkTnANJw+hdEi8jcROds95gWgqYgsBYYB\nw5NYHk9bdx8A4LWpq6P2Bfq8beVOY0ymyEnWhVV1DtDTY/tfQh7vA36RrDL4cdJDk9L58sYYU6XY\njOYQ4+dv9NxedQbJGmNMcmVsULi832FR2yYs9A4KxhiTKTI2KNTLi245mxJjiU6rKBhjMkXGBgUv\nq37cE/bchqIaYzJN0jqaq7qyvv2f8MBnbN9z0DnWqgrGmAyRsTWFK06M7lMIUFXWbNvLrv1FKSyR\nMcakX8YGhQZ5uTH3FUckwBPrVTDGZIiMDQrxWFJUY0ymyuigMLRbq6htz3+x3DMPEsDKLbspHD6G\nsfM2JLtoxhiTFhnb0QxQOyc6Jv59zAKKIpuP3Naj2Wu2A/DhnPUMOrJl0stnjDGpltE1hewYw4qe\n+3x5iktijDFVQ0YHhTq1sj23/+gmyQuwbmZjTKbI6KCQ5XcCgk1UMMZkiDKDgoj8QkTqu4//LCKj\nRKRX8ouWfLnZdrM3xphQfmoKd6rqThE5ATgdeBn4V3KLlRo3ntrBMzFeJAFe+moFz062vgZjTM3m\nJygUu/8OBf6lqu8DtZJXpNSpn5fLX87q6uvYv344n/nrfwIS62NYv2MvT01canmUjDHVgp+gsFZE\nngUuAD4Skdo+z6sx1u/YW+5zr/7PTB76dBHLNu+uxBIZY0xy+Lm5XwB8CgxW1e1AE+C2sk4SkbYi\nMlFEFojIPBG5yeOYhiLyoYjMdo+5LOF3UAkGdmked/9b09eEPf9g9jomLtzk69q73fxJVlMwxlQH\nZQYFVd0DvA/sFpF2QC6w0Me1i4BbVbUL0Be4TkQi22quA+aranfgZOBhEUl509Tzl/bhV8e0S+ic\nuz+c5+s4CwbGmOqkzBnNInIDcBewEQjkf1CgW7zzVHU9sN59vFNEFgCtgfmhhwH1RUSAesBWnGBS\n5fkezuqyUa3GmOrAT5qLm4BOquq9LJkPIlII9ASmRux6EvgAWAfUBy5U1ajEQyJyFXAVQLt2iX2j\n9y/Bb/QCj4xdxHm92lDYrK7nIb9/e7b1JRhjqhU/fQo/ADvK+wIiUg94F7hZVX+K2H06MAs4BOgB\nPCkiDSKvoaojVbW3qvYuKCgob1Eq1brte3n8s6Vc/tI0z/0/7trPOzNC+yKsqmCMqfr8BIXlwCQR\nuV1EhgV+/FxcRHJxAsJrqjrK45DLgFHqWAqsADr7LXxlKm/T//Itu/nXpGVR24/++/iw59Z8ZIyp\nDvwEhdXAOJy5CfVDfuJy+wleABao6iNxrn2qe3wLoBNOEEq56we0p06udy4kL/sOlrZyPfDJQn7Y\nWrq+87aI3Elg9QRjTPVQZp+Cqt4N4Ka6UFXd5fPa/YBLgLkiMsvddgfQzr3uM8A9wEsiMhfnvvlH\nVd2S2FuoHG0a57PgnsEcccdHUSuv+XHigxNZOWIoABc8+3VlF88YY1LCz+ijo4BXceYnICJbgN+o\natwxmar6JWV8QVbVdcAg36VNgWX3DaFw+JhynXvOk1/y/vUnsGRTdNwUEXbtL6Je7YxewsIYU8X5\naT4aCQxT1UNV9VDgVuC55Barepq9JnZ//KwftnHUXZ/aqm3GmCrNT1Coq6oTA09UdRLgPQbTxJys\n9uHs9QB8tTQtrWPGGOOLn7aM5SJyJ04TEsDFOKOEjIdYo5g+85kWwxhj0slPTeFyoAAYBbznPk5L\njqLqYO/B4rIPMsaYKsrP6KNtwI0pKEuVUSsniwNFUROrfTnriS8rpQwbf9qHKrRsmFcp16tMny3c\nSHZWFid1rBoTCY0xlSdmTUFEHnX//VBEPoj8SV0RU2/OXYM4qnXUxGpflm+pnLQWx943gb73T6iU\na1W2y1+azqX//jbdxTDGJEG8mkKgD+EfqShIVZKXm83oG04s99BUY4yprmLWFFR1hvuwh6pODv3B\nyVNU4314/QmVfs2Xv14FwKad+/jlyK/5cdf+Sn8NgANFJeVuAjPGZC4/Hc2Xemz7v0ouR5X0szYN\nk3btl6es5JvlW3n929VJuX73u8fS655xSbm2Mabmitl8JCK/Ai4CDovoQ6gPlDuNtnFkZznx+GBx\nchbhsVFQxpjyiNenMAVnkZxmwMMh23cCc5JZqKrk5oEdeHT8kkq/bm6WkwEkNM/Shh37eHv6D1w/\noH2lv54xxvgRMyio6ipgFXBc6opT9dTO8Z85NRFZblAocoPC2HkbeHjsYhZt3MlpR7YIO/bpSUvp\nd0QzurdtlJSyGGNMQJl9CiLSV0SmicguETkgIsUiErlYTo2Vm12a0+//9WpTKdfcuvsAu/c7q44q\nysotu7nq1Rks2rgTgI/mhudHevCTRZzz1Fdh28564kv6jfisUspjjDEBfjqanwR+BSwB6gBXAk8k\ns1BVyelHtgw+PrNbq0q5Zq97xvG0uzDPs5OXc/I/JoXtf3xC2c1Vc9fuYO32vZVSHmOMCfCVx1lV\nl4pItqoWAy+KyJQkl6vKaNskn5UjhvLjrv00rVc7rWWZs2Y7t4+ay9tXZ3SLnjEmifwEhT0iUguY\nJSIP4nQ+Z1yW1HQHBIC/j17AvHU/MSdOim5jjKkIP81Hl7jHXQ/sBtoC/6+sk0SkrYhMFJEFIjJP\nRG6KcdzJIjLLPWZyIoXPBB/PXR98rDid0ra0pzEmWeLWFEQkG7hXVS8G9gF3J3DtIuBWVZ3pLuU5\nQ0TGqer8kOs3Ap4GBqvqahFpnvhbqNmueW1m1DZn+WtjjKl8cWsKbh9Cgdt8lBBVXa+qM93HO4EF\nQOuIwy4CRqnqavc4W3QgjlhrNYSasnQLB4stvYUxpnz8NB+tBL4SkTtFZFjgJ5EXEZFCoCcwNWJX\nR6CxiEwSkRki8ptErpsOy+4bwrDTOgaft21SJ2Wv7ZWB9Z0Za1i+2VkT+rvV27jo+ak89OmilJXJ\nGFOz+AkK64DR7rH1Q358EZF6wLvAzaoaOb8hBzgaGAqcDtwpIh0jjkFErhKR6SIyffPmzX5fOimy\ns4TehzYG4JjCJpzV7ZCUvfbW3QcAuODZr4Pbfv/2bAY/9gUAP+5y9i/dtCvq3PU79sZcKnT55l1s\nSVJiPmNM9eJnkZ27AUSkrqomtFiAiOTiBITXVHWUxyFrgC3udXeLyOdAd2BxRBlGAiMBevfunZxk\nQQk4vn0zpt5xKi0a5HGgqISOLepz85uz0laeQDbUWF0NM1dv47ynp/Dg+d24oHfbqP0DHp5MXm4W\nC+85o9LKtH7HXhrVqUWdWsmZEW6MSQ4/M5qPE5H5OH0CiEh3EXnax3kCvAAsUNVHYhz2PnCiiOSI\nSD5wbOB1qroWDZwV0WrlZHFuz8iuktT7Yeue4OPQGoGqssSdKT195VYARs1cw71j5oedv+9g5fZD\nHHf/ZxVaiGfzzv3s2HuwEktkjPHDT/PRozhNOz8CqOpsoL+P8/rhDGcd4A45nSUiQ0TkahG52r3W\nAuATnAR73wLPq+r35XgfGW/Ntr2eNYUXvlwR1UE97K3ZPPfFiqSX6duVWykcPoZRM9ckfG6fe8dz\nfBVdec6YmszvjOYfIoZBlpmXWVW/xMeQelV9CHjITzlMbFe+PI1DmzpzCkNjwBOfLeX2MzqX65rb\n9xxg7LyNXNAnuskpEU9NXMp55cgbtfuApf82JtX81BR+EJHjARWRWiLye6pJE08qnX905STLK6/d\nB4qZv97pxw+tGYQ2JUmC095ueXMWf3h3Dos27PTcH6vjOuq4hF7VGJNOfoLC1cB1OHMM1uAsxXlt\nMgtVHf3jF93TXYSgyYtLR2j5vG+HeX/WWnbtL2LTTmdEUqxlPV+esrI8xQNg74FiLnlhanA4rTGm\navATFDqp6q9VtYWqNndnN3dJdsGqoy//eErc/UcUVK2UUa9+s4p568LzKH2/dgc3vTGL20fNDW6L\nNarpb6NLO6tVlRJ3bYioGoRHYPpiyWa+WLKF+z6ySqcxVYmfoOCVJjtjUmcnok3jfD65+UQ+vdm7\nH35g1xae25Np5/6imM03d/7ve4Y+/mXw+cRFm4LLeK7zkZY7ZNE4/vy/7zn8jo+itsfilaqjuET5\nbOFG381SxpjKFzMouENRb8VJczEs5OevgA0+j6FzywZ0auk9t++2QZ1SXBpHoE9gX1Ex13nkUgp4\neOyiYK+DqibU9PTa1NXBxyWRJ/rsynjui+Vc/tJ0xs7f6P+FjTGVKl5NoRZQD2eEUuhM5p+A85Nf\ntOrt+COaRm3Lyc7isV/2SHlZXnLb/t+ftY4xIVlXvQS+wSvhrT4PfbqQw28f4+v1iiOrCnGCS2j8\nCMy1CPRl1BQjPl5I4XB/n50x6RZvjebJwGQRecldr9kk4NUrjqW4ROn454/Dtp/TozU3vZG+2c/x\nCBLsP/hu9Xa6tGoQ3PfUxGW+r+OnhhG38lDDmo+emez/szMm3WIGBRF5VFVvBp4Ukai/UlU9O6kl\nq+ays4TsLOHMbq0YPWd9pa3vnEyKsmd/6dyABe4Q1zOf+DLWKQC88W1p05GqRjUf+b3FW0ZwY9Iv\n3uS1V91//5GKgtRUT17Uiycv8t731EW9uO6/sdv4U+37tT9x8QuRiWzDeXUCDw8ZqaTq0adgjKk2\n4jUfzXD/tdXQKlmW+BuhUxUVlVHw+et/om3jfN/Xm7BwE1t3H6BJ3dIlO6rpR2NMjeBnSKpJkurY\nXBJoUorlzCe+jFlTUNVgJ3Toe793zAKKikv4zzero855auLS8hfWGJMwCwppVB1bWb5e9mPY89D+\nhIBYQeG1qas54o6P2LRzX9h2VWVKyHUD60YAtmCQMSkWNyiISLaIWLK6StajbSPP7c9c3CvFJUnc\nlIigENqfEFAc2dHsPn9/1loAVmyOWJYjosb06PglFSylMaa8/KzRfLTYSvGV6sXLjuF/1/Wj16Hh\nwWHwUa1YOWJomkrlT2hepVhi1YDycp05j4FZ0wGJJupLlQNFJTwybjF7LVurySB+mo++A94XkUtE\n5LzAT7ILVpM1rJNLj7aNaNWwTpUPAuXhNSS1qLiE5W4NYff+4rA+BRH//SvFJcr8dd79Gj3+Npbh\n784JPn9s/BKmLNuSUNlDvTFtNY9PWGL9Gh4217AJhqaUn6DQBGeBnQHAWe7PmcksVKYZd0t/Xvy/\nPukuRqXxWiP64XGLWevmU6rIkNXHxi9myONfeAaG7XsO8sa0H4LP/zl+MRc9F3+IbTwzV20DYH+R\n1RRCjZ6zjj73jufbFVvTXRSTBH7WaL4sFQXJZB1a1KdDi9J8SY3zc9m2p/ouRXnJC+HLcG7ZuZ/v\n15ZmY1VgT0iTzDsz1vDFkrKbpQBmrXGus3HnPrrSoIyj49tfVMzfRy9g2GkdWb9jH8u37KJt43wu\nfmEqo645nv/NWleh6y/c8BNtQobnqqpnIsCq4kBRCbVyyv6eGAgG89ft4JjDmiS7WCbFygwKItIR\n+BfQQlWPEpFuwNmq+vcyzmsLvAK0BEqAkar6WIxj+wDfABeq6jsJvocap3F+rWodFCJFrqD25GdL\nWLwxvDax8afEmiP2HSjmn+MWc/2A9uRml28Q3Yez1/PqN6vYX1TMW9OdJUOH/KwlO/cVMW6B/6R8\nxSXKT3sP0jhkrkVJiTL40S/oe3jpTVO16g5DHj9/I1e+Mp3RN5zAUa0bprs4Jo38/DU9B9wOHARQ\n1TnAL32cVwTcqqpdgL7AdSLSNfIgEckGHgA+9Vvomu7VK4/lnnOPintMo/zcFJWm8kUGBD9+2neQ\nuWt2BLukH5uwhMcmLOGt6T/EPS+eQDPWyi17ovZ5tXBt33OAg8XRCw7dO2YBPe8Zx859pYE8cO3Q\nJpaqPAJ54qJNAHz3w/Y0l8Skm5+gkK+q30ZsKyrrJFVdr6oz3cc7cZbwbO1x6A3Au8AmH2XJCK0b\n1eGSvofGPeaOM6rXOkc79pa/5vPd6m2c+9RXnPVk6cS4hW468P0Ho2/Sizd6Lx8ay7cr47eNB5p8\nevxtHDe98V3U/jFznWamXftL/ywCASC0uagqp/8oT8mqclNYqhSXKP+atIw9B8q8JVYbfoLCFhE5\nAvf3RkTOB+LnX44gIoVAT2BqxPbWwM+BZ8o4/yoRmS4i0zdv9tf2XBNM/P3JsXcKaUnDXV5z1uwo\n+6AYfv70lODIpUiBb7ihBv3zc89Fgh4Zu4iHPl1Y7nIAfDR3Q9Q2ryG1gfu/eGyryvzc5qvD+0iV\nD2ev44FPFvLw2MXpLkql8RMUrgOeBTqLyFrgZpx1m30RkXo4NYGbVTVyyMijwB/d+RAxqepIVe2t\nqr0LCgr8vnS1d1izugzs0pw/Du4cvVOdNNyZ7oslWzyT9N3yZnR68sc/W8pTE5exJMGahF+hxVCi\n03lolW5ASpxVFErn3Ozen1k1hVWqOhAoADqr6gl+11cQkVycgPCaqo7yOKQ38IaIrMRZuOdpETnX\nX9Ezw/OX9uGak48gJ0u48dQOwe1+RonURF5NMF5J+uIl7jvtn58D8b8V+1kSdOvuA2FNRuHnx99W\nUqK+ljytimpacDPh/NxZVojISJzOYt89hO4s6BeABar6iNcxqnqYqhaqaiHwDnCtqv7P72tkkqX3\nDWHYaR05/UhnnedMDQpeN1uvzt/Q4FEZq555BYle94yj/4MTw7YVlyiFw8fwz/FOc0Ks2drPfL6M\n40d8xrLNiXe6R77ehc9+7XtIbyzWJFQxNenz83Nn6QSMx2lGWiEiT4rICT7O6wdcAgwQkVnuzxAR\nuVpEfDc/mXCBm0zkraZlgzzP43974mFMuPWkJJcqdbz++PYdLOGUf0wK21ZSztzkgT6D0NeRGK8L\nTm0htBmlqMQJUM9OXl56skvVCWA79x3kq6XOTOuK1ha27znA1BVbufH16A7w8rAmocTUxI/Lz+S1\nvcBbwFsi0hh4DJgMZJdx3pck8Jmp6v/5PTaTRVbdJ992MtlZwvn/+hqAXx/bjg9nr+OnfU6zRk52\nFnVy4/5XVSsrtkR3OG/dfSBqe+Q9vLxBAgDxNzpHia4ZhHU0o1z+0nS+WLKFfu2buvuj/0TuHTOf\nr5f/yOgbTiy7aCFraqdaTbwhGp+ps0XkJBF5GpgJ5AEXJLVUpkyBb3SHNq1Lm8b55Nd2bvw3ntqB\nUdceHzyue5tGNaoFeMNP+6K2eX27jRzttGBD+BiH79fuYLzPCWoL1++M28cQ+vKzIsb5h5atRJ2O\ncYjf3PDcFyv4fm38dSu8XrtiatJviakIPzOaVwCzcGoLt6mq99hAk1YvX3YMo+esp0WDPOrUcgLE\n7046nMFHteSHrdGTs2oSPzfGXfvCO4TLWnc61OTFm/3VFFTZtudA3P2RVmzZRedW9WlWr7bv8nhf\nu0KnB/nJWFuT2s8rS03qfPdTU+iuqj9X1dctIKRf/TxnJnPtnPAmobZN8rnm5CMAaJCXy8oRQ7m9\nmk1wKy8/k6iufGV6QteM/BPv8KePw56Hzo8Iff3IG2boTTZ0V+C4O9+fR78RnyVUtrDrS+B6Neem\nVB4/bN1D4fAxfLmk/Flxy6Mm9sH4CQoNROQ9EdkkIhtF5F0RaZP0khlPfzmrK7ef0ZmTOvqfr9G0\nXq2yD6rG/NwQd+5LbBx5vFFBxSXKZS9O81WW0JvGpEWlI4RCv1nuL4oePQXOqKnfvjK9jKar8D6F\n3fuLeOCThRyIcc2qrqREWV6OEVnT3Fnp78wof9oT4/ATFF4EPgAOwUlT8aG7zaRBg7xcfnfSEWRl\n+f+Kkl8rh+/vPj2JpUqvV7/xNW0mIe/HyZAaeZNeG2cEUej/UtSKcz6Mm7+RuH3kEb8Gj09Ywr8m\nLePNBHNClauikYSvyU9OXMqAhycnnKqkJn5jTxc/QaFAVV9U1SL35yWciWymGqnJzQsvfrUypa8X\n65NUJeoGHtq0FLlMqe/Xi3Heog07+U8gILqHBGodRR5zNwLen7WWv34wz3Ofn5trMn+TAt/4N+yI\nHlAQT7p/vdP9+pXJb+6ji931mrNF5GKcRXdMNVInNztjJ7xVtnhzFuJ1OP7vu7VlXsPL38csYJPH\nqKvTH/2chz5d5FzP/+W46Y1ZvDRlZZnH7dh7kNemrooZlJLx5TyYM6qcF091kr6qupRsRfi5S1yO\nMwR1A04ivPPdbaYaycnOYvHfz4h7zMhLjk5Raaq31TFGc/3+7dkeHc2ltu4uHZkUeZstKi6hcPgY\nz6U/X5qykvP+NSXuCnCBdBvlrRF6nXbHqLn86b3vU5pOu6KjeGpyjThVygwKqrpaVc9W1QJVba6q\n5/rNfWSqvg+u7xd8fErn5r7OqYnrSidi4COTPbfv3l8UdUvbGZIbKVaeJIDxC5zRTE/HWA96zba9\ndPrzJ2z0qDF48fr++uJXK6LmUcSzZZez8JFXevLyWrNtD1OXx25oKM0um9g3cOtTqDzWnpDh8muV\nDm3Ntr+sCivvN9Wr/zMDKLv549j7JrBmW+x5Jy9/Hfv72t0fzufcp74qV/lCVeTLeP8HJ3LhyG/K\nvHZFfxXv/3gBL3y5omIXSUBNqp9YUMhQH990Ik9d1IvAd8rDm9VNaERTpHq1y5wHmRF83zBjHBev\nNhEw8vPlZR5T3rb1wFklJRpMCx37NRK/vt9sIxX9evLs5OXcM3p+Ba/iQw38HmV/yRmmTeM6nH90\nG7q0akCXVg2C4/H9/K2OvORovnar/m2b1CE3K4vlbs6h7AoElJpi3Y59vtvEy1rtLZ59MW7WfpuW\n/Pjb6PnBVCHRN39/7/H7tTtYv2Mfp3VtwbcrtjLBR1qR4OdXzl+ndH1jr0ldGWXWFETkJhFpII4X\nRGSmiAxKReFM5fvyjwO4eWDH4PPA355Xs0dutvBeSB6lQUe25K6zjgTgiz8M4MHzuwX3RdYU/npW\n1HLcGSEVN4dYo02PvW9Cua8ZGcy81r4+UFSSUPPYmU98yW/dmeQXPPs1z/qo4ZS7TyHNX9lrUsur\nr9FH7oppg3DmJ1wGjEhqqUzKZLm/zTnZ0b8KhU3r0rNdY1/Xef23fcOeB9JxZJovUpBmQVHen7U2\n7jFj5qyncPgYVv0Ye8LcM5OXRW3zurkJzgS9jn/+mHtGL+D1bytn1nBRcQn7DhaHZbCtQV+4qy0/\nQSHwazIEeFFVZ1MjW9Iy06FNnZxJz/+mNwCvXnEMj1zQPaFr9GrXiHZN81l6b/whrwB5uTW7G+u9\n7+LfrCuFOnMN4gk0Ty1Y72QXfUqrAAAcj0lEQVRb9QoiIz5eGFzPIbICELaehEgw9cS/vyrtvJ20\naDPfrghvBtt3sJgdew76ehs97xlH5zs/ofOdn4S8cOA1fV0i5DT/4WTk58v4bvW2xF4gg/j5C50h\nImNxgsKnIlIfqJ6JVUwUEeGPgztT2KwuACd2KODIQxr6PNf5N/Dn6FXbiOS53rRJiNeSpLEEDo0V\nRIp99vx6Nc+Mm7+RC579OmzbuU99Rfe/jfV1zUA+qgMh7WHBta3LOLdw+Biue22mr9eJdN9HC/n5\n01PKdW6UGli18RMUrgCGA31UdQ+Qi9OEFJeItBWRiSKyQETmichNHsf8WkTmuD9TRCSxr6gmKRrl\nO00/fQ5rAsD4YScFaxLhYv/pihDWHxEQeg+68oTDKlTOTJXs+9D6HXvDRh75+da+e38RB4pKWLgh\nsZxFkQJB7K4YaThCjZm7Pvi4MvsUvlyyhcLhY1i/w/+qeDWp6cRPUDgOWKSq290UF38GdpRxDkAR\ncKuqdsFZ3/k6EYnsfVwBnKSq3YB7gJH+i26SpUWDPMYP689f3U7l9s3rMbBri6jjjihwahdXeNzc\nT+nUnJ7tGrNyxFDuOfeo4PbQVeAK6temfl5pB/W71xxHk7o1O6NrZfD77R6cADLszfhNTVAarLNE\nuPWt2QmX6ci7PuUXz5R++/7RnfiWqMA7W7hhJ8969HmkwmtTnbkeM1f5n+iXjEC9bvteXyO2Kpuf\noPAvYI/7Lf4PwCrglbJOUtX1qjrTfbwTWICTZTX0mCmqGmjc+wawlNxVRPvm9cvMldQovxYrRwzl\nzG6HRO1rHHJzv6TvocHHF/Qu/S8e8rNWYeeowp1nZsYaEBUxes76sg9ybd9zkFE++jkCo4r+880q\npiyLnnF88QtTy7zG7JDV7srq8/AT2O7/eGGZxySTVw1p3rodFA4f47ksbGUb+vgXXPFyYuuAVAY/\nQaFInd+Yc4DHVPUxoH4iLyIihUBPIN5v1hXAx3H2mxogtN8hJzv8r06Bgnp5KS5RzXbHe3Pj7v/n\nuMUUl5R2085eE90IsGRj4usbbN8bewU6gPs/WuC5PdEZ4ZGzu5M9JPi9mU6AHTd/Q9j2ZDQfbfPZ\nYV/Z/ASFnSJyO3AJMEZEsnH6FXwRkXrAu8DN7tBWr2NOwQkKf4yx/yoRmS4i0zdv3ux1iKmG6uRm\nh9W7Vf21X7dtUocHz+/Gd3eelrzCZYhR361lwoKNcTuvywosXspaY3rsfO9mkXj39IPFJdwzen5Y\nYsETHpjIgaKSlM0TqEnzEWLxExQuBPbjzFfYgNME9JCfi4tILk5AeE1VR8U4phvwPHCOqnpmylLV\nkaraW1V7FxTYUg7VUa92jaImuDXKr0WTkFXh/H5LLGxalwt6ty33+gQm3MFiZXE5agMVEXlz3bH3\nIOu27435TX/1j3vo8KePeeHLFVHpK/7wTuJ9IKHXnR2RJDBWGYqKS3ht6uq4xyRiw4597D0QP5VI\nOpSZ5kJVN4jIa0AfETkT+FZVy+xTECf5ygvAAlV9JMYx7YBRwCWqujixopuq6JvbT+WnfdHV3lHX\n9gt7/rPWzrDX1648lhMemAg4nZ0tGng3Hz3xq548On4xyzbvDv5BliTQ4WpiW711T3A+Q6pEfuE+\n9eFJbNl1gO5tG4Vt319UzP0fLQxL4VEU8f8+YcEmct1mychlVHfsPUhOllA3Rm6u/g85v3temX8j\ny/jqN6vYU4k38b73T6BPYWPevjp6lF46+UlzcQHwLfALnHUVporI+T6u3Q+nyWmAiMxyf4aIyNUi\ncrV7zF+ApsDT7v7U96qYStWyYR4dW8Tvcho/7CT++9tjAWjTOJ9j3aGvitK+eb3gcXeFpMo4q/sh\nwVnTF/RpC5R/JTMTbmWSO00Do3ni2bLLbRKK+D99c9oPvDRlJR9/v8HjrFKfzHP2z1sXHty63z2W\nY+4dX+brl5RomZ3foW38pXN0ws+ZuXobhcPHMGOVv9xW01ZWvUl0fpqP/oQzR+FSVf0NcAxwZ1kn\nqeqXqiqq2k1Ve7g/H6nqM6r6jHvMlaraOGS/12B4U8O0b14vLA1GsCnB/fs6zJ1I98s+7cLOa94g\nj5UjhnJ2d2e0U+N8G75aGRJdzzlRr3qk846VxTXytjzOo+/B68wrTzgcgHN7RI+E2+3j2/31r8/k\niDs+csvglOKaOJPjlmzcFVa2wNuZvMjp85y8OPnpTpLFT1DIUtVNIc9/9HmeMb4E8isV1K8NwH+u\nPJZHLuhOnVrZ3H5GZ4b8rKXneXm52awcMZQnL+pZqeXp3sbfjG7jj9eEtlj9tZGVP69cUlHrSQg0\nrOM0DzWs4z/n1tyQkVYfzY1fE4n09ow1wWR/UFruQPGrc3+0n9TZn4jIp8Dr7vMLgY+SVySTaW49\nrSNndTuEDm6zU+tGdTivlzOf4XcnHVHm+YOPbMkNA9qjCk/GWLns+COaeo6/97zeUa08h2Yab7e9\nnXhHb6z033PXlv25z1yd+PKgizfujGrWfGPaas9jY7ZKxmmufHvGGlo1LO0PC9Qcbn1rNv07NuOc\nHq1jnBlu38Fi8kImeAYUFZdQVKKe+yqbn+U4b8OZadwN6A6MVFXPoaPGlEdOdhZdD2lQofNvHdSJ\n35/eiXM8mg+AYG6nsG1N8z2PvbhvO8/tkbq0Kn+Za5K3Z6xJ+Jx1Oypv7YdADqV4InM0gf/hpQeL\nS9hzoIgtu+PPvXj8s6UhNQXh88WbeXfmmjIn8gV8t3obne/8hEmLNkXtu3DkN+GJA5PIVzOQqr6r\nqsNU9RZVfS/ZhTKmvB77ZU/PTKxeX/JeufxYz2v4/Tb24v/1SahsJn2KiqN/AfzmS7r039/S9S+f\n8t+p3jWLWFZtjb1sqpcZq5xO58mLN4cl+1PV4L5UiBkURGSniPzk8bNTRFI7fs2YBHgPIlGu6n94\n8NnUO06lXYyagt/24CzrWcsI8ZodD0QEmyc+c5ovI2shN73xXZmvE1i9UDU82V+qxfy1VtX6qtrA\n46e+qlq92VRZzTyS6pWUwPUD2gefx5oPAf7XN87OhOmt1cT6kHkMr369skLXSmSg853/+95zuxD+\n5eL9WevKvFYgKEQOjU31yGv7rmNqnDd/dxzDTusYtq1ElQbuMNhYy0kH1nqId6vPDcnXZOtSVx3P\nTnaW+nz561Xc+b532u1d+4t4ZOyi4HOvmD69AmtnhyrP94XAKoiR829SPRvHz+gjY6qVtk3yufHU\nDnRsUY+r/+O0zQb+sD65+UTP+Q1jb+lPxxb1uebkI8LSbUz700D63DueurWymXTbKRx3f+k6yFkW\nFKqNA0UlHHXXp2Ue986MNRwMWfTnJHfGc6L81jZDBYJC5Ez9RJMEVpTVFEyNNfio0tTcOe4NvHPL\nBp5NR6HDFUP/oAMTmerUyg7OowgIbT769Ob+dLP5DVVW6OpuAa94TKp7Y9oPTFpUmnRz1Y+JdRaH\n+iFiPsXHc9fHvcEHvmNEJidMdU3BgoLJCMPP8F4GtHPL+tweYx8QsmawhD4FwpuPOrWsT91aVvE2\npQJNWgHXvDYzOEkuNDh8ttCZGR1MnRFnvexUsKBgMkKjGCkxPrm5f9wJcoGavASfl/6F5kQ0Hx17\neJMKldHUHLFaj977bg2rf9wTdqO//KXpLN20i4kLnRrK+IjV1lKdrNCCgjFxBBeS9/gWF9nRfMOA\nDp7XuOlU7+2m5sqKERXGL9hE/4cmMjZikZ7Rc9YFk/pFLq5zzlNfJaeQMVhQMCYODdYUnD/yV684\nJrgvsjMx1mikwLKm9fNyeO/aqpUm2STHiDKWEg0MgAjYvLN8a1ongwUFU6M9emEPnrqoV7nPD1QM\nAvf7EzsUsPCewUy941Tf1zitawsADm2aT892jbkxgZrD/L+d7vtYU31VpSkvFhRMjXZuz9YM7daq\n7ANjaOpOhLtpYOmNPC83O+7kt1ArRwylY4v6PPeb3rx0mVPLuKTvoQA8/5uyM8X7TcVgTGWx4RLG\nxBFIz11RgdoCOCnCA9f8/LZT2HOwiMGPfhF1Tvc2DeN+g3ziVz254fWy0yeYqu8/3ySWVymZrKZg\nTCUae0t/Zt55mu/j2zXNp3NL76wxZa1g1yBk7YDASKhEmrWM8ZK0moKItAVeAVoCJTgptx+LOEaA\nx4AhwB7g/1Q19nJHxlRxgRt5o/xcjiks/xDVZy4+mpM7FcQ9pkeb0vWMl943pNyvZUyoZDYfFQG3\nqupMEakPzBCRcao6P+SYM4AO7s+xwL/cf41Jqz8M7sSJ7ePflOOZ9ZdBCR3/6IU9uPnN0rz7x7dv\nSl5uNvuLYi8l2TDf/ypjxviVtKCgquuB9e7jnSKyAGgNhAaFc4BX1Jne942INBKRVu65xqTNtSe3\nL/sg4J5zj+JwjwV8EnVuz9b8uPsA94x2/jwCXQnW0WxSLSV9CiJSCPQEpkbsag2Erhq+xt0Wef5V\nIjJdRKZv3rw5crcxaXNJ30Pp175ZpVzr8n6FUdtidTQHVpgb0Ll5/DQdxiQo6UFBROoB7wI3q2rk\nfG2vX/moTB+qOlJVe6tq74KC8lfpjanKRITmbtK9srJsHtrUqZ38+//6+FrHunPL8E7rIyuw/Kmp\n2ZIaFEQkFycgvKaqozwOWQO0DXneBih7NQpjaqg3f3ccfxrShXq1nZbdYDNSRIw4qWPsL0fd2zYK\ne/7fK4+ld2HjsG3PXHw0vdo5xzXOz2XF/UNirjPhVyKT8kzVlczRRwK8ACxQ1UdiHPYBcL2IvIHT\nwbzD+hNMJjusWV1+G7JsqFeNoax5E+9efRxz1u7gvKenAM6w1zuGdOGwZvWCfRZtm+Qz6tp+Yed5\nL2Pq36+Oacsvjm7DiQ+Wbw0CUzUks6bQD7gEGCAis9yfISJytYhc7R7zEbAcWAo8B1ybxPIYU+0E\nQkLtHP9/qjnZWfRq15jWjeoATv6m/Fo5XHHCYQBcf0r8TvQPrz+hXGWtlZ1F2ybe616b6iOZo4++\npIw10N1RR9clqwzGVHdZWcJtp3diYJcWvPfdWp6ZvKxC1/MzO7tpPSe1R4sGtdmzv5id+4t8XTtW\nQsDWjeqwdvte/4U0aWUzmo2p4q47pT2dWtZn+BmdE0q5MbBLc8DJzpqIsBarBPoZYg2fff/6fp7b\nTeJSsTSnBQVjaqg7z+zKN7efGnOBoUivXXksF/ZuG5UuPCByhvWQn7UMe56b4x0UmtWr7bndJK6i\n/T5+WFAwpobKyc6iZUN/2VwB+rVvxgPndwvmVDq7xyFhYeHS4wqDj1eOGMrTvz46+HzsLf3Jt+VI\nk85qCsaYlKtXO4c5fx3E8MGdw0Y/BR56TaGIlbzvzAqkLTfRVm3dk/TXsNBujInSIM+pLbxxVV/O\neMxJ650lwpy/DgpbanL8sP7UiVFDWHjPYHKz7XtnZVqycSdHFNRL6mtYUDDGxNSlVQMm3HoS93+0\ngGMOa0JebnbY/vbNY6f3Dj125YihFA4fA8C4W/qjwIGiEh4dv5jxCzYlpew1UXFJ8l/DgoIxJq4j\nCurx/KV9fB//yAXdWbppV8z9HUKamp6/tE8wWJiyFZUkPypYUDDGVKrzerXx3N6heT2WxAkWqfCz\n1g2Zu3ZHWstQEcUpGH5kDX7GmJR499rjmXDrSeU699Ob+3NYBVOUv3L5Mfzvun7cOMBfWvSq6GAK\n2o8sKBhjUqJBXq7vTtJfHdMu+Pi8Xq3p1LI+o645nuOPaApEZ331o1ZOFtlZwrBBnZjx54EJn18V\n7C+y5iNjTIZZft8QROCIgrq0a5LPgM7OzOzGdWtxapcWTFn2IyXlGK8fOpK2ab3a3HZ6Jx76dBEA\np3QqYOKiqr9Wy/6DVlMwxmSQn7VuSFaWICJceeLhDDqyJTkhw1oD6ZXK07QeOZHvF0eX9n28eNkx\n5SpvqsVbnrWyWFAwxlQJE249if9dFz9PUmCOxKEJZGNt0aA2X/zhlODCRAHNG/if7R1Y3yJUOtaP\nSEXzkQUFY0xarRwxlJUjhnJEQb2YmVYDAvPmWjXKY/ZfBsU99tjDmgDObOuKpPT++vYBngsQReaC\nSoW9B5JfU7A+BWNMtRFIu6EKDfNzmfangRSVlHDc/Z9FHfvm745j0qJN9GzXOGpfIhrn18KrCyO/\nljM5b2CXFhx5SAMem7CkQq/jR51a2WUfVEFWUzDGVBtt3IWDAsNTC+rXplXDOlHHBUYXndypOQ3d\nBH/llZebzcCuLaK2d27ZgAfP78bDv+gevbB8NZa0oCAi/xaRTSLyfYz9DUXkQxGZLSLzROSyZJXF\nGFMznNK5OW9c1ZfL+x0W97imlZSu++c9WwPwt3OO9Nx/Qe+2NMzPTUn2UoAjD2mQ9NdIZk3hJWBw\nnP3XAfNVtTtwMvCwiPhL/G6MyVh9D29KVoy+h1HXHs9jv+xRruvednonWkWMUGrhdkYHEvvVipHg\n7+zuh4Q9H9glumZRUZcedyiDj0p+1tmkBQVV/RzYGu8QoL44jYT13GP9rftnjDEhjj7U6Tfo1a4x\n5/RoXa5rXHdKe76+/dSwbadFNhvF6AfvEJE6/P7zfgZAs3q1OLyCM7FTLZ19Ck8CXYB1wFzgJlVN\nQQ5AY0xN89JlffjoxhMTPu+53/TmvWuP99y3csTQYLAJtA6VMTjK02e/P9lz+x8Gdwo+7tC8nmf6\njSZ1SxtPUtVvkc6gcDowCzgE6AE8KSKeDWYicpWITBeR6Zs3V/1Zh8aY1Kqfl0vXcrS3n9a1ha/R\nSYEZ1LHWoY7UtG4thvysJc9cfHTMY84PSRwoAsMGdYo6JjQIpajbIq1B4TJglDqWAiuAzl4HqupI\nVe2tqr0LClI/NtgYk9kC8yMa5efSulH0aKdQr/+2L1lZwtO/PprehU3C9oXe5Js3yOPTm/sDpTf8\n6PWsS0/QFNUV0jlPYTVwKvCFiLQAOgHL01geY4zxlF8rh7vPPpJTOjWnZcM8zxv0uFv607BObtyZ\n0l0PacD3a38KPg8Em8DVRt9wAn3vnxDcf2a3VhxRUJc7359XKe/Dj6QFBRF5HWdUUTMRWQPcBeQC\nqOozwD3ASyIyFycc/lFVtySrPMYY44eId1PNpccXxj0vsrM5VO2cLM8UFYF6QGBIa2iw6dG2EXee\n2ZX/Tl3lHhO/3JUlaUFBVX9Vxv51QPx56sYYk2Jz7hpU6Q01b1zVl58/PSXqxi4RXRQFIc1HjfNz\nnbQfgVnclVymWCzNhTHGhKifV7EZ0F4C8xxifdsPbA7NCBtI6VGOAU8VYmkujDEmyQKpNqJHSEV0\nKnjIcXunc8ozHrYcrKZgjDFJ1rZJPu9eczxHHtKAd2asCW6P7GgO1b65s0rdeb3asHTTLm5IUapu\nCwrGGJMCgYlwtwzsyAkdmpZ5/G2nO/MWauVk8eczuya1bKEsKBhjTArdNLD0G3+gWalPYfQEutwY\neZaSzYKCMcakSbN6tRk/rH/YIkBjbjyBaSvipY1LLgsKxhiTRu2bh89vOPKQhhx5SMM0lcZGHxlj\njAlhQcEYY0yQBQVjjDFBFhSMMcYEWVAwxhgTZEHBGGNMkAUFY4wxQRYUjDHGBImmauWGSiIim4FV\n5Ty9GZDpC/nYZ2CfAdhnkInv/1BVLXM942oXFCpCRKarau90lyOd7DOwzwDsM8j09x+PNR8ZY4wJ\nsqBgjDEmKNOCwsh0F6AKsM/APgOwzyDT339MGdWnYIwxJr5MqykYY4yJI2OCgogMFpFFIrJURIan\nuzyVSUT+LSKbROT7kG1NRGSciCxx/23sbhcRedz9HOaISK+Qcy51j18iIpem472Uh4i0FZGJIrJA\nROaJyE3u9kz6DPJE5FsRme1+Bne72w8Tkanu+3lTRGq522u7z5e6+wtDrnW7u32RiJyenndUPiKS\nLSLficho93lGvf9Koao1/gfIBpYBhwO1gNlA13SXqxLfX3+gF/B9yLYHgeHu4+HAA+7jIcDHgAB9\nganu9ibAcvffxu7jxul+bz7ffyugl/u4PrAY6Jphn4EA9dzHucBU9729BfzS3f4McI37+FrgGffx\nL4E33cdd3b+P2sBh7t9NdrrfXwKfwzDgv8Bo93lGvf/K+MmUmsIxwFJVXa6qB4A3gHPSXKZKo6qf\nA5Hr950DvOw+fhk4N2T7K+r4BmgkIq2A04FxqrpVVbcB44DByS99xanqelWd6T7eCSwAWpNZn4Gq\n6i73aa77o8AA4B13e+RnEPhs3gFOFRFxt7+hqvtVdQWwFOfvp8oTkTbAUOB597mQQe+/smRKUGgN\n/BDyfI27rSZroarrwblpAs3d7bE+ixrxGbnNAD1xviln1GfgNp3MAjbhBLRlwHZVLXIPCX0/wffq\n7t8BNKV6fwaPAn8AStznTcms918pMiUoiMe2TB12FeuzqPafkYjUA94FblbVn+Id6rGt2n8Gqlqs\nqj2ANjjfbrt4Heb+W6M+AxE5E9ikqjNCN3scWiPff2XKlKCwBmgb8rwNsC5NZUmVjW6TCO6/m9zt\nsT6Lav0ZiUguTkB4TVVHuZsz6jMIUNXtwCScPoVGIpLj7gp9P8H36u5viNMEWV0/g37A2SKyEqd5\neABOzSFT3n+lyZSgMA3o4I5EqIXTsfRBmsuUbB8AgdEzlwLvh2z/jTsCpy+ww21a+RQYJCKN3VE6\ng9xtVZ7bFvwCsEBVHwnZlUmfQYGINHIf1wEG4vStTATOdw+L/AwCn835wGfq9LR+APzSHZ1zGNAB\n+DY176L8VPV2VW2jqoU4f9+fqeqvyZD3X6nS3dOdqh+cESeLcdpZ/5Tu8lTye3sdWA8cxPmmcwVO\n++gEYIn7bxP3WAGecj+HuUDvkOtcjtOxthS4LN3vK4H3fwJOFX8OMMv9GZJhn0E34Dv3M/ge+Iu7\n/XCcm9pS4G2gtrs9z32+1N1/eMi1/uR+NouAM9L93srxWZxM6eijjHv/Ff2xGc3GGGOCMqX5yBhj\njA8WFIwxxgRZUDDGGBNkQcEYY0yQBQVjjDFBFhRMxhGRKe6/hSJyUSVf+w6v1zKmurAhqSZjicjJ\nwO9V9cwEzslW1eI4+3epar3KKJ8x6WA1BZNxRCSQTXQEcKKIzBKRW9yEcg+JyDR3nYXfucef7K7X\n8F+cyW6IyP9EZIa7dsFV7rYRQB33eq+FvpY7e/ohEfleROaKyIUh154kIu+IyEIRec2doY2IjBCR\n+W5Z/pHKz8hkrpyyDzGmxhpOSE3BvbnvUNU+IlIb+EpExrrHHgMcpU46ZYDLVXWrm1Jimoi8q6rD\nReR6dZLSRToP6AF0B5q553zu7usJHImTY+croJ+IzAd+DnRWVQ2ksDAm2aymYEypQTg5kWbhpN5u\nipP7BuDbkIAAcKOIzAa+wUmg1oH4TgBeVyeT6UZgMtAn5NprVLUEJ0VHIfATsA94XkTOA/ZU+N0Z\n44MFBWNKCXCDqvZwfw5T1UBNYXfwIKcvYiBwnKp2x8k5lOfj2rHsD3lcDOSok+P/GJzMr+cCnyT0\nTowpJwsKJpPtxFm+M+BT4Bo3DTci0lFE6nqc1xDYpqp7RKQzTorqgIOB8yN8Dlzo9lsU4CyhGjP7\nprs2RENV/Qi4GafpyZiksz4Fk8nmAEVuM9BLwGM4TTcz3c7ezZQu3xjqE+BqEZmDk0nzm5B9I4E5\nIjJTndTNAe8Bx+Gs/6vAH1R1gxtUvNQH3heRPJxaxi3le4vGJMaGpBpjjAmy5iNjjDFBFhSMMcYE\nWVAwxhgTZEHBGGNMkAUFY4wxQRYUjDHGBFlQMMYYE2RBwRhjTND/B9fEvYqFmIlSAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1113f20b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHM5JREFUeJzt3X20XXV95/H3J+HxCvIQ0o4D5N6E\nldHGjgq5IBRrlWEUUUE6qOBlBlSaoqIotRabLh+YsurIlLajzOhdKNrJrTw6NYsVQcqTqyrKDYRI\nQCDGBFNpQUBAAkKS7/yx9zk593Ie9jl377PPw+e11l737H32w2/vte7+nv37/fb3p4jAzMwMYF7Z\nBTAzs97hoGBmZlUOCmZmVuWgYGZmVQ4KZmZW5aBgZmZVDgpmZlbloGBmZlUOCmZmVrVb2QVo10EH\nHRRjY2NlF8PMrK+sXbv2lxGxsNV6fRcUxsbGmJ6eLrsYZmZ9RdKWLOu5+sjMzKocFMzMrMpBwczM\nqhwUzMysykHBzMyqHBTMzHrc1BSMjcG8ecnfqanijtV3XVLNzIbJ1BSsWAHbtiXzW7Yk8wATE/kf\nz08KZmY9bOXKXQGhYtu2ZHkRHBTMzHrYQw+1t3yuHBTMzHrYokXtLZ8rBwUzsx520UUwMjJz2chI\nsrwIDgpmZl3Ubk+iiQmYnITRUZCSv5OTxTQyg3sfmZl1Tac9iSYmigsCs/lJwcysiU7eEWi0TbOe\nRN18F6GpiChsAk4A7gc2AhfU+f4s4FFgXTqd3Wqfy5cvDzPrT6tWRYyORkjJ31Wryi5Rc6tWRYyM\nRMCuaWSkebmbbSPNXD57nXaO0y5gOrLct7Os1MkEzAd+CiwB9gDuBpbNWucs4Ivt7NdBwaw/dXKD\nbbW/RgEmr+AzOlr/Bj462tk2jb6bP7/947Qra1AosvroKGBjRGyKiOeBK4CTCzye2VDpVnVDXsfJ\n8yWsSt38li3J7bNSNz811fy7VvucfZ6t3hFod5tGPYl27Gh+nK7KEjk6mYBTgctq5v8rs54KSJ4U\nHgbWA9cAh7bar58UzLr3qzvP4zSqOpHa31cnv8ab/epudJ4LFjTeVyfbVI41+1p3UuZ20QPVR++s\nExS+MGudBcCe6edzgJsb7GsFMA1ML1q0KL+rZNan8ryJNLvx53mcPPfVLMA0+65R8GtUtgUL2r82\nzbZpJO8gX08vBIVjgBtq5j8JfLLJ+vOBJ1vt108KZt371d3Jcbrx1NHJk0Kzm3UngaSTbZopuhG+\nF4LCbsAmYDG7GppfOWudl9V8PgW4vdV+HRTMuveru93jtLrx53Xja3acTqp18m5Q7kWlB4WkDJwI\nPEDSC2lluuxC4KT0818BG9KAcQvwilb7dFAwa31TbOfG2+zm1u6v+27eKNvtfdTql30vVvnkqSeC\nQhGTg4JZot6Nr4ibWztBptNqrW68v9AqYPVilU+eHBTMhlCnv9TL7NffrV/c/fbLPm9Zg4KSdfvH\n+Ph4TE9Pl10Ms540b15yu5tNgp07iz/+7Nw+kPTDb5bAbWwseZdgttFR2Lw5//KtXJn0/1+0KHlv\noFs5hcomaW1EjLdaz7mPzAZIt3Pvz9ZJRs9uDiIzMZEEmp07k7/DEhDa4aBgNkC6nXu/nnZvvGUH\nMpvJQcGsB+SVSqKbuffzKnOrQNYz2UOHRZaGh16a3NBsva7dRtt+bADtxzQbww73PjLrvk5uYr3S\nt78d3Spzv70g1suyBgVXH5nlqJNBVLrV0Npp9tB2ypZ3mbvZCG0JBwWzDrR7g6/cgOvdkJs1tOZZ\nn55n6upuNQ67Ebr7HBTM2tTsF3ejm9X8+Y1vyI0aWk88Mb9f9pDvr+5u9XLqhd5UQydLHVMvTW5T\nsLJ1kiuo3vq16R+6kWM/7/11K8VDP6WS6GW4odls7tpNrNZom05uyHmmx66Uyz15hlfWoLBb2U8q\nZr1qdsqGSvXNgQfCY4+9eP1K1dHERP33Auqlf2hWDbJoUf30D53Wp1fKNKxpHiwbtymYNdCoYRba\nr+fu5KWyIurTnebBWnFQMGugUQPs44939tZwuzfkbr6dbFbhoGBd0Y+pCpp1h+zWL27/srduc1Cw\nwuX50tRcy9FOYHJ3SBtGDgpWuE5fmsrz6aJZYGp0HFff2DDyIDtWuE4GfulksJZmGg3ksmABPPts\nfscx61UeZMdKUe9Xd6tUBfW2yTMlAzRuNH7ssXyPY9bvHBSsoXarbxpV0Zx4YuO6+Ubb1PtVD50n\nQmu3b78TrtmwclCwujppHG70637NmsZ18422mT+//jGaPV0006jReMGC5scxGzZuU7C6OhlMvZO2\ng0bbQHLTrlfXD43bG6DxG7v1Bm1vti+3KdggydqmUHouo3Yn5z7qjk7y7nSS36dVcrl6idAabbNg\nQWe5fZxwzYYBGXMf+UnB6urkSaGTHkOdbNPs6aKeZmU2Gxa59T6SdJikPdPPb5D0EUn751FI612d\nvLjVSb/+TrZxo7FZcVo+KUhaB4wDY8ANwGrg5RFxYuGlq8NPCt1Trw6+F+rZGz1d7L13/eylflIw\ny/6kkCV19s6I2C7pFOBvI+ILku6aexGt1zVKAV22Rimgof301GY2U5ag8IKk04Ezgbeny3Yvrkhm\nrTULWL34dGPWL7K8p/Be4Bjgooj4maTFwKpii2V568cspZ1wVlGzuWn5pBAR9wIfqZn/GfC5Igtl\n+Wo0ghj4pmlmM2XpfXSspBslPSBpk6SfSdrUjcJZPvLOI2RmgytL9dFXgEuA1wFHkvREOrLIQlm+\nGnXJ7LSr5rBURZkNoywNzU9GxLcLL4kVJs8B4F0VZTbYsjwp3CLpYknHSDqiMhVeMstNniOIuSrK\nbLBleVJ4bfq39qWHAI7LvzhWhEb9+jv5ZZ93VZSZ9ZaWTwoR8cY6kwNCiTqp08+rq2arAXPMrL9l\n6X20n6RLJE2n019L2q8bhbMX62Scgzx5MHuzwZalTeGrwNPAu9LpKeDyIgtljZVdp+/B7M0GW5ag\ncFhEfDoiNqXTZ4ElWXYu6QRJ90vaKOmCJuudKikktR4AYsgVUaffbnWU3xo2G1xZgsKzkl5XmZF0\nLPBsq40kzQcuBd4CLANOl7Ssznr7krwx/cOshR5medfpl10dZWa9JUtQ+ABwqaTNkrYAXwTOybDd\nUcDG9OnieeAK4OQ66/134PPAcxnLPNTyrtMvuzrKzHpLlt5H6yLi1cCrgP8YEYdHxN0Z9n0w8POa\n+a3psipJhwOHRsR1bZR5qOVdp+8upmZWq2FQkHRG+vd8SecDZwNn18y3ojrLqiP6SJoH/A3wJy13\nJK2o9H569NFHMxx6sDWr02+3fcBdTM2sVrMnhZekf/etM+2TYd9bgUNr5g8BflEzvy/wu8CtkjYD\nRwOr6zU2R8RkRIxHxPjChQszHLp/5JlHqJP2AXcxNbMZIqLpBBybZVmddXYDNgGLgT2Au4FXNln/\nVmC81X6XL18eg2LVqoiRkYjkFp5MIyPJ8sr3o6MRUvK3sryR0dGZ+6pMo6Oty9HOccys/wDT0eL+\nGhGZ0lx8AZid66jestnBZrukc0nGdZ4PfDUiNki6MC3c6ixBa5C1auRtN/Fcp+0DvTrsppl1n5IA\nUucL6Rjg94CPktT9V7wUOCWSxueuGx8fj+np6TIOnbt585Lf8rNJjTObNhuEfmys/W3MbDhIWhsR\nLd8Fa9amsAdJ28FuzGxPeAo4NY9CDrtmjbyd/Op3+4CZzVXDoBARt0Xy9vLREfHZmumSiHiwi2Uc\nCPUalJvdxDvpFeQUFGY2V1leXtuWjqewRtLNlanwkg2QRr2CoPFNvNNf/U5BYWZzkaWheQq4Engb\nyZvMZwJ+WaANzRqUG9248xwDwcwsq4YNzdUVksaJ5ZLWR8Sr0mW3RcQfdKWEs/RjQ3OzBuWdO7tf\nHjMbPnk0NFe8kP59WNJb09QUh8ypdEPGbw2bWb/IEhT+Mh1U50+AjwOXAR8rtFQDxr2CzKxfNG1T\nSNNfL40kYd2TwBu7UqoB4/YBM+sXTZ8UImIHcFKXytL3muUxcq8gM+sHWXoffV/SF0l6ID1TWRgR\ndxZWqj5U6XbaTloKM7Nek6X30S11FkdEHFdMkZrr1d5HTjFhZr0sa++jlk8KEeF2hAw8WI2ZDYKW\nvY8k/bakr0j6djq/TNL7iy9af3G3UzMbBFm6pH6NJP31v0/nHyDJnGo13O3UzAZBlqBwUERcBeyE\nZJwEYEehpepDTkZnZoMgS1B4RtIC0vGVJR1N8s7C0GrU9dTdTs2s32Xpkno+sBo4TNL3gIXAOwst\nVQ9z11MzG2RZuqTuSVJd9HJAwP3AvIj4TfHFe7Gyu6S666mZ9aM8E+L9ICK2R8SGiLgnIl4AfjD3\nIvYndz01s0HWsPpI0r8DDgb2TjOjKv3qpcBIo+0GXaOxk9311MwGQbM2hTcDZ5Gkyb6kZvnTwJ8X\nWKaedtFFM9sUwF1PzWxwNAwKEfF14OuS/ktEXNvFMvU0Zzw1s0HWrProjIhYBYxJOn/29xFxSZ3N\nhsLEhIOAmQ2mZtVHL0n/7tONgpiZWfmaVR99Of372e4Vx8zMypSlS6qZmQ0JBwUzM6tqGhQkzZP0\nrm4VxszMytVqjOadwLldKouZmZUsS/XRjZI+LulQSQdWpsJLZmZmXZclS+r70r8fqlkWwJL8i2Nm\nZmXKMkbz4m4UxMzMypdljOYRSX8haTKdXyrpbcUXzczMui1Lm8LlwPPA76XzW4G/LKxEZmZWmixB\n4bCI+DzwAkBEPMuuNNoDq9GQm2ZmgyxLQ/PzkvZm1xjNhwGljLrWLR5y08yGVZYnhc8A1wOHSpoC\nbgI+UWShyrZy5czxEiCZX7mynPKYmXVLlt5H35G0FjiapNrovIj4ZeElK5GH3DSzYZWl99Fq4E3A\nrRFx3aAHBGg8tKaH3DSzQZel+uivgd8H7pV0taRTJe1VcLlKddFFyRCbtTzkppkNg5ZBISJui4gP\nkrzBPAm8C3gky84lnSDpfkkbJV1Q5/tzJP1Y0jpJ/yxpWbsnUISJCZichNFRkJK/k5NuZDazwaeI\naL1S0vvo7cC7gSOA6yLiwy22mQ88APxnkncb7gBOj4h7a9Z5aUQ8lX4+CfhgRJzQbL/j4+MxPT3d\nssxmZraLpLURMd5qvZYNzZKuBF5L0gPpUpK2hZ0ZynAUsDEiNqX7uQI4GagGhUpASL2EtNurmZmV\nI8t7CpcD74mIHW3u+2Dg5zXzW0mCywySPgScD+wBHFdvR5JWACsAFrm118ysMFkamm8CPiTpmnT6\nsKTdM2xX763nFz0JRMSlEXEY8GfAX9TbUURMRsR4RIwvXLgww6HNzKwTWYLC/wGWA/87nY5Il7Wy\nFTi0Zv4Q4BdN1r8CeEeG/ZqZWUGyVB8dGRGvrpm/WdLdGba7A1gqaTHwL8BpwHtqV5C0NCIeTGff\nCjyImZmVJsuTwo403xEAkpYALdsXImI7yVCeNwD3AVdFxAZJF6Y9jQDOlbRB0jqSdoUz2z6DOXLi\nOzOzXbI8KfwpcIukTSTtBKPAe7PsPCLWAGtmLftUzefzshc1f058Z2Y2U9b3FPYEXk4SFH4SEaVl\nSc3zPYWxsSQQzDY6Cps353IIM7OekNt7CgBpEFg/51L1GCe+MzObKUubwsBy4jszs5mGOig48Z2Z\n2UxZUmdfK+mtkgYugDjxnZnZTC0bmiUdT9Lb6GjgauBrEfGTLpStLifEMzNrX9aG5iyps/8pIiZI\n3mTeDNwo6fuS3psx3YWZmfWJTFVCkhYAZwFnA3cBf0cSJG4srGRmZtZ1WVJnfxN4BfB/gbdHxMPp\nV1dKcj2OmdkAyfKewhcj4uZ6X2SpnzIzs/6RpfrodyTtX5mRdICkDxZYJjMzK0mWoPBHEfGrykxE\nPAH8UXFFMjOzsmQJCvMkVQfMScde3qO4IpmZWVmytCncAFwl6UskI6edQzJes5mZDZgsQeHPgD8G\nPkCSJfU7wGVFFsrMzMrRMihExE6S4TezDMFpZmZ9LMt7CkuBvwKWAXtVlkfEkgLLZWZmJcjS0Hw5\nyVPCduCNwN+TvMhmZmYDJktQ2DsibiJJnrclIj4DHFdssczMrAxZGpqfS9NmPyjpXOBfgN8qtlhm\nZlaGLE8KHwVGgI8Ay4EzgDOLLJSZmZWj6ZNC+qLauyLiT4Ffk4yrYGZmA6rpk0JE7ACW177RbGZm\ngytLm8JdwLckXQ08U1kYEd8srFRmZlaKLEHhQOAxZvY4CsBBwcxswGR5o9ntCGZmQyLLG82XkzwZ\nzBAR7yukRGZmVpos1UfX1XzeCzgF+EUxxTEzszJlqT66tnZe0jeAfyqsRGZmVposL6/NthRYlHdB\nzMysfFnaFJ5mZpvCv5KMsWBmZgMmS/XRvt0oiJmZla9l9ZGkUyTtVzO/v6R3FFssMzMrQ5Y2hU9H\nxJOVmYj4FfDp4opkZmZlyRIU6q2TpSurmZn1mSxBYVrSJZIOk7RE0t8Aa4sumJmZdV+WoPBh4Hng\nSuAq4FngQ0UWyszMypGl99EzwAVdKIuZmZUsS++jGyXtXzN/gKQbii2WmZmVIUv10UFpjyMAIuIJ\nMo7RLOkESfdL2ijpRU8bks6XdK+k9ZJukjSavehmZpa3LEFhp6RqWov0xv2irKmzpUN5Xgq8BVgG\nnC5p2azV7gLGI+JVwDXA57MW3MzM8pela+lK4J8l3ZbOvx5YkWG7o4CNEbEJQNIVwMnAvZUVIuKW\nmvVvB87IUmgzMytGlobm6yUdARwNCPhYRPwyw74PBn5eM78VeG2T9d8PfLveF5JWkAaiRYuci8/M\nrChZs6TuAB4BngSWSXp9hm1UZ1ndaidJZwDjwMX1vo+IyYgYj4jxhQsXZiyymZm1K0uW1LOB84BD\ngHUkTww/YOaYzfVsBQ6tmT+EOoPzSDqepIrqDyLiN9mKbWZmRcjypHAecCSwJSLeCBwOPJphuzuA\npZIWS9oDOA1YXbuCpMOBLwMnRcQjbZXczMxylyUoPBcRzwFI2jMifgK8vNVGEbEdOBe4AbgPuCoi\nNki6UNJJ6WoXA/sAV0taJ2l1g92ZmVkXZOl9tDV9ee0fgRslPUHGMZojYg2wZtayT9V8Pr6NspqZ\nWcGy9D46Jf34GUm3APsB1xdaKjMzK0VbKbAj4rbWa5mZWb/K2iXVzMyGgIOCmZlVOSiYmVmVg4KZ\nmVU5KJiZWZWDgpmZVTkomJlZlYOCmZlVOSiYmVmVg4KZmVU5KJiZWZWDgpmZVTkomJlZlYOCmZlV\nOSiYmVmVg4KZmVU5KJiZWZWDgpmZVTkomJlZlYOCmZlVOSiYmVmVg4KZmVU5KJiZWZWDgpmZVTko\nmJlZlYOCmZlVOSiYmVmVg4KZmVU5KJiZWZWDgpmZVTkomJlZlYOCmZlVOSiYmVnVUASFqSkYG4N5\n85K/U1Nll8jMrDftVnYBijY1BStWwLZtyfyWLck8wMREeeUyM+tFA/+ksHLlroBQsW1bstzMzGYa\n+KDw0EPtLTczG2aFBgVJJ0i6X9JGSRfU+f71ku6UtF3SqUWUYdGi9pabmQ2zwoKCpPnApcBbgGXA\n6ZKWzVrtIeAs4B+KKsdFF8HIyMxlIyPJcjMzm6nIJ4WjgI0RsSkingeuAE6uXSEiNkfEemBnUYWY\nmIDJSRgdBSn5OznpRmYzs3qK7H10MPDzmvmtwGs72ZGkFcAKgEUd1PtMTDgImJllUeSTguosi052\nFBGTETEeEeMLFy6cY7HMzKyRIoPCVuDQmvlDgF8UeDwzM5ujIoPCHcBSSYsl7QGcBqwu8HhmZjZH\nhQWFiNgOnAvcANwHXBURGyRdKOkkAElHStoKvBP4sqQNRZXHzMxaKzTNRUSsAdbMWvapms93kFQr\nmZlZD1BER22/pZH0KLClw80PAn6ZY3H6ka+BrwH4Ggzj+Y9GRMueOn0XFOZC0nREjJddjjL5Gvga\ngK/BsJ9/MwOf+8jMzLJzUDAzs6phCwqTZRegB/ga+BqAr8Gwn39DQ9WmYGZmzQ3bk4KZmTUxNEGh\n1dgO/UzSVyU9IumemmUHSrpR0oPp3wPS5ZL0v9LrsF7SETXbnJmu/6CkM8s4l05IOlTSLZLuk7RB\n0nnp8mG6BntJ+pGku9Nr8Nl0+WJJP0zP58o0uwCS9kznN6bfj9Xs65Pp8vslvbmcM+qMpPmS7pJ0\nXTo/VOefi4gY+AmYD/wUWALsAdwNLCu7XDme3+uBI4B7apZ9Hrgg/XwB8D/SzycC3yZJWHg08MN0\n+YHApvTvAennA8o+t4zn/zLgiPTzvsADJGN4DNM1ELBP+nl34IfpuV0FnJYu/xLwgfTzB4EvpZ9P\nA65MPy9L/z/2BBan/zfzyz6/Nq7D+STjs1yXzg/V+ecxDcuTQsuxHfpZRHwXeHzW4pOBr6efvw68\no2b530fidmB/SS8D3gzcGBGPR8QTwI3ACcWXfu4i4uGIuDP9/DRJWpWDGa5rEBHx63R293QK4Djg\nmnT57GtQuTbXAP9JktLlV0TEbyLiZ8BGkv+fnifpEOCtwGXpvBii88/LsASFemM7HFxSWbrltyPi\nYUhumsBvpcsbXYuBuEZpNcDhJL+Uh+oapFUn64BHSALaT4FfRZKHDGaeT/Vc0++fBBbQ39fgb4FP\nsGvQrgUM1/nnYliCQm5jOwyARtei76+RpH2Aa4GPRsRTzVats6zvr0FE7IiI15DkEzsK+J16q6V/\nB+oaSHob8EhErK1dXGfVgTz/PA1LUBjGsR3+La0SIf37SLq80bXo62skaXeSgDAVEd9MFw/VNaiI\niF8Bt5K0KewvqZL4svZ8queafr8fSRVkv16DY4GTJG0mqR4+juTJYVjOPzfDEhSGcWyH1UCl98yZ\nwLdqlv+3tAfO0cCTadXKDcCbJB2Q9tJ5U7qs56V1wV8B7ouIS2q+GqZrsFDS/unnvYHjSdpWbgFO\nTVebfQ0q1+ZU4OZIWlpXA6elvXMWA0uBH3XnLDoXEZ+MiEMiYozk//vmiJhgSM4/V2W3dHdrIulx\n8gBJPevKssuT87l9A3gYeIHkl877SepHbwIeTP8emK4r4NL0OvwYGK/Zz/tIGtY2Au8t+7zaOP/X\nkTzirwfWpdOJQ3YNXgXclV6De4BPpcuXkNzUNgJXA3umy/dK5zem3y+p2dfK9NrcD7yl7HPr4Fq8\ngV29j4bu/Oc6+Y1mMzOrGpbqIzMzy8BBwczMqhwUzMysykHBzMyqHBTMzKzKQcGGjqTvp3/HJL0n\n533/eb1jmfULd0m1oSXpDcDHI+JtbWwzPyJ2NPn+1xGxTx7lMyuDnxRs6EiqZBP9HPD7ktZJ+lia\nUO5iSXek4yz8cbr+G9LxGv6B5GU3JP2jpLXp2AUr0mWfA/ZO9zdVe6z07emLJd0j6ceS3l2z71sl\nXSPpJ5Km0je0kfQ5SfemZfmf3bxGNrx2a72K2cC6gJonhfTm/mREHClpT+B7kr6TrnsU8LuRpFMG\neF9EPJ6mlLhD0rURcYGkcyNJSjfbHwKvAV4NHJRu8930u8OBV5Lk2PkecKyke4FTgFdERFRSWJgV\nzU8KZru8iSQn0jqS1NsLSHLfAPyoJiAAfETS3cDtJAnUltLc64BvRJLJ9N+A24Aja/a9NSJ2kqTo\nGAOeAp4DLpP0h8C2OZ+dWQYOCma7CPhwRLwmnRZHROVJ4ZnqSklbxPHAMRHxapKcQ3tl2Hcjv6n5\nvAPYLZIc/0eRZH59B3B9W2di1iEHBRtmT5MM31lxA/CBNA03kv6DpJfU2W4/4ImI2CbpFSQpqite\nqGw/y3eBd6ftFgtJhlBtmH0zHRtiv4hYA3yUpOrJrHBuU7Bhth7YnlYDfQ34O5KqmzvTxt5H2TV8\nY63rgXMkrSfJpHl7zXeTwHpJd0aSurni/wHHkIz/G8AnIuJf06BSz77AtyTtRfKU8bHOTtGsPe6S\namZmVa4+MjOzKgcFMzOrclAwM7MqBwUzM6tyUDAzsyoHBTMzq3JQMDOzKgcFMzOr+v+9thg6Xjd8\nFwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1114b7320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "plotting ended\n",
      "----------------------------\n",
      "evaluation started\n",
      "The accuracy over test dataset is 0.5199\n"
     ]
    }
   ],
   "source": [
    "layer_dimensions = [3072, 1000, 100, 10]  # including the input and output layers  \n",
    "# 3072 is the input feature size, 10 is the number of outputs in the final layer\n",
    "FCN = FullyConnectedNetwork(layer_dimensions, lambd=0.0001)\n",
    "FCN.train(X_train, Y_train, max_iters=10000, batch_size=500, learning_rate=0.001, validate_every=100)\n",
    "# lambd, the L2 regularization penalty hyperparamter will not be 0 for this part\n",
    "y_predicted = FCN.evaluate(X_test, Y_test)  # print accuracy on test set\n",
    "print(\"The accuracy over test dataset is\", y_predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Credit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}