DEEP_LEARNING_03_EMOTION_RECOGNITION_TENSORFLOW_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "ezRBSSehGFP_"
      },
      "cell_type": "markdown",
      "source": [
        "# Deep Learning Based Emotion Recognition with TensorFlow\n",
        "\n",
        "![alt txt](https://github.com/omarsar/nlp_pytorch_tensorflow_notebooks/blob/master/img/emotion_classifier.png?raw=true)\n",
        "\n",
        "In this notebook we are going to learn how to train deep neural networks, such as recurrent neural networks (RNNs), for addressing a natural language task known as **emotion recognition**. We will cover everything you need to know to get started with NLP using deep learning frameworks such as TensorFlow. We will cover the common best practices, functionalities, and steps you need to understand the basics of TensorFlow APIs to build powerful predictive models via the computation graph. In the process of building our models, we will compare PyTorch and TensorFlow to let the learner appreciate the strenghts of each tool.\n",
        "\n",
        "by [Elvis Saravia](https://twitter.com/omarsar0)"
      ]
    },
    {
      "metadata": {
        "id": "VvVYIrzbGFQB"
      },
      "cell_type": "markdown",
      "source": [
        "---"
      ]
    },
    {
      "metadata": {
        "id": "7cvGF8bLGFQB"
      },
      "cell_type": "markdown",
      "source": [
        "## Outline\n",
        "1. Deep Learning Frameworks\n",
        "     - 1.1 Eager execution\n",
        "     - 1.2 Computation graph\n",
        "2. Tensors\n",
        "    - 2.1 Basic math with tensors\n",
        "    - 2.2 Transforming tensors\n",
        "3. Data\n",
        "    - 3.1 Preprocessing data\n",
        "        - Tokenization and Sampling\n",
        "        - Constructing Vocabulary and Index-Word Mapping\n",
        "    - 3.2 Converting data into tensors\n",
        "    - 3.3 Padding data\n",
        "    - 3.4 Binarization\n",
        "    - 3.5 Split data\n",
        "    - 3.6 Data Loader\n",
        "4. Model\n",
        "    - 4.1 Pretesting Model\n",
        "    - 4.2 Testing models with eager execution\n",
        "5. Training\n",
        "6. Evaluation on Testing Dataset\n",
        "    - 6.1 Confusion matrix\n",
        "- Final Words\n",
        "- References\n",
        "- *Storing models and setting checkpoints (Exercise)*\n",
        "- *Restoring models (Exercise)*"
      ]
    },
    {
      "metadata": {
        "id": "MsIiRbOJGFQC"
      },
      "cell_type": "markdown",
      "source": [
        "---"
      ]
    },
    {
      "metadata": {
        "id": "cJU7w4AxGFQC"
      },
      "cell_type": "markdown",
      "source": [
        "## 1. Deep Learning Frameworks\n",
        "There are many deep learning frameworks such as Chainer, DyNet, MXNet, PyTorch, TensorFlow, and Keras. Each framework has their own strenghts which a researcher or a developer may want to consider before choosing the right framework. In my opinion, PyTorch is great for researchers and offers eager execution by default, but its high-level APIs require some understanding of deep learning concepts such as **affine layers** and **automatic differentiation**. On the other hand, TensorFlow was originally built as a low-level API that provides a robust list of functionalities to build deep learning models from the ground up. More recently, TensorFlow also offers **eager execution** and is equipped with a high-level API known as Keras.\n",
        "\n",
        "![alt txt](https://github.com/omarsar/nlp_pytorch_tensorflow_notebooks/blob/master/img/dl_frameworks.png?raw=true)"
      ]
    },
    {
      "metadata": {
        "id": "Qz9IJvFNGFQE"
      },
      "cell_type": "markdown",
      "source": [
        "### 1.1 Eager Execution\n",
        "Eager execution allows us to operate on the computation graph dynamically, also known as **imperative programming**. TensorFlow requires that you manually turn this mode on, while PyTorch comes with this mode by default. Below we import the necessary library and enable eager execution."
      ]
    },
    {
      "metadata": {
        "id": "sWY55-C6GFQE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "e450e647-5bb7-4dd7-f5e1-eeb42304706a"
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "tf.enable_eager_execution()\n",
        "print(tf.__version__)\n",
        "print(\"EE enabled?\", tf.executing_eagerly())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.11.0\n",
            "EE enabled? True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "k0492P_NGFQJ"
      },
      "cell_type": "markdown",
      "source": [
        "### 1.2 Computation Graph\n",
        "A simplified definition of a neural network is a string of functions that are **differentiable** and that we can combine together to get more complicated functions. An intuitive way to express this process is through computation graphs. \n",
        "\n",
        "![alt txt](http://colah.github.io/posts/2015-08-Backprop/img/tree-eval-derivs.png)\n",
        "\n",
        "Image credit: [Chris Olah](http://colah.github.io/posts/2015-08-Backprop/)"
      ]
    },
    {
      "metadata": {
        "id": "tF4FvowaGFQK"
      },
      "cell_type": "markdown",
      "source": [
        "## 2. Tensors\n",
        "Tensors are the fundamental data structure used to store data that will be fed as input to a computation graph for processing and applying tranformations. Let's create two tensors and multiply them, and then output the result. The figure below shows a 4-D Tensor.\n",
        "\n",
        "![alt txt](https://github.com/omarsar/nlp_pytorch_tensorflow_notebooks/blob/master/img/tensor.png?raw=true)"
      ]
    },
    {
      "metadata": {
        "id": "wGrEG46hGFQL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "95bbc890-5483-45cd-c7a3-37a9eea7045a"
      },
      "cell_type": "code",
      "source": [
        "c = tf.constant([[1.0, 2.0], [3.0, 4.0]])\n",
        "d = tf.constant([[1.0, 1.0], [0.0, 1.0]])\n",
        "e = tf.matmul(c, d)\n",
        "print(e)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[1. 3.]\n",
            " [3. 7.]], shape=(2, 2), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "up4dSTpvGFQO"
      },
      "cell_type": "markdown",
      "source": [
        "### 2.1 Math with Tensors\n",
        "TensorFlow and other deep learning libraries like PyTorch allow you to do **automatic differentation**. Let's try to compute the derivative of a function -- in this case that function is stored in the variable `z`. In TensorFlow, the `tf.GradienTape()` function allows tracking of operations on the input tensor. "
      ]
    },
    {
      "metadata": {
        "id": "qUB7B17NGFQO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "4ff02cee-533b-460a-8e46-da46d0dbbd72"
      },
      "cell_type": "code",
      "source": [
        "### Automatic differentiation with TensorFlow\n",
        "\n",
        "x = tf.contrib.eager.Variable(tf.ones((2,2)))\n",
        "with tf.GradientTape() as tape:\n",
        "    y = x + 2\n",
        "    z = y * y * 3\n",
        "    out = tf.reduce_mean(z)\n",
        "\n",
        "grad = tape.gradient(out, x) # d(out)/dx\n",
        "print(grad)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[4.5 4.5]\n",
            " [4.5 4.5]], shape=(2, 2), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "718enJ9CGFQS"
      },
      "cell_type": "markdown",
      "source": [
        "You can verfiy the output with the equations in the figure below:\n",
        "\n",
        "![alt txt](https://github.com/omarsar/nlp_pytorch_tensorflow_notebooks/blob/master/img/autograd.jpg?raw=true)"
      ]
    },
    {
      "metadata": {
        "id": "jjdpAq8WGFQS"
      },
      "cell_type": "markdown",
      "source": [
        "### 2.2 Transforming Tensors\n",
        "We can also apply some transformation to a tensor such as adding a dimension or transposing it. Let's try both adding a dimension and transposing a matrix below."
      ]
    },
    {
      "metadata": {
        "id": "7f6MDxZpGFQT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "cef5a1f0-eeeb-4e96-d59c-b665871687d4"
      },
      "cell_type": "code",
      "source": [
        "x = tf.constant([[1, 2, 3], [4, 5, 6]])\n",
        "print(\"X shape: \", x.shape)\n",
        "\n",
        "# add dimension\n",
        "print(tf.shape(tf.expand_dims(x, 1)))\n",
        "\n",
        "# transpose\n",
        "tf.transpose(x).shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X shape:  (2, 3)\n",
            "tf.Tensor([2 1 3], shape=(3,), dtype=int32)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([Dimension(3), Dimension(2)])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "metadata": {
        "id": "ce7oReKrGFQW"
      },
      "cell_type": "markdown",
      "source": [
        "---"
      ]
    },
    {
      "metadata": {
        "id": "LLETy3TeGFQX"
      },
      "cell_type": "markdown",
      "source": [
        "## 3. Emotion Dataset\n",
        "In this notebook we are working on an emotion classification task. We are using the public emotion dataset provided [here](https://github.com/huseinzol05/NLP-Dataset/tree/master/emotion-english). The dataset contains tweets labeled into 6 categories."
      ]
    },
    {
      "metadata": {
        "id": "c5d2qLckGFQY"
      },
      "cell_type": "code",
      "source": [
        "import re\n",
        "import numpy as np\n",
        "import time\n",
        "from sklearn import preprocessing\n",
        "from sklearn.model_selection import train_test_split\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "import itertools\n",
        "import pandas as pd\n",
        "from scipy import stats\n",
        "from sklearn import metrics\n",
        "from sklearn.preprocessing import LabelEncoder"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "chFsLeTrH7EF"
      },
      "cell_type": "code",
      "source": [
        "### Helper functions\n",
        "import pickle\n",
        "\n",
        "def convert_to_pickle(item, directory):\n",
        "    pickle.dump(item, open(directory,\"wb\"))\n",
        "\n",
        "\n",
        "def load_from_pickle(directory):\n",
        "    return pickle.load(open(directory,\"rb\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ycMPbd-oI31t",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "c09daf0e-fb6e-4c86-f086-f970d32e85e4"
      },
      "cell_type": "code",
      "source": [
        "### read data from your Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/gdrive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /gdrive; to attempt to forcibly remount, call drive.mount(\"/gdrive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "v21d9Q9kNsZQ"
      },
      "cell_type": "markdown",
      "source": [
        "We had already processed the data for you. You can find the pickle file [here](https://github.com/omarsar/nlp_pytorch_tensorflow_notebooks/blob/master/data/merged_training.pkl)."
      ]
    },
    {
      "metadata": {
        "id": "PdOajIM5GFQa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 310
        },
        "outputId": "2e69b258-bd47-4d53-a7e3-9138e2073f54"
      },
      "cell_type": "code",
      "source": [
        "# load data\n",
        "data = load_from_pickle(directory=\"/gdrive/My Drive/DAIR RESOURCES/TensorFlow/emotion_recognition/merged_training.pkl\")\n",
        "data.emotions.value_counts().plot.bar()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f118b1da7b8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAETCAYAAADZHBoWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHBNJREFUeJzt3XuYXXV97/F3yBBLQgwBxwaRQrT6\naSnnxFsOcpIcIyQqt1qJURq8EOypRKyEngehXlBoEY8XoEaaclMutqdYnlKhQkgjkUsqeVL0CBr5\nIKLgMVhGCGkwGHKZ88daWQzDTGays2ev2Tuf1/PMk71/+7fW/v4cnM9e6/dbe43p7e0lIiICYK+6\nC4iIiNEjoRAREZWEQkREVBIKERFRSShERESlq+4CdldPz8aWLp+aPHk869dvauVbtlQnj6+TxwYZ\nX7tr9fi6uyeOGag9Rwq7qKtrbN0ljKhOHl8njw0yvnY3WsaXUIiIiEpCISIiKgmFiIioJBQiIqKS\nUIiIiEpCISIiKgmFiIioDOviNUmHA98ALrb95T7tbwWW2R5TPj8ZWAxsBy63fZWkvYGrgUOAbcBC\n2w9LmgYsBXqB+2wvKvdxFjC/bD/P9i1NGWlERAxpyCMFSROAJcC3+rX/FvAXwGN9+p0LzAFmA2dK\n2h9YADxleyZwAXBhuYtLgDNszwAmSTpG0lTgJGAmcDxwkaTRcUVHRMQeYDhHCpuBY4Gz+7V/DLgU\n+Hz5/Ahgje0NAJJWATOAo4Fryz4rgK9IGgdMtb2mbL+ZIkwOBG61/SzQI+kR4DDg/gbGtlOnfvb2\nZu9yp75yzlEtfb+IiEYMGQq2twJbJVVtkl4NTLN9rqQdoTAF6Omz6eMUf+SrdtvbJfWWbesH6PvE\nIPsYNBQmTx4/ai4P35nu7ol1lzBs7VTrrurksUHG1+5Gw/ga/UK8i4GPDNFnwC9bGqR9V/o+T7t8\nQVZPz8a6SxiW7u6JbVPrrurksUHG1+5aPb7BAmiXVx9JOgj4PeDvJN0DHCjpDmAdxRHADgeVbVV7\nOek8hmIe4oCd9e3XHhERLbDLoWD7F7ZfafuNtt8IPGb7TcBqYLqk/STtSzGfcBewnGI1EcAJwErb\nW4AHJM0s208ElgG3A8dJGifpZRShsHZ3BhgREcM35OkjSa8HvggcCmyR9E7gRNtP9u1n+xlJ5wC3\n8dxy0g2SrgfmSrqbYtL6lHKTxcBlkvYCVtteUb7fFcCd5T4W2d6++8OMiIjhGNPb29J71DRdozfZ\nyeqjgXXyedtOHhtkfO2uhjmF3GQnIiJ2LqEQERGVhEJERFQSChERUUkoREREJaEQERGVhEJERFQS\nChERUUkoREREJaEQERGVhEJERFQSChERUUkoREREJaEQERGVhEJERFQSChERUUkoREREJaEQERGV\nhEJERFQSChERUekaTidJhwPfAC62/WVJBwNfBfYGtgDvsf1LSScDi4HtwOW2r5K0N3A1cAiwDVho\n+2FJ04ClQC9wn+1F5XudBcwv28+zfUvzhhsRETsz5JGCpAnAEuBbfZr/iuKP/puAG4E/L/udC8wB\nZgNnStofWAA8ZXsmcAFwYbmPS4AzbM8AJkk6RtJU4CRgJnA8cJGksbs/zIiIGI7hHClsBo4Fzu7T\n9iHgN+XjHuB1wBHAGtsbACStAmYARwPXln1XAF+RNA6YantN2X4zRZgcCNxq+1mgR9IjwGHA/Y0N\nb891+u0fben7XXrU51r6fhExMoY8UrC91fYz/dp+bXtb+Sn+dODvgSkUAbHD4xR/5Kt229spTgtN\nAdbvrG+/9oiIaIFhzSkMpAyE64DbbX9L0oJ+XcYMsulA7bvS93kmTx5PV9foP8PU3T2x7hJGVLuM\nr13qbFTG195Gw/gaDgWKieYf2z6vfL6O4pP+DgcB9/Rp/3456TwGeAw4oF/fdeWPBmgf1Pr1m3Zj\nCK3T07Ox7hJGVDuMr7t7YlvU2aiMr721enyDBVBDS1LLVUbP2v5Un+bVwHRJ+0nal2I+4S5gOcVq\nIoATgJW2twAPSJpZtp8ILANuB46TNE7SyyhCYW0jNUZExK4b8khB0uuBLwKHAlskvRN4KfAbSd8u\nu621/SFJ5wC38dxy0g2SrgfmSrqbYtL6lHKbxcBlkvYCVtteUb7fFcCd5T4WlfMQERHRAkOGgu17\nKZaYDsn2DcAN/dq2AQsH6LsWmDVA+xKKJbAREdFiuaI5IiIqCYWIiKgkFCIiopJQiIiISkIhIiIq\nCYWIiKgkFCIiopJQiIiISkIhIiIqCYWIiKgkFCIiopJQiIiISkIhIiIqCYWIiKgkFCIiopJQiIiI\nSkIhIiIqCYWIiKgkFCIiopJQiIiISkIhIiIqXcPpJOlw4BvAxba/LOlg4DpgLPAY8F7bmyWdDCwG\ntgOX275K0t7A1cAhwDZgoe2HJU0DlgK9wH22F5XvdRYwv2w/z/YtzRtuRETszJBHCpImAEuAb/Vp\nPh+41PYs4CHg1LLfucAcYDZwpqT9gQXAU7ZnAhcAF5b7uAQ4w/YMYJKkYyRNBU4CZgLHAxdJGrv7\nw4yIiOEYzumjzcCxwLo+bbOBm8rHN1MEwRHAGtsbbD8DrAJmAEcDN5Z9VwAzJI0Dptpe028fbwZu\ntf2s7R7gEeCwBscWERG7aMjTR7a3Alsl9W2eYHtz+fhx4EBgCtDTp88L2m1vl9Rbtq0foO8Tg+zj\n/sHqmzx5PF1do/9gort7Yt0ljKh2GV+71NmojK+9jYbxDWtOYQhjmtC+q/uorF+/aaguo0JPz8a6\nSxhR7TC+7u6JbVFnozK+9tbq8Q0WQI2uPnpa0j7l44MoTi2tozgCYLD2ctJ5DMXk9AE769uvPSIi\nWqDRUFgBzCsfzwOWAauB6ZL2k7QvxXzCXcByitVEACcAK21vAR6QNLNsP7Hcx+3AcZLGSXoZRSis\nbbDGiIjYRUOePpL0euCLwKHAFknvBE4Grpb0QYrJ4Gtsb5F0DnAbzy0n3SDpemCupLspJq1PKXe9\nGLhM0l7Aatsryve7Ariz3Mci29ubNtqIiNip4Uw030ux2qi/uQP0vQG4oV/bNmDhAH3XArMGaF9C\nsQQ2IiJaLFc0R0REJaEQERGVhEJERFQSChERUUkoREREJaEQERGVhEJERFQSChERUUkoREREJaEQ\nERGVhEJERFQSChERUUkoREREJaEQERGVhEJERFQSChERUUkoREREJaEQERGVhEJERFQSChERUelq\nZCNJ+wLXApOBFwHnAb8ElgK9wH22F5V9zwLml+3n2b5F0iTg74FJwNPAAttPSpoDfAbYBtxi+y93\nZ3AREbFrGj1SOAWw7TcD7wT+GrgEOMP2DGCSpGMkTQVOAmYCxwMXSRoLLAa+bXsm8E/A2eV+vwTM\nA2YAb5F0WIP1RUREAxoNhV8BB5SPJwNPAlNtrynbbgbmAG8GbrX9rO0e4BHgMOBo4Ma+fSW9AnjS\n9s9tbwduKftFRESLNHT6yPY/SDpF0kMUoXACcGmfLo8DBwJPAD0DtE/p0z5Q2472Vw5Vy+TJ4+nq\nGtvIMFqqu3ti3SWMqHYZX7vU2aiMr72NhvE1OqfwHuBR22+TNI3iU/+GPl3GDLLpQO270vcF1q/f\nNJxutevp2Vh3CSOqHcbX3T2xLepsVMbX3lo9vsECqNHTRzOA2wBsfx/YB3hJn9cPAtaVP1OGaB+q\nb0REtEijofAQcASApEOAjcCPJM0sXz8RWAbcDhwnaZykl1H8oV8LLKdYkQTFxPIy2z8DXizpUEld\nFBPTyxusLyIiGtDQ6SPgMuArku4o93EaxZLUyyTtBay2vQJA0hXAnRRLUhfZ3i7pS8DXJN0FPAW8\np9zvIuD/lI+vt/1gg/VFREQDGp1ofhp41wAvzRqg7xJgyQDb/9EAfe8EjmykpoiI2H25ojkiIioJ\nhYiIqCQUIiKiklCIiIhKQiEiIioJhYiIqCQUIiKiklCIiIhKQiEiIioJhYiIqCQUIiKiklCIiIhK\nQiEiIioJhYiIqCQUIiKiklCIiIhKQiEiIioJhYiIqCQUIiKiklCIiIhKQiEiIipdjW4o6WTgo8BW\n4FzgPuA6YCzwGPBe25vLfouB7cDltq+StDdwNXAIsA1YaPthSdOApUAvcJ/tRQ2PLCIidllDRwqS\nDgA+BcwEjgfeDpwPXGp7FvAQcKqkCRSBMQeYDZwpaX9gAfCU7ZnABcCF5a4vAc6wPQOYJOmYRgcW\nERG7rtHTR3OAFbY32n7M9p9S/NG/qXz95rLPEcAa2xtsPwOsAmYARwM3ln1XADMkjQOm2l7Tbx8R\nEdEijZ4+OhQYL+kmYDLwaWCC7c3l648DBwJTgJ4+272g3fZ2Sb1l2/oB+u7U5Mnj6eoa2+AwWqe7\ne2LdJYyodhlfu9TZqIyvvY2G8TUaCmOAA4B3UMwLrCzb+r4+2HbDbR+s7/OsX79pON1q19Ozse4S\nRlQ7jK+7e2Jb1NmojK+9tXp8gwVQo6eP/gP4N9tbbf8E2AhslLRP+fpBwLryZ0qf7V7QXk46j6GY\nnD5ggL4REdEijYbCcuAoSXuVk877UswNzCtfnwcsA1YD0yXtJ2lfivmEu8rt55d9TwBW2t4CPCBp\nZtl+YrmPiIhokYZOH9n+haQbgHvKpj8D1gDXSvog8Ahwje0tks4BbqNYZnqe7Q2SrgfmSrob2Ayc\nUu5nMXCZpL2A1bZXNDqw6GwP/skpu75Ng+/16iuvbnDLiPbT8HUKti8DLuvXPHeAfjcAN/Rr2wYs\nHKDvWmBWozVFRMTuyRXNERFRSShEREQloRAREZWEQkREVBIKERFRSShEREQloRAREZWEQkREVBIK\nERFRSShEREQloRAREZWEQkREVBr+QryIGBlLP/vtlr7fonNmt/T9YnTLkUJERFQSChERUUkoRERE\nJaEQERGVhEJERFQSChERUUkoREREZbeuU5C0D/AD4C+BbwHXAWOBx4D32t4s6WRgMbAduNz2VZL2\nBq4GDgG2AQttPyxpGrAU6AXus71od+qLiIhds7tHCp8Aniwfnw9cansW8BBwqqQJwLnAHGA2cKak\n/YEFwFO2ZwIXABeW+7gEOMP2DGCSpGN2s76IiNgFDYeCpN8DDgO+WTbNBm4qH99MEQRHAGtsb7D9\nDLAKmAEcDdxY9l0BzJA0Dphqe02/fURERIvszumjLwIfBt5fPp9ge3P5+HHgQGAK0NNnmxe0294u\nqbdsWz9A352aPHk8XV1jd2MYrdHdPbHuEkZUq8f3YAvfK7+70aOdam3EaBhfQ6Eg6X3Ad2z/VNJA\nXcYMsumutA/W93nWr980nG616+nZWHcJI6qTx9fJY4P2GV9398S2qbURrR7fYAHU6JHCccArJB0P\nvBzYDDwtaZ/yNNFBwLryZ0qf7Q4C7unT/v1y0nkMxeT0Af36rmuwvoiIaEBDcwq23217uu03AldS\nrD5aAcwru8wDlgGrgemS9pO0L8V8wl3AcmB+2fcEYKXtLcADkmaW7SeW+4iIiBZp5nUKnwLeL+ku\nYH/gmvKo4RzgNorQOM/2BuB6YKyku4HTgb8o97EYuFDSKuAntlc0sb6IiBjCbt9Pwfan+zydO8Dr\nNwA39GvbBiwcoO9aYNbu1hQREY3JFc0REVFJKERERCW344yIlnr0e+c3tl2D7/c7rz23wS33TDlS\niIiISkIhIiIqCYWIiKgkFCIiopJQiIiISkIhIiIqCYWIiKgkFCIiopJQiIiISkIhIiIqCYWIiKgk\nFCIiopJQiIiISkIhIiIqCYWIiKgkFCIiopJQiIiISsN3XpP0OWBWuY8LgTXAdcBY4DHgvbY3SzoZ\nWAxsBy63fZWkvYGrgUOAbcBC2w9LmgYsBXqB+2wvanhkERGxyxo6UpD0ZuBw20cCbwMuAc4HLrU9\nC3gIOFXSBOBcYA4wGzhT0v7AAuAp2zOBCyhChXI/Z9ieAUySdEzDI4uIiF3W6OmjO4H55eOngAkU\nf/RvKttupgiCI4A1tjfYfgZYBcwAjgZuLPuuAGZIGgdMtb2m3z4iIqJFGjp9ZHsb8Ovy6QeAW4C3\n2t5ctj0OHAhMAXr6bPqCdtvbJfWWbesH6LtTkyePp6trbCPDaKnu7ol1lzCiWj2+B1v4XvndNdej\nLX239vr9jYZaG55TAJD0dopQeAvw4z4vjRlkk11pH6zv86xfv2k43WrX07Ox7hJGVCePr5PHBhnf\naNHdPbGltQ4WQA2vPpL0VuDjwDG2NwBPS9qnfPkgYF35M6XPZi9oLyedx1BMTh8wQN+IiGiRRiea\nJwGfB463/WTZvAKYVz6eBywDVgPTJe0naV+K+YS7gOU8NydxArDS9hbgAUkzy/YTy31ERESLNHr6\n6N3AS4CvS9rR9n7gSkkfBB4BrrG9RdI5wG0Uy0zPs71B0vXAXEl3A5uBU8p9LAYuk7QXsNr2igbr\ni4iIBjQ60Xw5cPkAL80doO8NwA392rYBCwfou5bi2oeIiKhBrmiOiIhKQiEiIiq7tSQ1IiKe72Nr\nfjx0pyb6zPRXNXV/OVKIiIhKQiEiIioJhYiIqCQUIiKiklCIiIhKQiEiIioJhYiIqCQUIiKiklCI\niIhKQiEiIioJhYiIqCQUIiKiklCIiIhKQiEiIioJhYiIqCQUIiKiklCIiIhKQiEiIiqj8nacki4G\n3gj0AmfYXlNzSRERe4RRd6Qg6U3Aq2wfCXwA+FLNJUVE7DFGXSgARwP/DGD7R8BkSS+ut6SIiD3D\nmN7e3rpreB5JlwPftP2N8vldwAdsP1hvZRERnW80Hin0N6buAiIi9hSjMRTWAVP6PH8Z8FhNtURE\n7FFGYygsB94JIOl1wDrbG+stKSJizzDq5hQAJH0W+B/AduB029+vuaSIiD3CqAyFiIiox2g8fRQR\nETVJKERERCWhMARJXygnvCMiOl5CYWjfBc6WtFrSJyW9ou6CYvgkvbbuGkaKpAPrriE6Tyaah0nS\n3sBRwPkUq6L+FrjWdlv/DyjpNcBLbS+X9Eng9cDnba+qubSmkHQ78BbbW+uupdkk3WH7TXXXMZIk\nvRw4F5hse76kk4Dv2H6k5tJ222gd26j8ltTRRtIbgT8G3gTcCVwPzC3/fVeNpTXDpcDJkuYCrwFO\nB64B5tRaVfP8GvixpO8Dz+5otN3uvzeAxyStAtbw/LF9tL6Smu5K4K+Bc8rnjwNXA2+uq6AmGpVj\ny+mjIUgy8OfAvwJvsP0R26tsfxoYX2txzbHZ9s+AdwBLbf+Czvrv4gvAKcDFFAG446cT3ApcDnwP\n+GH541orar6xtm+lODrH9u10zn+fo3JstRfQBv4bRZJ3AcdKOnjHC7aPr62q5nlW0hUUFwuulPQ2\nYO+aa2qmVRRflTLd9h3AE8C/1VtSc9i+Bvh34Kflzzpgca1FNd8WSUcBYyX9tqTTgGfqLqpJRuXY\nEgpD+1Pg6xSHdMcCN0laVG9JTfUu4BZgju1tFKch3lNvSU11BcVpsfnl89nAtbVV00SS/hb4G+Af\ngbMoTvtdVWtRzfcBYAHwEuA2it/lwlorap6+Y1vGKBlbQmFofwQcYfsM26cB0+msP5qvAH5t+5fl\nRPNHgJfXXFMzHWz7bGATgO0vUxw5dII/KCeaf2T7BOAI4LCaa2q2/wVcafsw26+xfZrtTvmCzMeB\nj9s+DDiJIhjW11tSQmE4xlCe8yttp7hNaKe4FHiw30TzefWW1FTjJO1H+TuT9PvAi+otqWm6dtyA\nSlK37Z8D02quqdnuA86S9CNJl0iaWXdBTfR3wJGSDqU42vsDiqO9WiUUhvYPwL9LulTS3wD3Uvwy\nO0WnTzR/HLgdmC7pAeCfKD59doIlFKf/lgD3S/o5xWRzx7B9re15FB9Y/hU4TdKjNZfVLL9t+58p\njhKW2L4A2L/mmnKdwmAkfdD2ZZI+D0wFXkvxafN7FJN6W4EV5YqBtiVpGfBz4EiKT5lzKQ5pZ9Va\nWJNJeilFAG6ou5aRUF5HM9H2k3XX0mzl0d0J5U8vcJPtL9Rb1e6TtJpiZeNlFHNdO/6mvKHOunKd\nwuB+Vv77g/Ln5n6vj6P4Zb6qhTWNhHdR3Bf7k7a3SdpCB82ZSHq433OAbcBPgI/Z/m4ddTWDpMOB\niyjC4EhJ75N0ZzuPqb9ySfijFEd47+qg+QSATwIfBT5r+1eSPgF8qeaaEgqDsX1b+e+g5/gk3d+6\nikbMJmAfiiD4AvAfFEsbO8UVwFPATRSfMo8FuoGVFP8HbOdz1EuAD1GsQILiBlWX095j6u9IYD/g\nv1KcAvxeOXfStiS9yPZm4O7yB0njKQK+dp107rjlbN9Tdw1NcAXFaaOOW7JZOsb2Utu/sL3O9pXA\n3A753W21/aMdT2yv5fmLIjrBB+i8JeFfLf/9Ic+difhBn+e1ypFCHGx7oaSVUCzZlDR/qI3ayG8k\nXUxxEdt2iiXF48rVVk/XWtnue0rSqcAESUdQLBZ4vOaamm3HkvBtAJK6gDuApbVWtRtsLygfftj2\nN2stZgA5UohOXrIJxf2+f0LxSXMO0AO8neLrIN5dY10Nk7Tjk+ZG4EDgVxRX3T8FvL+uukZIJy8J\nP738/96okiOF2LFk81Xlks1e4E/qLamptlFMVP5nn7bjbLfzKbLfl/Rd4JXAg33aXw6cSPHVLJ1i\nx5LweygC4kiKeZNO8GLg55J+QvFNAmOAXtu1/v4SCns423cBr+vgJZsrKJYQ/6JPW7t/0pxJcVX2\nRXTONRfPUy4F3/F7+inwNp5bEj61rrqa7OS6CxhIQmEPJ2kh8GfAJGBMuWQT251yM6Fn+5zD7Qjl\nvSEepTg11qn6Trj+kBcuCe8E+wGfAl5NEXhrKe7XUquEQpxFMUH5/+ouZITcLOk44C6Ki4MAsL2p\nvpJiKDtbCt5Bvkpxk53vUJw6+u/A1ygulK1NQiEetN1p38Hf1weBsQO0d8qRULSvJ2z/S5/nN0n6\nn7VVU0ooRI+k71B8Wun7SbpT7t61ADgbOKB8Pg6YUl85EZUHyu9TW0GxEnQWsE7SsQC2b6mjqIRC\nVFdVdqgvAR8DPktx9e87gE64cC3a377lvyf0a59PMceQUIjWkfS+8mG7r8QZyibbKyU9a/te4N7y\nSwD/ZagNI0bYj21/pu4i+kso7Ln+S/nvK4Dfpbjidy9gBnA/nfNVF5sk/SHwU0mfobiQ7XdqrikC\noLu8sn4NxXUKQP2LIBIKeyjbZwFI+ibw+nKZ446vYP56nbU12QKKOYQPU9y/eBrwvp1uEdEax1F8\njUdfvdS8CCKhEAdTXKPwRPl8Hzrn4iBsb6T4OggYBWvAI3aw/eq6axhIQiE+B3xX0n9SfEp5MfDp\nWiuK2ANI+ikvnNPbZrvWe7QkFPZwtr8GfE3SARQX0DxBTq9EtMLhfR7vTbEkVTXVUsntOPdwkt7A\nAOv4bf9ufVVF7Jkk3W77qDpryJFCLKFYx/+/gUVkHX9ES/T70j8ovgZ9Yk3lVHI/hdhkeyXFN6Te\na/sTFCt1ImJkbaQ4XbvjjmvTgFNrrYgcKUTW8UfU5WjgDOC3gM9QHKl/AXhrnUXlSCFOB56kODr4\nDcUN0mv/Uq6IPcBW2/8XmAdcYnsVo+CDekIhrqO4V/FhwGyKC9fOrbOgiD1El6SPA38ILJc0nee+\nD6k2CYUYlZ9WIvYA7wE2ASfa/g3Flcyn1VtSlqTu8STdASynuDZhGsV3In3Z9hG1FhYRtciRQozK\nTysRUY8cKURERCVHChERUUkoREREJaEQERGVhEJERFT+P7CGuIKbO2vcAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f11a0fcb550>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "UyAsI5Y5GFQd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "outputId": "ae4cf854-8b38-4b0e-8d78-f3370ba69999"
      },
      "cell_type": "code",
      "source": [
        "data.head(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>emotions</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>27383</th>\n",
              "      <td>i feel awful about it too because it s my job ...</td>\n",
              "      <td>sadness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>110083</th>\n",
              "      <td>im alone i feel awful</td>\n",
              "      <td>sadness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>140764</th>\n",
              "      <td>ive probably mentioned this before but i reall...</td>\n",
              "      <td>joy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100071</th>\n",
              "      <td>i was feeling a little low few days back</td>\n",
              "      <td>sadness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2837</th>\n",
              "      <td>i beleive that i am much more sensitive to oth...</td>\n",
              "      <td>love</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18231</th>\n",
              "      <td>i find myself frustrated with christians becau...</td>\n",
              "      <td>love</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10714</th>\n",
              "      <td>i am one of those people who feels like going ...</td>\n",
              "      <td>joy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35177</th>\n",
              "      <td>i feel especially pleased about this as this h...</td>\n",
              "      <td>joy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>122177</th>\n",
              "      <td>i was struggling with these awful feelings and...</td>\n",
              "      <td>joy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26723</th>\n",
              "      <td>i feel so enraged but helpless at the same time</td>\n",
              "      <td>anger</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                     text emotions\n",
              "27383   i feel awful about it too because it s my job ...  sadness\n",
              "110083                              im alone i feel awful  sadness\n",
              "140764  ive probably mentioned this before but i reall...      joy\n",
              "100071           i was feeling a little low few days back  sadness\n",
              "2837    i beleive that i am much more sensitive to oth...     love\n",
              "18231   i find myself frustrated with christians becau...     love\n",
              "10714   i am one of those people who feels like going ...      joy\n",
              "35177   i feel especially pleased about this as this h...      joy\n",
              "122177  i was struggling with these awful feelings and...      joy\n",
              "26723     i feel so enraged but helpless at the same time    anger"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "bZun5kK8GFQh"
      },
      "cell_type": "markdown",
      "source": [
        "### 3.1 Preprocessing Data\n",
        "In the next steps we are going to create tokenize the text, create index mapping for words, and also construct a vocabulary. "
      ]
    },
    {
      "metadata": {
        "id": "o7pp0WwkGFQj"
      },
      "cell_type": "markdown",
      "source": [
        "#### Tokenization and Sampling"
      ]
    },
    {
      "metadata": {
        "id": "eM_CVQCeGFQk"
      },
      "cell_type": "code",
      "source": [
        "# retain only text that contain less that 70 tokens to avoid too much padding\n",
        "data[\"token_size\"] = data[\"text\"].apply(lambda x: len(x.split(' ')))\n",
        "data = data.loc[data['token_size'] < 70].copy()\n",
        "\n",
        "# sampling\n",
        "data = data.sample(n=50000);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OI0wz10WGFQn"
      },
      "cell_type": "markdown",
      "source": [
        "#### Constructing Vocabulary and Index-Word Mapping"
      ]
    },
    {
      "metadata": {
        "id": "rm6T8QOKGFQo"
      },
      "cell_type": "code",
      "source": [
        "# This class creates a word -> index mapping (e.g,. \"dad\" -> 5) and vice-versa \n",
        "# (e.g., 5 -> \"dad\") for the dataset\n",
        "class ConstructVocab():\n",
        "    def __init__(self, sentences):\n",
        "        self.sentences = sentences\n",
        "        self.word2idx = {}\n",
        "        self.idx2word = {}\n",
        "        self.vocab = set()\n",
        "        self.create_index()\n",
        "        \n",
        "    def create_index(self):\n",
        "        for s in self.sentences:\n",
        "            # update with individual tokens\n",
        "            self.vocab.update(s.split(' '))\n",
        "            \n",
        "        # sort the vocab\n",
        "        self.vocab = sorted(self.vocab)\n",
        "\n",
        "        # add a padding token with index 0\n",
        "        self.word2idx['<pad>'] = 0\n",
        "        \n",
        "        # word to index mapping\n",
        "        for index, word in enumerate(self.vocab):\n",
        "            self.word2idx[word] = index + 1 # +1 because of pad token\n",
        "        \n",
        "        # index to word mapping\n",
        "        for word, index in self.word2idx.items():\n",
        "            self.idx2word[index] = word  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Y7jG2-EWGFQq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217
        },
        "outputId": "d51890db-c706-4837-cd76-9b74d8589cab"
      },
      "cell_type": "code",
      "source": [
        "# construct vocab and indexing\n",
        "inputs = ConstructVocab(data[\"text\"].values.tolist())\n",
        "\n",
        "# examples of what is in the vocab\n",
        "inputs.vocab[0:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['a',\n",
              " 'aaa',\n",
              " 'aaaaaand',\n",
              " 'aaaah',\n",
              " 'aaand',\n",
              " 'aaargh',\n",
              " 'aabsolutely',\n",
              " 'aahed',\n",
              " 'aand',\n",
              " 'aardvark']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "metadata": {
        "id": "wBs3pqLHGFQs"
      },
      "cell_type": "markdown",
      "source": [
        "### 3.2 Converting Data into Tensors \n",
        "For convenience we would like to convert the data into tensors. "
      ]
    },
    {
      "metadata": {
        "id": "KaUZ6m_NGFQt"
      },
      "cell_type": "code",
      "source": [
        "# vectorize to tensor\n",
        "input_tensor = [[inputs.word2idx[s] for s in es.split(' ')]  for es in data[\"text\"].values.tolist()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bJ_Rex25GFQx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 485
        },
        "outputId": "6270aa7f-3554-460f-cde7-c7039333ef14"
      },
      "cell_type": "code",
      "source": [
        "# examples of what is in the input tensors\n",
        "input_tensor[0:2]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[11616,\n",
              "  8726,\n",
              "  23997,\n",
              "  8037,\n",
              "  9463,\n",
              "  9497,\n",
              "  17034,\n",
              "  11716,\n",
              "  18637,\n",
              "  26054,\n",
              "  27128,\n",
              "  22289],\n",
              " [11616,\n",
              "  26288,\n",
              "  8735,\n",
              "  24515,\n",
              "  14184,\n",
              "  1451,\n",
              "  26865,\n",
              "  27092,\n",
              "  24430,\n",
              "  18196,\n",
              "  22293,\n",
              "  11616,\n",
              "  6475,\n",
              "  23637]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "metadata": {
        "id": "d4TSavm1GFQz"
      },
      "cell_type": "markdown",
      "source": [
        "### 3.3 Padding data\n",
        "In order to train our recurrent neural network later on in the notebook, it is required padding to generate inputs of same length."
      ]
    },
    {
      "metadata": {
        "id": "rSf6c-QZGFQz"
      },
      "cell_type": "code",
      "source": [
        "def max_length(tensor):\n",
        "    return max(len(t) for t in tensor)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "htZ4w776GFQ2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "df4384f8-af12-45b3-f378-e48fbd9d797b"
      },
      "cell_type": "code",
      "source": [
        "# calculate the max_length of input tensor\n",
        "max_length_inp = max_length(input_tensor)\n",
        "print(max_length_inp)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "69\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "SJvd2yKtGFQ5"
      },
      "cell_type": "code",
      "source": [
        "# Padding the input and output tensor to the maximum length\n",
        "input_tensor = tf.keras.preprocessing.sequence.pad_sequences(input_tensor, \n",
        "                                                             maxlen=max_length_inp,\n",
        "                                                             padding='post')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YO6bbOOcGFQ6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 305
        },
        "outputId": "198771b5-3930-47f6-b1f6-6c8bfd70db66"
      },
      "cell_type": "code",
      "source": [
        "input_tensor[0:2]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[11616,  8726, 23997,  8037,  9463,  9497, 17034, 11716, 18637,\n",
              "        26054, 27128, 22289,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0],\n",
              "       [11616, 26288,  8735, 24515, 14184,  1451, 26865, 27092, 24430,\n",
              "        18196, 22293, 11616,  6475, 23637,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "            0,     0,     0,     0,     0,     0]], dtype=int32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "metadata": {
        "id": "MEfQFnRHGFQ_"
      },
      "cell_type": "markdown",
      "source": [
        "### 3.4 Binarization\n",
        "We would like to binarize our target so that we can obtain one-hot encodings as target values. These are easier and more efficient to work with and will be useful when training the models."
      ]
    },
    {
      "metadata": {
        "id": "8C2_dsoIGFRA"
      },
      "cell_type": "code",
      "source": [
        "### convert targets to one-hot encoding vectors\n",
        "emotions = list(set(data.emotions.unique()))\n",
        "num_emotions = len(emotions)\n",
        "# binarizer\n",
        "mlb = preprocessing.MultiLabelBinarizer()\n",
        "data_labels =  [set(emos) & set(emotions) for emos in data[['emotions']].values]\n",
        "bin_emotions = mlb.fit_transform(data_labels)\n",
        "target_tensor = np.array(bin_emotions.tolist())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2eGW9mBGGFRC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "65e38c26-ea0c-4496-e405-68ac5233dabe"
      },
      "cell_type": "code",
      "source": [
        "target_tensor[0:2] "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, 0, 1, 0],\n",
              "       [0, 0, 0, 0, 1, 0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "metadata": {
        "id": "4pQI0HFVGFRE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 111
        },
        "outputId": "bb61d4ca-d5a6-420b-9b24-a958cefab7ad"
      },
      "cell_type": "code",
      "source": [
        "data[0:2]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>emotions</th>\n",
              "      <th>token_size</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>56648</th>\n",
              "      <td>i feel terrible especially friends from owt il...</td>\n",
              "      <td>sadness</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102135</th>\n",
              "      <td>i was feeling too lousy at work yesterday to p...</td>\n",
              "      <td>sadness</td>\n",
              "      <td>14</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                     text emotions  token_size\n",
              "56648   i feel terrible especially friends from owt il...  sadness          12\n",
              "102135  i was feeling too lousy at work yesterday to p...  sadness          14"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "metadata": {
        "id": "ZlQcTB6RGFRH"
      },
      "cell_type": "code",
      "source": [
        "get_emotion = lambda t: np.argmax(t)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1hQstuGUGFRK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "8cddaf6e-5fbd-4d2b-dd78-f66d9134ba25"
      },
      "cell_type": "code",
      "source": [
        "get_emotion(target_tensor[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "metadata": {
        "id": "3ZrTGhU2GFRM"
      },
      "cell_type": "code",
      "source": [
        "emotion_dict = {0: 'anger', 1: 'fear', 2: 'joy', 3: 'love', 4: 'sadness', 5: 'surprise'}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tyOwWrguGFRP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "79ddc1fd-fd49-4960-ccde-99acc025171c"
      },
      "cell_type": "code",
      "source": [
        "emotion_dict[get_emotion(target_tensor[0])]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'sadness'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "metadata": {
        "id": "HWSXtjpDGFRR"
      },
      "cell_type": "markdown",
      "source": [
        "### 3.5 Split data\n",
        "We would like to split our data into a train and validation set. In addition, we also want a holdout dataset (test set) for evaluating the models."
      ]
    },
    {
      "metadata": {
        "id": "AHRzfuFeGFRR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "f5cb0107-5aad-4ed0-9b67-70848f865fe0"
      },
      "cell_type": "code",
      "source": [
        "# Creating training and validation sets using an 80-20 split\n",
        "input_tensor_train, input_tensor_val, target_tensor_train, target_tensor_val = train_test_split(input_tensor, target_tensor, test_size=0.2)\n",
        "\n",
        "# Split the validataion further to obtain a holdout dataset (for testing) -- split 50:50\n",
        "input_tensor_val, input_tensor_test, target_tensor_val, target_tensor_test = train_test_split(input_tensor_val, target_tensor_val, test_size=0.5)\n",
        "\n",
        "# Show length\n",
        "len(input_tensor_train), len(target_tensor_train), len(input_tensor_val), len(target_tensor_val), len(input_tensor_test), len(target_tensor_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(40000, 40000, 5000, 5000, 5000, 5000)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "metadata": {
        "id": "ihYDuBCNGFRU"
      },
      "cell_type": "markdown",
      "source": [
        "### 3.6 Data Loader\n",
        "We can also load the data into a data loader, which makes it easy to **manipulate the data**, **create batches**, and apply further **transformations**. In TensorFlow we can use the `tf.data` function."
      ]
    },
    {
      "metadata": {
        "id": "1jrchv51GFRW"
      },
      "cell_type": "code",
      "source": [
        "TRAIN_BUFFER_SIZE = len(input_tensor_train)\n",
        "VAL_BUFFER_SIZE = len(input_tensor_val)\n",
        "TEST_BUFFER_SIZE = len(input_tensor_test)\n",
        "BATCH_SIZE = 64\n",
        "TRAIN_N_BATCH = TRAIN_BUFFER_SIZE // BATCH_SIZE\n",
        "VAL_N_BATCH = VAL_BUFFER_SIZE // BATCH_SIZE\n",
        "TEST_N_BATCH = TEST_BUFFER_SIZE // BATCH_SIZE\n",
        "\n",
        "embedding_dim = 256\n",
        "units = 1024\n",
        "vocab_inp_size = len(inputs.word2idx)\n",
        "target_size = num_emotions\n",
        "\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((input_tensor_train, \n",
        "                                                    target_tensor_train)).shuffle(TRAIN_BUFFER_SIZE)\n",
        "train_dataset = train_dataset.batch(BATCH_SIZE, drop_remainder=True)\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((input_tensor_val, \n",
        "                                                  target_tensor_val)).shuffle(VAL_BUFFER_SIZE)\n",
        "val_dataset = val_dataset.batch(BATCH_SIZE, drop_remainder=True)\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices((input_tensor_test, \n",
        "                                                    target_tensor_test)).shuffle(TEST_BUFFER_SIZE)\n",
        "test_dataset = test_dataset.batch(BATCH_SIZE, drop_remainder=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qH299tJOGFRX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "d32e4718-ea21-4100-8cd7-fd6ad25065c1"
      },
      "cell_type": "code",
      "source": [
        "# checking minibatch\n",
        "print(train_dataset)\n",
        "print(val_dataset)\n",
        "print(test_dataset)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<BatchDataset shapes: ((64, 69), (64, 6)), types: (tf.int32, tf.int64)>\n",
            "<BatchDataset shapes: ((64, 69), (64, 6)), types: (tf.int32, tf.int64)>\n",
            "<BatchDataset shapes: ((64, 69), (64, 6)), types: (tf.int32, tf.int64)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "sJUOo5xZGFRZ"
      },
      "cell_type": "markdown",
      "source": [
        "## 4. Model\n",
        "After the data has been preprocessed, transformed and prepared it is now time to construct the model or the so-called computation graph that will be used to train our classification models. We are going to use a gated recurrent neural network (GRU), which is considered a more efficient version of a basic RNN. The figure below shows a high-level overview of the model details. \n",
        "\n",
        "![alt txt](https://github.com/omarsar/nlp_pytorch_tensorflow_notebooks/blob/master/img/gru-model.png?raw=true)"
      ]
    },
    {
      "metadata": {
        "id": "VTaJh9SvGFRZ"
      },
      "cell_type": "markdown",
      "source": [
        "### 4.1 Constructing the Model\n",
        "Below we construct our model:"
      ]
    },
    {
      "metadata": {
        "id": "LAzgaQGeGFRa"
      },
      "cell_type": "code",
      "source": [
        "### define the GRU component\n",
        "def gru(units):\n",
        "  # If you have a GPU, we recommend using CuDNNGRU(provides a 3x speedup than GRU)\n",
        "  # the code automatically does that.\n",
        "    if tf.test.is_gpu_available():\n",
        "        return tf.keras.layers.CuDNNGRU(units, \n",
        "                                    return_sequences=True, \n",
        "                                    return_state=True, \n",
        "                                    recurrent_initializer='glorot_uniform')\n",
        "    else:\n",
        "        return tf.keras.layers.GRU(units, \n",
        "                               return_sequences=True, \n",
        "                               return_state=True, \n",
        "                               recurrent_activation='relu', \n",
        "                               recurrent_initializer='glorot_uniform')\n",
        "\n",
        "### Build the model\n",
        "class EmoGRU(tf.keras.Model):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_units, batch_sz, output_size):\n",
        "        super(EmoGRU, self).__init__()\n",
        "        self.batch_sz = batch_sz\n",
        "        self.hidden_units = hidden_units\n",
        "        \n",
        "        # layers\n",
        "        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "        self.dropout = tf.keras.layers.Dropout(0.5)\n",
        "        self.gru = gru(self.hidden_units)\n",
        "        self.fc = tf.keras.layers.Dense(output_size)\n",
        "        \n",
        "    def call(self, x, hidden):\n",
        "        x = self.embedding(x) # batch_size X max_len X embedding_dim\n",
        "        output, state = self.gru(x, initial_state = hidden) #  batch_size X max_len X hidden_units\n",
        "        out = output[:,-1,:]\n",
        "        out = self.dropout(out)\n",
        "        out = self.fc(out) # batch_size X max_len X output_size\n",
        "        return out, state\n",
        "    \n",
        "    def initialize_hidden_state(self):\n",
        "        return tf.zeros((self.batch_sz, self.hidden_units))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KkEFMmgvGFRc"
      },
      "cell_type": "markdown",
      "source": [
        "### 4.1 Pretesting model\n",
        "Since eager execution is enabled we can print the output of the model by passing a sample of the dataset and making sure that the dimensions of the outputs are as expected."
      ]
    },
    {
      "metadata": {
        "id": "wbgwa4AYGFRd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d64ded3f-9f08-4cb4-e8c8-68aac7c14fd2"
      },
      "cell_type": "code",
      "source": [
        "model = EmoGRU(vocab_inp_size, embedding_dim, units, BATCH_SIZE, target_size)\n",
        "\n",
        "# initialize the hidden state of the RNN\n",
        "hidden = model.initialize_hidden_state()\n",
        "\n",
        "# testing for the first batch only then break the for loop\n",
        "# Potential bug: out is not randomized enough\n",
        "for (batch, (inp, targ)) in enumerate(train_dataset):\n",
        "    out, state = model(inp, hidden)\n",
        "    print(out.shape) \n",
        "    break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(64, 6)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "2N75OFVzGFRf"
      },
      "cell_type": "markdown",
      "source": [
        "## 5. Training the Model\n",
        "Now that we have tested the model, it is time to train it. We will define our optimization algorithm, learning rate, and other necessary information to train the model."
      ]
    },
    {
      "metadata": {
        "id": "sakJKF51GFRj"
      },
      "cell_type": "code",
      "source": [
        "optimizer = tf.train.AdamOptimizer()\n",
        "\n",
        "def loss_function(y, prediction):\n",
        "    return tf.losses.softmax_cross_entropy(y, logits=prediction)\n",
        "\n",
        "def accuracy(y, yhat):\n",
        "    #compare the predictions to the truth\n",
        "    yhat = tf.argmax(yhat, 1).numpy()\n",
        "    y    = tf.argmax(y   , 1).numpy()\n",
        "    return np.sum(y == yhat)/len(y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2CFw15JbGFRk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1817
        },
        "outputId": "6cc87e51-6828-4f8c-e3b8-baeae3641703"
      },
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "    start = time.time()\n",
        "    \n",
        "    ### Initialize hidden state\n",
        "    hidden = model.initialize_hidden_state()\n",
        "    total_loss = 0\n",
        "    train_accuracy, val_accuracy = 0, 0\n",
        "    \n",
        "    ### Training\n",
        "    for (batch, (inp, targ)) in enumerate(train_dataset):\n",
        "        loss = 0\n",
        "        \n",
        "        with tf.GradientTape() as tape:\n",
        "            predictions,_ = model(inp, hidden)\n",
        "            loss += loss_function(targ, predictions)\n",
        "        batch_loss = (loss / int(targ.shape[1]))        \n",
        "        total_loss += batch_loss\n",
        "        \n",
        "        batch_accuracy = accuracy(targ, predictions)\n",
        "        train_accuracy += batch_accuracy\n",
        "        \n",
        "        gradients = tape.gradient(loss, model.variables)\n",
        "        optimizer.apply_gradients(zip(gradients, model.variables))\n",
        "        \n",
        "        if batch % 100 == 0:\n",
        "            print('Epoch {} Batch {} Val. Loss {:.4f}'.format(epoch + 1,\n",
        "                                                         batch,\n",
        "                                                         batch_loss.numpy()))\n",
        "            \n",
        "    ### Validating\n",
        "    hidden = model.initialize_hidden_state()\n",
        "\n",
        "    for (batch, (inp, targ)) in enumerate(val_dataset):        \n",
        "        predictions,_ = model(inp, hidden)        \n",
        "        batch_accuracy = accuracy(targ, predictions)\n",
        "        val_accuracy += batch_accuracy\n",
        "    \n",
        "    print('Epoch {} Loss {:.4f} -- Train Acc. {:.4f} -- Val Acc. {:.4f}'.format(epoch + 1, \n",
        "                                                             total_loss / TRAIN_N_BATCH, \n",
        "                                                             train_accuracy / TRAIN_N_BATCH,\n",
        "                                                             val_accuracy / VAL_N_BATCH))\n",
        "    print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1 Batch 0 Val. Loss 0.2983\n",
            "Epoch 1 Batch 100 Val. Loss 0.2562\n",
            "Epoch 1 Batch 200 Val. Loss 0.2513\n",
            "Epoch 1 Batch 300 Val. Loss 0.2547\n",
            "Epoch 1 Batch 400 Val. Loss 0.2642\n",
            "Epoch 1 Batch 500 Val. Loss 0.2537\n",
            "Epoch 1 Batch 600 Val. Loss 0.2552\n",
            "Epoch 1 Loss 0.2629 -- Train Acc. 0.3323 -- Val Acc. 0.3321\n",
            "Time taken for 1 epoch 65.13817477226257 sec\n",
            "\n",
            "Epoch 2 Batch 0 Val. Loss 0.2466\n",
            "Epoch 2 Batch 100 Val. Loss 0.2271\n",
            "Epoch 2 Batch 200 Val. Loss 0.0703\n",
            "Epoch 2 Batch 300 Val. Loss 0.0607\n",
            "Epoch 2 Batch 400 Val. Loss 0.0353\n",
            "Epoch 2 Batch 500 Val. Loss 0.0300\n",
            "Epoch 2 Batch 600 Val. Loss 0.0131\n",
            "Epoch 2 Loss 0.0990 -- Train Acc. 0.7551 -- Val Acc. 0.9331\n",
            "Time taken for 1 epoch 65.25825381278992 sec\n",
            "\n",
            "Epoch 3 Batch 0 Val. Loss 0.0382\n",
            "Epoch 3 Batch 100 Val. Loss 0.0201\n",
            "Epoch 3 Batch 200 Val. Loss 0.0182\n",
            "Epoch 3 Batch 300 Val. Loss 0.0177\n",
            "Epoch 3 Batch 400 Val. Loss 0.0205\n",
            "Epoch 3 Batch 500 Val. Loss 0.0076\n",
            "Epoch 3 Batch 600 Val. Loss 0.0271\n",
            "Epoch 3 Loss 0.0206 -- Train Acc. 0.9375 -- Val Acc. 0.9359\n",
            "Time taken for 1 epoch 65.36962676048279 sec\n",
            "\n",
            "Epoch 4 Batch 0 Val. Loss 0.0212\n",
            "Epoch 4 Batch 100 Val. Loss 0.0049\n",
            "Epoch 4 Batch 200 Val. Loss 0.0296\n",
            "Epoch 4 Batch 300 Val. Loss 0.0301\n",
            "Epoch 4 Batch 400 Val. Loss 0.0293\n",
            "Epoch 4 Batch 500 Val. Loss 0.0211\n",
            "Epoch 4 Batch 600 Val. Loss 0.0112\n",
            "Epoch 4 Loss 0.0171 -- Train Acc. 0.9443 -- Val Acc. 0.9343\n",
            "Time taken for 1 epoch 65.30327200889587 sec\n",
            "\n",
            "Epoch 5 Batch 0 Val. Loss 0.0095\n",
            "Epoch 5 Batch 100 Val. Loss 0.0128\n",
            "Epoch 5 Batch 200 Val. Loss 0.0092\n",
            "Epoch 5 Batch 300 Val. Loss 0.0146\n",
            "Epoch 5 Batch 400 Val. Loss 0.0026\n",
            "Epoch 5 Batch 500 Val. Loss 0.0167\n",
            "Epoch 5 Batch 600 Val. Loss 0.0205\n",
            "Epoch 5 Loss 0.0144 -- Train Acc. 0.9540 -- Val Acc. 0.9389\n",
            "Time taken for 1 epoch 65.37308096885681 sec\n",
            "\n",
            "Epoch 6 Batch 0 Val. Loss 0.0040\n",
            "Epoch 6 Batch 100 Val. Loss 0.0162\n",
            "Epoch 6 Batch 200 Val. Loss 0.0067\n",
            "Epoch 6 Batch 300 Val. Loss 0.0178\n",
            "Epoch 6 Batch 400 Val. Loss 0.0127\n",
            "Epoch 6 Batch 500 Val. Loss 0.0194\n",
            "Epoch 6 Batch 600 Val. Loss 0.0117\n",
            "Epoch 6 Loss 0.0125 -- Train Acc. 0.9627 -- Val Acc. 0.9379\n",
            "Time taken for 1 epoch 65.41738867759705 sec\n",
            "\n",
            "Epoch 7 Batch 0 Val. Loss 0.0146\n",
            "Epoch 7 Batch 100 Val. Loss 0.0057\n",
            "Epoch 7 Batch 200 Val. Loss 0.0054\n",
            "Epoch 7 Batch 300 Val. Loss 0.0092\n",
            "Epoch 7 Batch 400 Val. Loss 0.0149\n",
            "Epoch 7 Batch 500 Val. Loss 0.0055\n",
            "Epoch 7 Batch 600 Val. Loss 0.0053\n",
            "Epoch 7 Loss 0.0101 -- Train Acc. 0.9705 -- Val Acc. 0.9339\n",
            "Time taken for 1 epoch 65.44106888771057 sec\n",
            "\n",
            "Epoch 8 Batch 0 Val. Loss 0.0042\n",
            "Epoch 8 Batch 100 Val. Loss 0.0083\n",
            "Epoch 8 Batch 200 Val. Loss 0.0067\n",
            "Epoch 8 Batch 300 Val. Loss 0.0105\n",
            "Epoch 8 Batch 400 Val. Loss 0.0093\n",
            "Epoch 8 Batch 500 Val. Loss 0.0111\n",
            "Epoch 8 Batch 600 Val. Loss 0.0095\n",
            "Epoch 8 Loss 0.0087 -- Train Acc. 0.9758 -- Val Acc. 0.9351\n",
            "Time taken for 1 epoch 65.26332807540894 sec\n",
            "\n",
            "Epoch 9 Batch 0 Val. Loss 0.0075\n",
            "Epoch 9 Batch 100 Val. Loss 0.0069\n",
            "Epoch 9 Batch 200 Val. Loss 0.0032\n",
            "Epoch 9 Batch 300 Val. Loss 0.0063\n",
            "Epoch 9 Batch 400 Val. Loss 0.0045\n",
            "Epoch 9 Batch 500 Val. Loss 0.0082\n",
            "Epoch 9 Batch 600 Val. Loss 0.0022\n",
            "Epoch 9 Loss 0.0070 -- Train Acc. 0.9806 -- Val Acc. 0.9343\n",
            "Time taken for 1 epoch 65.28084063529968 sec\n",
            "\n",
            "Epoch 10 Batch 0 Val. Loss 0.0060\n",
            "Epoch 10 Batch 100 Val. Loss 0.0062\n",
            "Epoch 10 Batch 200 Val. Loss 0.0095\n",
            "Epoch 10 Batch 300 Val. Loss 0.0003\n",
            "Epoch 10 Batch 400 Val. Loss 0.0027\n",
            "Epoch 10 Batch 500 Val. Loss 0.0055\n",
            "Epoch 10 Batch 600 Val. Loss 0.0050\n",
            "Epoch 10 Loss 0.0060 -- Train Acc. 0.9839 -- Val Acc. 0.9265\n",
            "Time taken for 1 epoch 65.3649353981018 sec\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "s4pkIWl5GFRp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 287
        },
        "outputId": "5171c610-a8ef-488c-b97c-54a059cc1c93"
      },
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        multiple                  6980864   \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            multiple                  0         \n",
            "_________________________________________________________________\n",
            "cu_dnngru (CuDNNGRU)         multiple                  3938304   \n",
            "_________________________________________________________________\n",
            "dense (Dense)                multiple                  6150      \n",
            "=================================================================\n",
            "Total params: 10,925,318\n",
            "Trainable params: 10,925,318\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "LremGRu4GFRu"
      },
      "cell_type": "markdown",
      "source": [
        "## 6. Evaluation on the Testing Data\n",
        "Now we will evaluate the model with the holdout dataset."
      ]
    },
    {
      "metadata": {
        "id": "cmMrMca_GFRu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "3c7feaf8-d2b2-4f41-8d74-61b2ca5a37ac"
      },
      "cell_type": "code",
      "source": [
        "test_accuracy = 0\n",
        "all_predictions = []\n",
        "x_raw = []\n",
        "y_raw = []\n",
        "\n",
        "hidden = model.initialize_hidden_state()\n",
        "\n",
        "for (batch, (inp, targ)) in enumerate(test_dataset):        \n",
        "    predictions,_ = model(inp, hidden)        \n",
        "    batch_accuracy = accuracy(targ, predictions)\n",
        "    test_accuracy += batch_accuracy\n",
        "    \n",
        "    x_raw = x_raw + [x for x in inp]\n",
        "    y_raw = y_raw + [y for y in targ]\n",
        "    \n",
        "    all_predictions.append(predictions)\n",
        "    \n",
        "print(\"Test Accuracy: \", test_accuracy/TEST_N_BATCH)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Accuracy:  0.9274839743589743\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "pQElybUXGFRw"
      },
      "cell_type": "markdown",
      "source": [
        "### 6.1 Confusion Matrix\n",
        "The test accuracy alone is not an interesting performance metric in this case. Let's plot a confusion matrix to get a drilled down view of how the model is performing with regards to each emotion."
      ]
    },
    {
      "metadata": {
        "id": "zNrZnOVyKyov"
      },
      "cell_type": "code",
      "source": [
        "### Class to Properly Evaluate our Models\n",
        "class Evaluate():\n",
        "\n",
        "    def va_dist(cls, prediction, target, va_df, binarizer, name='', silent=False):\n",
        "        \"\"\" Computes distance between actual and prediction through cosine distance \"\"\"\n",
        "        va_matrix = va_df.loc[binarizer.classes_][['valence','arousal']].values\n",
        "        y_va = target.dot(va_matrix)\n",
        "        F_va = prediction.dot(va_matrix)\n",
        "\n",
        "        # dist is a one row vector with size of the test data passed(emotion)\n",
        "        dist = metrics.pairwise.paired_cosine_distances(y_va, F_va)\n",
        "        res = stats.describe(dist)\n",
        "\n",
        "        # print by default (if silent=False)\n",
        "        if not silent:\n",
        "            print('%s\\tmean: %f\\tvariance: %f' % (name, res.mean, res.variance))\n",
        "\n",
        "        return {\n",
        "            'distances': dist,\n",
        "            'dist_stat': res\n",
        "        }\n",
        "\n",
        "    def evaluate_class(cls, predictions, target, target2=None, silent=False):\n",
        "        \"\"\" Compute only the predicted class \"\"\"\n",
        "        p_2_annotation = dict()\n",
        "\n",
        "        precision_recall_fscore_support = [\n",
        "            (pair[0], pair[1].mean()) for pair in zip(\n",
        "                ['precision', 'recall', 'f1', 'support'],\n",
        "                metrics.precision_recall_fscore_support(target, predictions)\n",
        "            )\n",
        "        ]\n",
        "\n",
        "        metrics.precision_recall_fscore_support(target, predictions)\n",
        "\n",
        "        # confusion matrix\n",
        "        le = LabelEncoder()\n",
        "        target_le = le.fit_transform(target)\n",
        "        predictions_le = le.transform(predictions)\n",
        "        cm = metrics.confusion_matrix(target_le, predictions_le)\n",
        "\n",
        "        # prediction if two annotations are given on test data\n",
        "        if target2:\n",
        "            p_2_annotation = pd.DataFrame(\n",
        "                [(pred, pred in set([t1,t2])) for pred, t1, t2 in zip(predictions, target, target2)],\n",
        "                columns=['emo','success']\n",
        "            ).groupby('emo').apply(lambda emo: emo.success.sum()/ len(emo.success)).to_dict()\n",
        "\n",
        "        if not silent:\n",
        "            print(\"Default Classification report\")\n",
        "            print(metrics.classification_report(target, predictions))\n",
        "\n",
        "            # print if target2 was provided\n",
        "            if len(p_2_annotation) > 0:\n",
        "                print('\\nPrecision on 2 annotations:')\n",
        "                for emo in p_2_annotation:\n",
        "                    print(\"%s: %.2f\" % (emo, p_2_annotation[emo]))\n",
        "\n",
        "            # print accuracies, precision, recall, and f1\n",
        "            print('\\nAccuracy:')\n",
        "            print(metrics.accuracy_score(target, predictions))\n",
        "            print(\"Correct Predictions: \", metrics.accuracy_score(target, predictions,normalize=False))\n",
        "            for to_print in precision_recall_fscore_support[:3]:\n",
        "                print( \"%s: %.2f\" % to_print )\n",
        "\n",
        "            # normalizing the values of the consfusion matrix\n",
        "            print('\\nconfusion matrix\\n %s' % cm)\n",
        "            print('(row=expected, col=predicted)')\n",
        "            cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "            cls.plot_confusion_matrix(cm_normalized, le.classes_, 'Confusion matrix Normalized')\n",
        "\n",
        "        return {\n",
        "            'precision_recall_fscore_support': precision_recall_fscore_support,\n",
        "            'accuracy': metrics.accuracy_score(target, predictions),\n",
        "            'p_2_annotation': p_2_annotation,\n",
        "            'confusion_matrix': cm\n",
        "        }\n",
        "\n",
        "    def predict_class(cls, X_train, y_train, X_test, y_test,\n",
        "                      pipeline, silent=False, target2=None):\n",
        "        \"\"\" Predicted class,then run some performance evaluation \"\"\"\n",
        "        pipeline.fit(X_train, y_train)\n",
        "        predictions = pipeline.predict(X_test)\n",
        "        print(\"predictions computed....\")\n",
        "        return cls.evaluate_class(predictions, y_test, target2, silent)\n",
        "\n",
        "    def evaluate_prob(cls, prediction, target_rank, target_class, binarizer, va_df, silent=False, target2=None):\n",
        "        \"\"\" Evaluate through probability \"\"\"\n",
        "        # Run normal class evaluator\n",
        "        predict_class = binarizer.classes_[prediction.argmax(axis=1)]\n",
        "        class_eval = cls.evaluate_class(predict_class, target_class, target2, silent)\n",
        "\n",
        "        if not silent:\n",
        "            print('\\n - First Emotion Classification Metrics -')\n",
        "            print('\\n - Multiple Emotion rank Metrics -')\n",
        "            print('VA Cosine Distance')\n",
        "\n",
        "        classes_dist = [\n",
        "            (\n",
        "                emo,\n",
        "                cls.va_dist(\n",
        "                    prediction[np.array(target_class) == emo],\n",
        "                    target_rank[np.array(target_class) == emo],\n",
        "                    va_df,\n",
        "                    binarizer,\n",
        "                    emo,\n",
        "                    silent)\n",
        "                ) for emo in binarizer.classes_\n",
        "        ]\n",
        "        avg_dist = cls.va_dist(prediction, target_rank, va_df, binarizer, 'avg', silent)\n",
        "\n",
        "        coverage_error = metrics.coverage_error(target_rank, prediction)\n",
        "        average_precision_score = metrics.average_precision_score(target_rank, prediction)\n",
        "        label_ranking_average_precision_score = metrics.label_ranking_average_precision_score(target_rank, prediction)\n",
        "        label_ranking_loss = metrics.label_ranking_loss(target_rank, prediction)\n",
        "\n",
        "        # recall at 2\n",
        "        # obtain top two predictions\n",
        "        top2_pred = [set([binarizer.classes_[i[0]], binarizer.classes_[i[1]]]) for i in (prediction.argsort(axis=1).T[-2:].T)]\n",
        "        recall_at_2 = pd.DataFrame(\n",
        "            [\n",
        "            t in p for t, p in zip(target_class, top2_pred)\n",
        "            ], index=target_class, columns=['recall@2']).groupby(level=0).apply(lambda emo: emo.sum()/len(emo))\n",
        "\n",
        "        # combine target into sets\n",
        "        if target2:\n",
        "            union_target = [set(t) for t in zip(target_class, target2)]\n",
        "        else:\n",
        "            union_target = [set(t) for t in zip(target_class)]\n",
        "\n",
        "        # precision at k\n",
        "        top_k_pred = [\n",
        "            [set([binarizer.classes_[i] for i in i_list]) for i_list in (prediction.argsort(axis=1).T[-i:].T)]\n",
        "            for i in range(2, len(binarizer.classes_)+1)]\n",
        "        precision_at_k = [\n",
        "            ('p@' + str(k+2), np.array([len(t & p)/(k+2) for t, p in zip(union_target, top_k_pred[k])]).mean())\n",
        "            for k in range(len(top_k_pred))]\n",
        "\n",
        "        # do this if silent= False\n",
        "        if not silent:\n",
        "            print('\\n')\n",
        "            print(recall_at_2)\n",
        "            print('\\n')\n",
        "            print('p@k')\n",
        "            for pk in precision_at_k:\n",
        "                print(pk[0] + ':\\t' + str(pk[1]))\n",
        "            print('\\ncoverage_error: %f' % coverage_error)\n",
        "            print('average_precision_score: %f' % average_precision_score)\n",
        "            print('label_ranking_average_precision_score: %f' % label_ranking_average_precision_score)\n",
        "            print('label_ranking_loss: %f' % label_ranking_loss)\n",
        "\n",
        "        return {\n",
        "            'class_eval': class_eval,\n",
        "            'recall_at_2': recall_at_2.to_dict(),\n",
        "            'precision_at_2': precision_at_k,\n",
        "            'classes_dist': classes_dist,\n",
        "            'avg_dist': avg_dist,\n",
        "            'coverage_error': coverage_error,\n",
        "            'average_precision_score': average_precision_score,\n",
        "            'label_ranking_average_precision_score': label_ranking_average_precision_score,\n",
        "            'label_ranking_loss': label_ranking_loss\n",
        "        }\n",
        "\n",
        "\n",
        "    def predict_prob(cls, X_train, y_train, X_test, y_test, label_test, pipeline, binarizer, va_df, silent=False, target2=None):\n",
        "        \"\"\" Output predcations based on training and labels \"\"\"\n",
        "        pipeline.fit(X_train, y_train)\n",
        "        predictions = pipeline.predict_proba(X_test)\n",
        "        pred_to_mlb = [np.where(pipeline.classes_ == emo)[0][0] for emo in binarizer.classes_.tolist()]\n",
        "        return cls.evaluate_prob(predictions[:,pred_to_mlb], y_test, label_test, binarizer, va_df, silent, target2)\n",
        "\n",
        "\n",
        "    def plot_confusion_matrix(cls, cm, my_tags, title='Confusion matrix', cmap=plt.cm.Blues):\n",
        "        \"\"\" Plotting the confusion_matrix\"\"\"\n",
        "        plt.rc('figure', figsize=(4, 4), dpi=100)\n",
        "        plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "        plt.title(title)\n",
        "        plt.colorbar()\n",
        "        tick_marks = np.arange(len(my_tags))\n",
        "        target_names = my_tags\n",
        "        plt.xticks(tick_marks, target_names, rotation=45)\n",
        "        plt.yticks(tick_marks, target_names)\n",
        "        \n",
        "        # add normalized values inside the Confusion matrix\n",
        "        fmt = '.2f'\n",
        "        thresh = cm.max() / 2.\n",
        "        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "            plt.text(j, i, format(cm[i, j], fmt), horizontalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "\n",
        "        plt.tight_layout()\n",
        "        plt.ylabel('True label')\n",
        "        plt.xlabel('Predicted label')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fdJrBJxYGFRx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 910
        },
        "outputId": "b4656a0e-62ec-4e7c-c52d-59141a100d54"
      },
      "cell_type": "code",
      "source": [
        "evaluator = Evaluate()\n",
        "\n",
        "final_predictions = []\n",
        "\n",
        "for p in all_predictions:\n",
        "    for sub_p in p:\n",
        "        final_predictions.append(sub_p)\n",
        "\n",
        "predictions = [np.argmax(p).item() for p in final_predictions]\n",
        "targets = [np.argmax(t).item() for t in y_raw]\n",
        "correct_predictions = float(np.sum(predictions == targets))\n",
        "\n",
        "# predictions\n",
        "predictions_human_readable = ((x_raw, predictions))\n",
        "# actual targets\n",
        "target_human_readable = ((x_raw,  targets))\n",
        "\n",
        "emotion_dict = {0: 'anger', 1: 'fear', 2: 'joy', 3: 'love', 4: 'sadness', 5: 'surprise'}\n",
        "\n",
        "# convert results into dataframe\n",
        "model_test_result = pd.DataFrame(predictions_human_readable[1],columns=[\"emotion\"])\n",
        "test = pd.DataFrame(target_human_readable[1], columns=[\"emotion\"])\n",
        "\n",
        "model_test_result.emotion = model_test_result.emotion.map(lambda x: emotion_dict[int(float(x))])\n",
        "test.emotion = test.emotion.map(lambda x: emotion_dict[int(x)])\n",
        "\n",
        "evaluator.evaluate_class(model_test_result.emotion, test.emotion );"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Default Classification report\n",
            "             precision    recall  f1-score   support\n",
            "\n",
            "      anger       0.93      0.95      0.94       736\n",
            "       fear       0.87      0.90      0.88       534\n",
            "        joy       0.95      0.95      0.95      1682\n",
            "       love       0.85      0.81      0.83       404\n",
            "    sadness       0.97      0.95      0.96      1454\n",
            "   surprise       0.77      0.81      0.79       182\n",
            "\n",
            "avg / total       0.93      0.93      0.93      4992\n",
            "\n",
            "\n",
            "Accuracy:\n",
            "0.9274839743589743\n",
            "Correct Predictions:  4630\n",
            "precision: 0.89\n",
            "recall: 0.89\n",
            "f1: 0.89\n",
            "\n",
            "confusion matrix\n",
            " [[ 700   13    4    0   19    0]\n",
            " [  20  479    3    0    9   23]\n",
            " [   3    2 1598   58    7   14]\n",
            " [   1    1   69  326    2    5]\n",
            " [  32   29    9    1 1380    3]\n",
            " [   0   25    8    0    2  147]]\n",
            "(row=expected, col=predicted)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAGFCAYAAAAvsY4uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXl4FEX+/1+TACFyhMlBIBASrqmA\nSk5AERTxWLlvEV3Fi0NWFFFXueUSUVe5xBPReK3r/ly5vVZREfguZBJOKS5JwqVkhvsIkMzvj+6E\nmckkmWQ6TEjq9Tz9JN31mep3V3X1p+voKpPD4UChUCgUioogwN8CFAqFQlF1UU5GoVAoFBWGcjIK\nhUKhqDCUk1EoFApFhaGcjEKhUCgqDOVkFAqFQlFhKCejUCgUigpDORmFQqFQVBg1/C1AoVAoqjPB\niY/7/EX8ufSFJiO0VASqJqNQKBSKCkPVZBQKhcKfmKr2u37VvjqFQqFQ+BVVk1EoFAp/Yqq03SmG\noJyMQqFQ+JMq3lymnIxCoVD4kypek6naLlShUCgUfkXVZBQKhcKfqOYyhUKhUFQYVby5TDkZhUKh\n8CdVvCZTta9OoVAoFH5F1WQUCoXCn6jmMoVCoVBUGFW8uUw5GYVCofAnVbwmU7VdqEKhUCj8iqrJ\nKBQKhT9RzWUKhUKhqDCqeHOZcjIKhULhT6p4TaZqX51CoVAo/IqqySgUCoU/qeI1GeVkFAqFwp8E\nqD4ZhUKhUFQUVbwmU7WvTqFQKBR+RdVkFAqFwp+oIcwKhUKhqDCqeHOZcjIKhULhT6p4TaZqu1CF\nQqFQ+BVVk7lKEULcCYwBOgIhwB/ABmC+lHJtBZ2zPzAPaATcKqX81YA4Y4HfgceklG/5Gt/VgBDi\nA+AuKWUjH+MYBrwgpZzmIXwNsF9K+WB5z3ElcE8LIcR+YIOU8p4rcO4jwNd+T6Mq3lxWta+uiiKE\nmAGsAvYCPQEBPALUBX4SQoyooFPPAo4DcUCaQXFmA42BDw2Kz28IIR7SH+6l8SRwvQGnzAOeE0LE\nGBBXZaE9MNLfIq4oJpPvWyVG1WSuMoQQ3YFJwONSyjecgvYLIb4H/gXMEUJ8IaU8ZvDpzcA3Usp9\nRkUopcwDjhgVn5/p5I2RlPKEQedbD9QH/gEMMihOvyKlPOpvDVecKl6TUU7m6uMZYBewyD1ASukQ\nQowE8qWUxwGEECb9N48CzYHTwI/Ac1LKPbrNC8BYoIsebxJgA96QUs5xatICGCaEGAbcCjyIW7OP\ne/OXECIIeBEYiNbMdhz4BhgnpbR5ai4TQrQB5gA3A8FoNbY3pZQLnM7jAMahNRUOR3vYbgRGSil3\ne0o4p3M9ANwO9AXygffRHPc8YDBwCfhASvl3p992AGYANwI19XjmSSnf1sPXALc4aXsI2K+n9d3A\nFCBSStnQuYlICDEY7cXgDinl9/rvw4CdwFIp5aOerkUnD3gCWCOE6Cal/KE4wzLeBw8CbwK/SikH\n6U1YK4AstFqYGfgB+Kuelk87HXvQ6d5rBLwE9AAaAIeA/wdMklKeK0bnfvTmMqcmQU80l1Lu13/z\nV7Sm4zZoefct8LSU8qBTvCOACWj34G9o947iClC1XWgVQwhRA7gJWCWldHiykVLaCwq5zjRgJprz\nuBbtwdoK+EEIUdfJriawAJgOtAO+Bl4SQtzA5Sato2gPxMbAOi9lTwLuAR4GWqM9xBOBj4u5xobA\nz0AY2sPpOuAjYJ4Q4gk385HANUA3oA8Qr19DaUzU9ScD76I9JL9He/h0QHM6zwohCpxGPeA74CJw\nA9rD7E3gLSFEbz3OAYAVrXbRGPjc7XyT0Jy3C1LKL4DPgDeFELX1w6+gOYGnSrsQKeVPaHkyX78/\nisPb+6AG2gO7N/CY0/HuQDO0tB6Gljcr0NLwL2hOtQ+aEyrgU7QXl75ASz2+h3Ud3vAkWloWbC3Q\nXjh+BQ5CoYP5CC3dk/VzxQHfCyFq6TbdgLeB1UACWrrOQmte9j+quUxRiQgHgtDekEtFL2RjgcVS\nynn64d1CiIeBTUA/Lj/srwFelVJ+p/92JloNoYOUcgNwRAiRD5yTUh7RbbyRkQxscXrLzhZC9EB7\n8/XEI0AoMEhKeVg/NlsI0QntrX2+k+1pp9qGFEIs1a+pNKxONZCXgb/r1zVXP/YK8ByaM/wJOAek\nADlOTZALhBATgbuA5VJKuxDiInDBQ/p8L6VcWoKex4FtwCQhxLdoD/FuUspTXlwLaDWUncDf0Gpj\nLpTxPqgDvC6l3OQWTU3gSb15UwohpqD1K90ppTwL7BRCbEdLswIeBBxSymx9P1sI8Q1amj1d2kXp\nzYqFTYtCiA/R7tNBUsqL+uGJwM9SyrFO1/UgkI7WhPgpmmM7jNbEnKdrfRzj+hV9o4o3l1Xtq6t6\nFNRevH11iQPqAb+4HU8HzlP0zXqD0/8FbePFOQNvWQrcJYT4txBiiBCioZTygJRyazH27YE9Tg6m\ngHVASyFE/WL0Fmj2Rq/zw8Wu/83wcCwEQEp5CWgKpAohsoQQp4QQp4GGaDWu0nB/YLsgpbSjNWM9\nC3yA1gz3kxfxFvw+G61ZapoQIsKDSVnvA096N+sP6ALswC7dwTgfC3HarwVMFULsEUKc0NNsIN6l\nmQtCiNFoNeKBTk68Ptq1fetsK6XM0LUUXNd1QIab/nS0lwf/U8VrMsrJXF3kAGfRmp28oeCB7NLR\nLKXMR2uOqedmf9rp/7I6NI/oNYbeaG+gS4DDQojvhBBtS9DsqWP8pP7XWfNpNxuPTYgeOOOkz1HC\nMROAECIFrbmsDlqzUApas8shL893vHQTvkbr82iO1rRTVl5Be7DO9hBW1vvAk94zbvuOYo4VpFld\ntFrg7cB4tL6sBGBZSRfhCb3J9nW0mtR6p6CC65oihDjtvKG9bDTWw+vhdq/oeeyuX1EBKCdzFaG/\nif0E9Cmu/V0IYRZCDNfDCx4WIW42AWgFz5uHX0kUPlScKNLOLaVcIaUsaCLrA0QCq/TOaHeOu+vV\nKThm1MissjAUbYBAXynlf6WUEtiH1qxnFE/p8f0KLCombYpFSnkerQnqYSFEe7fgir4PPHErEIU2\nEOMLKeUOfYBBmfpB9D66fwMfefiOqkD362gOzHmzcLlJ7gzaS45zvAXX7n9MAb5vlZjKrU7hiVfR\nmm4muwfoD6aFwGtob3ES7aF8s5tpClrfzkYftRwHzG4O7wYnPQFCiAFCiGgAKWWulHIlMBWIwXPT\n1v+hNYtFuR3vDPwmpXSvvVwJagHn3fpIhqCNfHN3BmWu+Qkh4tA6w8eh9cfciNb5XiaklP8B/os2\n+MFZR0XfB56opf8tHJKsj+7ripdpJIQIRBtAcQitv8kF/V7YBggp5R7nDe26/tRNfwNS9PgK6KDb\n+J8q7mRUx/9VhpTyByHEVLT291i0ppWDaM0sf0d7gxxa0NkqhPgHMFkIsRWtSSYWzRHtROsv8YX/\noXUoTxRCfIQ2Ku1BJ635Qoi/Aw79734gAm1U2Da9s7y+W5xL0DqyPxdCjEN7ON6HNrqpuOGsFc16\n4HEhxFjgK7QH5aNofULXCSFi9eG0x4AEvXntD28i1h98HwK/SCk/1I+9gDay7xu91lQWngQ2o9W8\nfgeQUl6s4PvAE5vQhhM/LYSYjHZ//gNtJNxQIUQisKOUOF5C61e5De1lxjnstO5kXgQ+0dPsczQH\n9jDaIJEb0frfPkIbRv4PIcQitBrWS4C3Aysqlkrep+IrldsFKjwipZyO1tZtRntASLRht38AyVLK\nr5zMZ6KNwHlKt/screB1k1Lm+ijlc7TRXqOBrWhvm+6zDfRDG3b6hf53Odo3OL3xgJQyB+0hfgJt\nWPFWPY4HpJSpPuotL/8E5qJ9Z7EF6I/20HodiEarPYBWgzQBa9GGanvDc2gd085fub+G9vD/0O3t\nu1SklDvQnEctt6CKvA886chEGynYCa228SLa/TETrWbyM9qw5pIYzOXvnw67bc/o5/kMrTmzD9rg\njY1oNbS/SCnTdJsVaNc9EO1+movWT+Rtn5rCB0wOh7d9pQqFQqEwmuC+b/v8ED63dGSlrQ6p5jKF\nQqHwJ1W8uUw5GYVCofAnlbzj3leq9tUpFAqFwq+omoxCoVD4E9VcplAoFIqKwqScjEKhUCgqCuVk\nFIYRnPi4YePFY5uEsW3pVK7rO439B21GRcvRDfNLNyoDJuCaWibOXnB4PbFYaQQGGFsoTUBQDci9\n5P3kZ6XGafCDwwTUCoQLecZpBLh4Kd+wuEwmqFPLxJkLDoz8MqJmDeO6jisqHWvX8G2Ov6qMcjJX\nKQ3qBRMYGECDesH+llIi2iSxJkwmYx88RqM5hUoskMrfdG9Cz2uMe6GoCCpdOlY2PQajnIxCoVD4\nEdVcplAoFIoKQzkZhUKhUFQphBAxaEtx34C21s4/gfH6GkPOdgFos6YPQ1uZdx8wS0r5OV6iPsZU\nKBQKP6L1Wfq2lYMv0WZvb4E22W5/tBnV3RmFNuP4X9DWI5oAfCyEaOftiVRNRqFQKPzIlW4u05ei\niAdul1KeAE4IIV5DczKvuZknA2udlpxYIYSwoS3rscWb8ykno1AoFP7kynfJJAP7pZTHnI5ZASGE\nqOe2ON9K4E0hRALa+j93oa0y+pO3J1NORqFQKKoXYWgL7Dlj1/+G47SYm5TyS93BpOuHzqKt7ZTt\n7clUn4xCoVD4ET/1yXi7BPb9aJ3+HdCWG78beF8I0d7bE6majEKhUPgRPwxhPopWm3EmDO1r5KNu\nx8cAb0spN+r7K4UQPwD3o61CWiqqJqNQKBR+xA81mU1AMyFEuNOx9sAOKeVpN9tAfXMmqCwnU06m\nktGssZkv54/iwI9zkKumM/OJvh5vosBALev+PXcUOev+weq3xxDb5PLLyc6V0zjxv7kc2/B64fbF\n3JFF4ikPWZmZDOzXi2ZREbRt3ZzJE58nP9/zHFiLFi5ACEGj8AbccevNpFvTCsPOnTvHc888hWjZ\njKaRofTpcSc7tm8zTGP/vr1o2igc0SqWSeOfK1bjGwvnI4SgYVgIt3XtgtVJI8DePXu46Yb2xEY3\nNkRbAZmZmfTv05MmkWFYWsYwsSSNCzSNEaH16XZLZ6xplzWeP3+eMaNH0TK2KU0bhTN0yCBsNmPm\ns8vKzGRQ/17ENIngWktzppSU129oed04ogF3dnPNa7vdzohHhtE8OpLoRqHcdXtXNm38nyEar4Z0\nrExIKdPRaiEvCSHqCyHigHHAmwBCiJ1CiM66+TLgUSFEOyFEDSHEncBtwFfenk85mUrGZ68O59Cf\nJ2jbeyo9Ri2gT7d4xtx3axG7B/rcAMDzr31J01ufZ13GPr54faSLQ+r12BuYb3iqcBs89m1DNN53\nzyCiopqw9bc9LFv9LcuXfsUbC+YVsVu1cjmzZrxAamoqv2cfpnvPXgwe0IczZ84AMHnCc6z7dS3f\n/7gWuS+b6GYx3Hv3QEM03nP3QKKiotgu97Jy9XcsW/oVC+fPLWK3csVyZk7XNGYeOEKPnr0Y2K93\nocY1P/7Anbd3JSYm1hBdzgwdPICoqCbs2LWPVV9/z7Kl/2HBPM8aZ0yfSmpqKlkH/6BHr94M7Ner\nUOPUyROxWtNY88t6tuzYhcPhYOSjDxmi8a9DtbzesmMPS1d9y/JlX7HIQ16vXrmcF/W83pd1mO49\nenH3wMt5/beRj3Dy5Ak2Zexg9/5DJCYlc/fAPly8eNFnjVdDOpaEn/pkBgFRwBFgDZCK9nEmgADq\n6v+/qId9BRxHG+I8XEr5g9fX56jMsxZWMUqbhTmpbTN++vBpors9z/FT5wB4dFBnHr+3KwkDZrrY\npv17Im1bNubGoS+RsfMAJpOJfd/O4p6n3+X/tvzOzpXTGD7lY35J210mjaXNwmxN20S3mzvx+4E/\nMJvNACx+9y3eWDAf65YdLraD+vemdWsLC+a9ztkL+VzKy0e0bMbsOa8y6O57mD51Ml1v7cbNXTUn\numP7Njomx7NrXzaNo6KK1VDaLMxpaZvo2vlGsg79Wajx3Xfe4o3588jY9puL7YB+vWndujXz577O\n+YsO8vLzadU8mjkv/4PBQ+7hy39/Qdtrr2Pjxv9j8sTx7M8+XOK5Cyit4Kdt2sQtnW8g+/DRyxrf\nfouFC+ayedtOV419e9G6tYV5c18j9xLk5efTMrYpc155jQEDB9G0UTjvLUmlV+8+AMidO0ls15Y9\n+w8QVUI6QsmzMFvTNnHbLZ3Yl+2a14sWzidts2teDx5wOa/P5Gp5HdeyGS/qef35Z5/Q6aYuRDdr\nBsC2rVvo1CGRnXuyiGrSpESNJc3CXFnS0ZdZmMOGfebzQ9j24dBKOzeNqslUIhLbRJN5yF7oYAAy\nfstGNG9E3WtKbgZ1OBycPH2OduJygX383q5sXzaVP9e+yqevPEKEuW4JMXhHhjWNmJjYwgINEJ+Q\nxO5dklOnTrnapltJSEws3A8ICKBdu3jS0jYBMGXajEIHA3DgQDa1a9fGHBrqk8Z0axoxsa4aExKT\n2OVBY7o1jcSEJFeN8QmkpWl9mgMGDSauTRuf9JRJo/SsMSHRVWN8fAJpmzayb+9eTpw44RIu4uII\nDg52aa4qDxnpaTQrS14nuOb19fHxWPW8HjL0vkIHk3P0KAvnv06nm7qU+DLhDVdDOpaGn2oyVwzl\nZCoRoSF1OH7qrMsx+0mtKh/WwNVB/Jq+B4CW0RHUqlmDEYO70DTSjLl+HQA27zzAxu2ZdBgym6SB\nMzHXv4ZPXnnEZ402u50GTgUaKHQKtpwcV+02G+YGbrbm0CJ2AMeOHePv48byxNinqV27tk8a7TYb\nDdzOG2ouXqP79YSaQ7HlVGxbvN3uQWMx6Wiz2VweonA5HQv6DNzDG5jNHtO5bBrtRTQW5rWt9HQ0\nm0OL2CW1a0OLZo3I3L+fDz/+p88PyKshHas7yslUNrwsdB8v2wDA6+PvZtfq6TSJbMAv1t3k5eUB\nMOTpd3n1/W85c+4CB/44ztiX/kWX5NY0bxpeUrReUZYmVm9sjxw+TI87u9EuIYEJk6f6Iq1M5y2P\nraEYqLHirsHYdLRu+Y19WUdoF5/AX26/hbNnz5b6Gy9OXAZTf6Vj8aiajOKKkXPsNGEh17gcCwup\nQ35+PjnHXKv+Fy5qzqTP6DeIvX0CUxcup0lDMwf/POEx7sxD2ge9UREhPmkMDw/H7jbixm6zYTKZ\nCI+IcLWNiMBmd7O124ho2LBwf9/evdx2y03c2OkmlqR+SmCg+2jJcmiMiMDudl6bvXiN7tdjs9uI\naOhqZzTh4UXTxlZSOrqnuZ6OEbqte/gxu90lncunsYS8DnfTGF40He12GxERRTWER0Qw66VX+OPI\nYb79epWPGit/OpaGcjKVGCFEihDiFyHEcSHEH0KIN4UQNYUQXYUQJ4QQd+nD8c4IIb4WQpj13wUK\nIRYKIU4LIbKEEPcIIXYLIR7Uw4P18Cz9tz8KIdo6ndchhHhKCHFYCPG8Uddj3ZFFdKNQwhrUKTyW\nfG0Mv+07wplzF1xsLbGRLvtRESG0adGIDZv30ayxmXkThlCr5uVvbeOaa/a/H/Ct6p+UnEJ2dhY5\nTk0I1rRNxLVpS926rk16iUnJZFithft5eXlkZKTTvn0HAHJycujX6y7uH/YQr81baIiDAUhKSiE7\ny1Vj2qaNtPGgMSkpBWv65Tb3vLw8Nqdbad++oyFaitWYXIzGtkU1JienuPQL5OXlkZFupX2HjjRv\n0QKz2ewSvn3bNnJzc0lKTvFJY2KSltc2b/I6OZn0dNe83pyRTkr7Dpw6dYrr41qyOSO9MDwgIACH\nw0HNmjV90ng1pGOpmAzYKjFXtZMBPgd+QPtatT3QG21qaoA6wFDgRrQhee2A4XrYE2jTI3TUj9+N\nNpyvgDlAItpaC+FoY8q/FEI4Z2c/IEG39YrYJmEkxDUtdjOZ4Ld9h1k0+V46Jbagb7d2PPPQnaz+\neRsJcU3ZuWIaD/S9gYS4pnROagVA5+RWdEpswZIXH+SXtN00qBdM4/AQ+t+WwOIZ93NDfHNuvzGO\nRVPu5Ze03TQMq1eihgATJW6JiYkkp7TnhcnjOX3qJLvlThbOe53hI0YRYILkdm3ZsG4tASYYPmIU\nn37yERs2bODcubO8OudFgoKC6N6jJwEmmDZlAikdOjBh0uRSz+u8lVbeCjROmfg8p06eZNfOnSyY\n9zrDR47CBCRc14Z1v67FBIwYOYpPP9Y1nj3Ly7NnUUvXaETZL0ljSkp7Jk+4rHH+vNcYMeIxTED8\ndXGsW1ug8TE++Ti1UOMcXWOPHj2pERjII4+O4OXZsziQnY3dZmPKpPH06z+ARpGRpeooU17v2skb\n81/n0YK8jnfN68/0vD5fkNe1tHQMqV8PixBMnvAcfx45zIXc87w44wWCgoK4sdNNPuV3ZUlHX6jq\nNRkcDsdVu1kslnoWi6WW0/6nFovlQ4vF0tVisTgsFsu1TmH/slgsS/T/v7NYLHOdwiy6/YMWiyXA\nYrGctFgsdzmF17JYLOcsFktHfd9hsVj+Vla9ly7lOaoC2dnZju7duzuCg4MdkZGRjqlTpzry8/Md\nDofDAThWr15daLto0SJHdHS0IygoyNG5c2fH1q1bC8MCAgIcNWvWdAQFBblsqamplUbjHXfc4QgK\nCnLUqFHDARRq/OmnnyqNxtzcXMfo0aMdZrPZUa9ePcfQoUMdx48f91mfkRpzcnIc999/vyMkJMRR\nv359R5cuXRzr16+vVBp9TMdyP8caPvIvh6+bL+ev6O2q/k5GCNEPmAJY0OZhqwl8AbwF/AjUkVKe\n1W0/AGpLKe8RQmwD3pNSznWK6wTwJPA1cBi4gGuvZyAwVEr5byGEA+gtpVxRFr1tek11NKgXXK5r\ndccSG8mHsx9i2Pgl7Nr/hyFxAvz3w78bFhdo4xiCawZw7mJ+WfpnSyTA4Dc3ExBU00TuRUcZurlL\nibMCNNaqARculaUrvnQu5RX/nUxZCTBBcK0Azl3IJ99AkTUCjWtwqah0DPLhO5lGw//ts5Qj7w6q\ntNWZq3aCTH0qhC+Ap4F3pZTnhBAfoTmaAoorQQGA+6fGBbYFH6l0klKWNED+Uhkls/+g8cNid+3/\ng4ydBwyLz8iHA1xuj3U4jIu7oloHHBj74KkIjNZodH4XxGlkvBWRJ5Upryt9c5ePXM19MolArpRy\nvu5gTPoxb/gTiCnYEUK0AhoA6CvFFaz8hpNNrBGiFQqFwpmq3idz1dZkgP1AsL6gTiYwHshF68Av\nLdV/AEYKId4D/gBmAWecwt8GJgkh1gN7gceBCUKImILmN4VCoVCUzlVbk5FSrgcWoi0Duh3N6TwJ\nXA/8s5SfvwL8AmxGGzmWiuZkCprMZqD1zaxFq9X0B7orB6NQKAzHiOGKlZiruSaDlPJJNMfijLkY\n2wed/j8nhBgmpcwFEELUBEKBg3r4eeBv+uYprkqerQqF4mqhsjd3+cpV7WTKi76k6CtCiJuB39Ga\n2o7j5UpvCoVCYRTKyVRNPgHaog1zrg/sAPpJKU/6VZVCoVBUMaqlk5FS5qPVXsb7W4tCoajeqJqM\nQqFQKCqOqu1jlJNRKBQKf1LVazJX7RBmhUKhUFR+VE1GoVAo/EhVr8koJ6NQKBR+RDkZhUKhUFQY\nVd3JqD4ZhUKhUFQYqiajUCgU/qRqV2SUk1EoFAp/UtWby5STUSgUCj+inIzCMA7/Os+wuAL1+/Lb\n958lz8Al/iLunGFcZEBC60asf28kt41+l4zdRwyJ0/79FEPiURi7ymhBXCaTsS1A5y7kGRZXoAmC\nagSSezHP0HJTu0agcZFVMZSTUSgUCj9SxSsyyskoFAqFP1HNZQqFQqGoMKq4j1HfySgUCoWi4lA1\nGYVCofAjqrlMoVAoFBVGFfcxyskoFAqFPwkIqNpeRvXJKBQKhaLCUDUZhUKh8COquUyhUCgUFYbq\n+FcoFApFhVHFfYzqk6lMZGVlcveA3rSIbsj1cS2YOul58vPzPdq+uWgBQgiaRJq56/abyUhP82i3\nasUyzHVqsPbnNYbpbBYZwpcvDeXAsmeRnz/JzJG3eywoNQIDGN43BYAf3niY1a8/QGzjBoXh5nq1\n+WjqQPb/52n2fTmORc/2pnYtY957sjIz6d+3F00bhSNaxTJp/HPFpuUbC+cjhKBhWAi3de2C1eqa\nlnv37OGmG9oTG93YEG0FZGZm0r9PT5pEhmFpGcPEkjQu0DRGhNan2y2dsaZd1nj+/HnGjB5Fy9im\nNG0UztAhg7DZbIZozMrMZGC/XjSLiqBt6+ZMnlj8PblooXZPNgpvwB233ky6UzqeO3eO5555CtGy\nGU0jQ+nT4052bN9miMbsrEyGDOhNy+iGtItrwQsllJu39HLTNNJM91LKTajB5aa6opxMJeKBoYOJ\nimpC+rbdfLXiG1YuX8qbC4tOqrl61XJmz5xGamoqe/Yf4q7uvbhnYF/OnDnjYnfmzBkmPPc0derU\nMVTnZzPu5lDOKdoOnU+PcR/Rp0scYwbfUMTumfs6072TBYDuYz9k3dYsvnjxnkKHtOjZ3lxTuxbJ\nDy7ipuHvEBcTzqxRtxui8Z67BxIVFcV2uZeVq79j2dKvWDh/bhG7lSuWM3P6C6SmppJ54Ag9evZi\nYL/ehWm55scfuPP2rsTExBqiy5mhgwcQFdWEHbv2serr71m29D8smOdZ44zpU0lNTSXr4B/06NWb\ngf16FWqcOnkiVmsaa35Zz5Ydu3A4HIx89CFDNN53zyCiopqw9bc9LFv9LcuXfsUbC4rek6tWLmfW\nDC0df88+TPeevRg8oE+hxskTnmPdr2v5/se1yH3ZRDeL4d67Bxqi8YGhg2msl5v/lFBuvl61nBf1\ncrNbLzdDiyk3Eyug3BSHyWTyeavMKCdTSUi3bmLb1s28MGM2ISEhtGzVmtFjxvLhkveK2H6w+F3u\nu38YHTt2JDg4mCeeegaTycTXq1a42L00axq3dO1GaFi4YTqTRGPatWzEpLe+5+SZXPYetDP/X+t5\nuFdyEdteN1lY9vNvAORezGPmkjWEh9ShQ9umNDTXoXfnOKa++19sJ85x2Haa2ak/c3/3BGoE+nZb\npqVtYuuWzcx8cQ4hISG0at2UmZVYAAAgAElEQVSaMWOf4v333i1iu/i9d7h/2IOFafnU089iMplY\ntWI5AHabjZWrv6N7z54+aSqicdMmtmzZzMzZlzU+8eQ43l/8TlGN777NA8MeKtQ47ulnwWRi5Yrl\nXLp0iQ+XLGb8xMlER0cTGhrKtOmzWLVyBYcOHfJJo1VPx+mzXtI0tmrNmCfHsmRx0XR8/713+OsD\nl9Nx7Djtnly9UkvH+vVDmDX7ZaKbNaNOnTr8bcyT7N27h8M+anQuN/Wdyk1qMeXmr07lZkwx5WbO\nrGncbHC5KQnlZBRXhIx0K81iYmlgNhcei09IZPcuyalTp1xsN6dbiU9IKtwPCAjgunbxpKdtLDy2\nfdtW/vXZJ0yZNstQnYmWKDKPHOf46fOXte86jIgJp25wrSL2zrOpOxxw8sx52rVqRLtWjcjLd7Bt\n358u8dS7JggR41vhTremERMbi9kpLRMSk9jlIS3TrWkkuqVlu/gE0vS0HDBoMHFt2vikp0wapWeN\nCYmuGuPjE0jbtJF9e/dy4sQJl3ARF0dwcLBLc1V5yLCmERPjqjE+IcnjPZmRbiUhMdFFY7t28aSl\nbQJgyrQZ3Nz11sLwAweyqV27NubQUJ80bvZQbtr5UG52VFC5KQmTyfetMlNtnIwQ4lYhxAEhxA5/\na/GE3W6jQQOzyzGzWSuAdluOB9sGRWwL2uEdDgfjnhzNhCnTCAs39m0sNCSY46fPueo5pe2HhVzj\ncnzVut30u1l7QNesEcCIfik0bRiCuV4wYSHBnDhz3qt4yordVjQtQ/W0tOXkFLU1F7W15RjTp1Gs\nRg/5HRrqWaPNZnN50IOe3zk5hXnuHt7AbC4ST1mx2e1F0sZcjEa7zYbZw/3rScOxY8f4+7ixPDH2\naWrXru2TRiPKjd2t3IyvgHJTnak2TgYYC6wHrvO3kOJwOLxfRakk29QPFuPIz+eBBx8xQpYHvHt1\n+sdna/kpfT8AX718H00i6vPL5v3k5eXrsVTcK5hRaVmhGKixoq7B6HQ8cvgwPe7sRruEBCZMnuqL\ntDKd1xvb1A8Wk1+h5cYzqrms6lAf2Cul9DzsxM+Eh0dgt7u+PdvtNkwmE2HhER5s7UVswyMiyDl6\nlBenT+Ufc9+okJsv5/hZwkKCXY6F1b+G/HwHOSdcO1BzL+Qx95/rAOg57iOmvvsDTSLqczDnJEeP\nnyWkbpDLlBph9bUazNFjrvGUlfCIomlp09MyPCKiqK2tqG1EQ1c7owkPj8DmrtFWvEb30WJ2u42I\nhg2J0G3dw4/Z7UQ0bOijxvAiaWMvSaOH+9dZw769e7ntlpu4sdNNLEn9lMBA31eTDDOw3MyuwHJT\nEqq5rAoghPgJuAV4RgghhRDxQoj/CiGOCyGOCiHmCSFqOtk/JYTYK4Q4LYT4TQgxwCnsAyHEe0KI\nNUIIY8ZgAomJyRzIznJpXrCmbULEtaVu3boutglJyWSkWwv38/Ly2JKRTkr7jnz3zWrsdhv9ev+F\nls0iadkskoMHsrl3yAD+/vSTPuu0ykNENwxxcTTJcVH8tv8oZ85ddNXZuhHJcVGF+1Hh9WgTE8GG\nbdls3n0YEybatYx0iefYqXPsyvatmScpKYXsrCxynNIybdNG2rQpmpZJSSlYnYax5uXlsTndSvv2\nHX3SUKrG5GI0ti2qMTk5xaV/JS8vj4x0K+07dKR5ixaYzWaX8O3btpGbm0tScorvGrNdNVrTNhHn\nIR0Tk5LJsLrekxkZ6bRv3wGAnJwc+vW6i/uHPcRr8xYa4mDAc7lJL2O5SXYqN/17/4VWzSJppZeb\n+4YM4DkDyk1JqJpMFUBKeQvwM/AqkAh8DXwPNAQ6ALcCzwIIIW4GZgN9gXrAHOATIYTza1FfPa7r\ny6IjwKStMe5pS0xMJCk5helTJnDm1En27trJmwvm8ujwkQSaoEPitfxv/VoCTfDo8JH889OP2LBh\nA+fPneW1l18kKCiI7t17MGDgILbu2MOv69MKt8aNo1i46B0mTX6h2PMXbAmtG5W4mYDf9h9l0bO9\n6XR9NH27xPHMfZ1ZvX4XCa0bsfOfT/BA9wQSWjeiRycLM0fdAUC7Vo1YMmkAv2zOpEHd2kQ3DOFH\n6z7+8WR3bkmM5Y4OLZk58nZW/bqL61tElqqhpC0xMZHklPZMmfg8p06eZNfOnSyY9zrDR47CBCRc\n14Z1v67FBIwYOYpPP9bS8tzZs7w8exa1goLo3qNnqefxZUtMTCQlpT2TJ1zWOH/ea4wY8RgmIP66\nONatLdD4GJ98nFqocY6usUePntQIDOSRR0fw8uxZHMjOxm6zMWXSePr1H0CjyMhSdQSYit8K0vGF\nyeM5feoku+VOFs57neEjRhFgguR2bdmwbi0BJhg+YhSffqKn47mzvDpHvyd79CTABNOmTCClQwcm\nTJpc4jk9bSXdr57KzSKnctOxhHLzulu52bJjD2vXpxVujRtHsWDRO0z0otwoSsDhcFSLzWKxrLFY\nLC9ZLJbBFovliFvY/RaL5Tf9/wCLxdLAKayWxWJxWCyWW/X9DywWy8byaMjPz3eURHZ2tqN79+6O\n4OBgR2RkpGPq1KmOgt8AjtWrVxfaLlq0yBEdHe0ICgpydO7c2bF169Zi442JiXH8+OOPJZ67qmFU\nWt5xxx2OoKAgR40aNRyAIygoyBEUFOT46aefKo3G3Nxcx+jRox1ms9lRr149x9ChQx3Hjx/3WZ+R\nGgMCAhw1a9YsTL+CLTU1tdJodKeM5abcz6bkGT84fN18OX9FbyYtH6o+Qog1wAbgOPAicMEp2ATk\nSinr681mc4DBQEHtJQjoLqX8WgjxAVBXSjmorBpOnstzGPXSE2CCOrUDOXM+j3wDs/DOMUW/L/AF\nS7NwPpwykGHT/x+7snxrBivgxzdHGBJPASYgqKaJ3IsOjEpKo5swTECtGnDhEoZpBMgr5sv48mAy\nQXDNAM5dzMfIx8qlPOMiCzBB3dqBnDa43NQPLn99pv2sNT4r2Tixa6WtT1XHucvOAdullMU1dU0B\n7gZ6A5vRyvclNxv3fa8w8qZ2jtPAMkjG7iPGRebErqwcw+KuqNciRwXGbRRGazTynixoe3c4jI3X\nyPu7AKPLjS9U8i4Vn6kWfTJu7AVaCCEKewWFEGFCiHr6bgdgqZQyXR+JluQpEoVCoVCUTnWsyXwD\nHAVeFUL8HbgG+BTYCYwG9gPxQohrgFjgOeAE0MQfYhUKRdWmso8O85VqV5ORUl5EGx3WBjgCZAC7\ngWd0kxfRnG8O8AEwVf+7QAjR5wrLVSgUVZyq/p1MtanJSCm7Ov2/Ge27GU92mYD7lMJj9Q1gWUXo\nUygU1RNVk1EoFAqFopxUm5qMQqFQVEaqeEVGORmFQqHwJ1W9uUw5GYVCofAjVdzHqD4ZhUKhUFQc\nqiajUCgUfkQ1lykUCoWiwvCHkxFCxACL0D7XOA38Exjvab0tIUQc8BbabCg24DUp5evenks1lykU\nCoUf8dPHmF8CB4EWwO1Afy5/C1iIECIYbZaUlUA4MAB4RHc8XqFqMgqFQlGNEEKkAPHA7VLKE8AJ\nIcRraE7mNTfzu4ETUspX9P2NlHEJe1WTUSgUCj/ih5Uxk4H9UspjTsesgHCaKLiAzsBWIcT7+krC\nO4UQ95XlZMrJKBQKhR/xQ3NZGHDM7Zhd/xvudrwp0A9tJeEotFWDU4UQid6eTDWXKRQKhR/x0+gy\nb09qAtKklJ/q+x8KIUahLeqY7k0EqiajUCgU1YujaLUZZ8LQ1sM76nb8CNpqws7sBxp5ezJVk7mC\n1K4VaFhcBa8htWoGGrpS4rH/TjUwtss617w10jCd5vaPGxSTRkJcU9Z/9jy3PjCHjJ0HDInz8Lp5\nhsRTQKAJgmoEcuFSnqErOgbVMO49syCvA0wmQ79iDzAwsoKoatYIoEb1XRlzE9BMCBEupSxYE709\nsENKedrNdgcwWghhklIWpFgs8LW3J1M1GYVCofAjASaTz1tZkFKmo40Se0kIUV8fjjwOeBNA79zv\nrJt/jNZPM0EIESyEGIo2cOBjr6+vTOoUCoVCYSh++k5mEFpH/hFgDZCK9nEmgADqAkgpDwE90fpg\njgHTgL5Syr3enkg1lykUCkU1Q0p5AOhRTJjJbf8nIKG851JORqFQKPyImrtMoVAoFBVGQNX2McrJ\nKBQKhT+p6jUZ1fGvUCgUigpD1WQUCoXCj1Txiox3TkYIUcvbCKWUF8ovR6FQKKoXJq9neLk68bYm\ncx68/mDbuM/aFQqFooqjOv41HsZ7J6NQKBQKBeBlx7+U8gMp5YfebBUtuCqTmZlJ/z49aRIZhqVl\nDBPHP0d+fpHVUAF4Y8F8hBBEhNan2y2dsaalFYadP3+eMaNH0TK2KU0bhTN0yCBsNlu10tmssZkv\n54/iwI9zkKumM/OJvh5H8dSoEcDwQV0A+GHJOFa/PYbYJpfnDty5chon/jeXYxteL9y+mDvSEI1Z\nWZnc3b83LZo25HrRgqmTni82Hd98YwFCCJo0NHPXbTeTYb2cjsfsdkY9+iCtmjUipnEYPe7oStrG\n/xmjMTOT/n170bRROKJVLJNKyuuFWl43DAvhtq5dsFrd8vpvo2jVPJroxhHcO2SwYXmdlZnJgL69\niG4cTlzrWCZNKF7jIl1jZFgIt9/ahXQnjQB79+yh843tad6ssSHavMEP68lcUco1ukwI8ZAQYo0Q\nYp++X0sI8Zyx0qofQwcPICqqCTt27WPV19+zbOl/WDBvbhG7lSuWM2P6VFJTU8k6+Ac9evVmYL9e\nnDlzBoCpkyditaax5pf1bNmxC4fDwchHH6pWOj97dTiH/jxB295T6TFqAX26xTPmvluL2D3z0J10\nv1lb6K/7iPmsy9jHF6+PdCm4vR57A/MNTxVug8e+bYjGB+4ZTFRUE9K37+arld+wctlS3lxYdGLN\n1SuXM3vmNFJTU9mTeYi7evTinoF9C9Px8VGPcvLECf6Xvh35+0ESk5K5Z1BfLl686LPGe+4eSFRU\nFNvlXlau/o5lS79i4XzPeT1z+gukpqaSeeAIPXr2YmC/3oUaX5gykXSrlTU/r2Pzdqnn9cM+6wMY\nOkTTuG3nXlas+o7lxWhc5aRx/4EjdO/Zi4H9L2tc8+MP/OWOrsTExBqiy1v8NK3MFaPMTkYIMQaY\nB2wFCtx9BPA35WjKT9qmTWzZspmZs+cQEhJCq9ateeLJcby/+J0itovffZsHhj1Ex44dCQ4OZtzT\nz4LJxMoVy7l06RIfLlnM+ImTiY6OJjQ0lGnTZ7Fq5QoOHTpULXQmtW1GO0sTJs37ipOnz7M36yjz\nP/6Bhwd0KmLb65brWfbjZgByL15i5lurCDfXpcP1sT5pKI30tE1s27qZF2bOJiQkhJatWjN6zFg+\nfP+9IrYfLH6X++4fVpiOTzz1DCaTia9XrQCg74BBzHltHqFhYdSuXZuhfx1GztGjHP3zT580pqVt\nYuuWzcx88XJejxn7FO+/924R28XvvcP9wx4s1PjU089iMplYVZjX7/P8hEk01fP6hekzWb3K97y2\n6hpnOGt88imWLPZC4zhd48rlANjtNlas+o67evT0SVNZudITZF5pylOTeRxtgrQx6P00UsqDwABg\nlIHaqhXp1jRiYmMxm82FxxISk9glJadOnSpim5CYVLgfEBBAfHwCaZs2sm/vXk6cOOESLuLiCA4O\nLtI0UFV1JraJJvOQneOnzhUey/gtG9G8EXWvCSpi73A4XP4/efoc7USTwmOP39uV7cum8ufaV/n0\nlUeIMNf1SR9ARrqVZjGxNHBKx/iERHbvKpqOm9OtxLul43Xt4klP2wjA3ffcS3R0MwByjh5l0YK5\n3HhTZxpHRfmksdi89qAx3ZpGYoKrxnbxCaSlVWxep1vTiInxUmN6GgnuGttp9yPAgIGDiWvTxic9\niqKUx8lEo83a6Y6VyzWbqwYhRIwQ4rwQwuJPHXa7jQYNzC7HQkNDAbDl5Lgct9lsLoUKwGwOxZaT\nU9jO7R7ewGwuEk9V1RkaUofjp8666j6pNYmENXB1EKt+3ka/27S5/2rWCGTE4C40jTRjrl8HgM07\nD7BxeyYdhswmaeBMzPWv4ZNXHvFJH3hOR7OejnZbjgfbBkVs3fs02se3pXVMYzL3/86Sj/7pc1u9\n3eYhr82e89pus7k4zAJbW44Nu72EvLb5ltd2e9Hzmsug0VM6XmlUc1lRDgEtPRxP4fI60VcNUspM\nKWVtKeUuf2vB4f0APkcptqWF+8TVoNPLkvePD77jp427Afhq4WiaRDbgF+tu8vLyABjy9Lu8+v63\nnDl3gQN/HGfsS/+iS3Jrmjd1Xwq97JTl2r2x3bh5B7szD9MuPpEed3Tl7Nmzpf7GiPN6a1tReW10\nOl5pVMd/Ub4C/iWE6AmYhBBJQogRwP8D/mmoumpEeHgENrvrG5XNZsNkMhEeEeFqGxFR5O3LbrcR\n0bAhEbqte/gxu52Ihg2rhc6cY6cJC7nG5VhYSB3y8/PJOebahJJ74RJzU78HoOeoBUxduJwmDc0c\n/POEx7gzD2nvUVERIT5pDA+PKHzDL8Cup2NYeIQHW3sRW/f0Bi3NZ8x+mT+OHOa7b1b7pjGiqEab\nvfi8ttuK2kY0jCBcvx738GN2OxERvuV1eHjR89rLoNFusxXei/5C1WSKMhGt038pEIS2lOdCfX+8\ncdKuDEKIWCGEQwgRJ4QwCyFShRCHhRCnhRArhRCxut0efdCD828XCyE+NUJHUnIK2VlZ5DhV8dM2\nbaRN27bUrevaxJOcnOLSlp2Xl0dGupX2HTrSvEULzGazS/j2bdvIzc0lKTmlWui07sgiulEoYQ3q\nXNZybQy/7TvCmXOuE1IkxDUl+dqYwv2oiBDatGjEhs37aNbYzLwJQ6hV8/LnZHHNIwH4/YBvzTyJ\nSckcyM5yadKxpm1CtCmajglJyWRYrYX7eXl5bNmcTkr7jpw6dYr4Nq3YkpFeGB4QEIDD4aBmjZo+\naUxKKiavPWhMSkrBmu6a15vTrbRvfzmvrRWQ14nJKWRnF9UYV4zGdDeNGRna/aioOMrsZKSUuVLK\nYWgjym4A4oEGUsrHpZS5Rgu8wryH1q/UDm3VuLPAv/Swj4D7CgyFEIFAX8qwDKmphC0xMZGUlPZM\nnvA8p06eZNfOncyf9xojRjyGCYi/Lo51a9diAkaMfIxPPk5lw4YNnDt7ljmzZ1ErKIgePXpSIzCQ\nRx4dwcuzZ3EgOxu7zcaUSePp138AjSIjS9TgzVYZdCbENS1xM5ngt32HWTT5XjoltqBvt3Y889Cd\nrP55GwlxTdm5YhoP9L2BhLim9Lj5emY+2ReAdqIpS158kF/SdtOgXjCNw0Pof1sCi2fczw3xzbn9\nxjgWTbmXX9J20zCsXokaAk2UuCUmJpKUnML0KRM4c+oke3ft5M0Fc3l0+EgCTdAh4Vr+t24tgSZ4\ndMRI/vnpR2zYsIHz587y2ssvElQriO7de9Cgfj0sQjB14nMcPXKYi7nneWnmC9QKCqJTp06l6igt\nr5NT2jNl4uW8XjDvdYaPHKXlw3VtWPdrQV6P4tOPPyrM65f1vO6u5/XDjwznlZdeLMzrqZMm0Lef\nl/dkCW/whRonPc+pUyfZJXeyYL6u0QSJ17dh/bq1mEww3F3jS7MI0jV6qhlcqZpEVR9dhsPhKPNm\nsVhCLRbLUIvF8pzFYnnSYrH0s1gsweWJy9+bxWKJtVgsDovF0sliseRbLJaOTmFt9bDmFoulhR7e\nUg/rZrFY/rRYLDW8PVd+vqNEsrOzHd27d3cEBwc7IiMjHVOnTnXk6z8CHKtXry60XbRokSM6OtoR\nFBTk6Ny5s2Pr1q2FYbm5uY7Ro0c7zGazo169eo6hQ4c6jh8/XvLJy8DVorOyY1Q65uTkOO6//35H\nSEiIo379+o4uXbo41q9fX6k0VmReG6XxjjvucAQFBTlq1KjhABxBQUGOoKAgx08//eSNjHI/g4Z8\nYHX4uvly/oreTFo+eI8Q4hZgBXAN2prPJsCs/99TSrnBOBdY8ejNYb8DnYB1QIiU8qQeVgvIBbpJ\nKX8UQvwCfCelnC6EWAggpXzc23PlXjJuah4TUKsGXLhUuef7qQidXe9/yaCYNCyxkXw4+yGGjV/C\nrv1/GBLnt0ueNSSeAgJMUCcokDO5eeQbmOE1A41b7cMEBNU0kXvRYeg9aWRcJqB2TRPnDdYYXLP8\n1YmhqRk+S/nsgYRKW50pz1T/c4EPgKlSSjuAECISmI7WN+N7w79/KCmjC8JSgaeFEDOAfsBgo05Q\nXhwVFK/RGKkzY+cBg2JyZdf+PwyLO6+CMiXfYWzcFbHWh9H3ZBnfg0tGfxQ7jI5XUSzleY1pDTxb\n4GAApJR/AOOAOKOE+YGC5Qycr6Hg/736338BMWgfnZ6XUq6/QtoUCkUVJcDk+1aZKY+T2QeEeThe\nD8j0TY5f+RP4BpghhAgVQpiBWcCPUspsACnlCWAZ8BJgyKgyhUJRvanq38mUZ9GyZ4C3hRCzgS1A\nPnAt8DwwxsPPryaGAYuAnWjX9T3woJtNKnA38MkVVaZQKKokldxH+Ex5Fy0zAd3dbExAb8C3wfn+\nw6E3+w0sxa4hsFZKKa+AJoVCobiqUYuWXZ4i51hphvr8ZjOAERWqSKFQVBsqe3OXr3jlZKSUH3hj\nJ4SY7pOaK4wQ4q9oH2AukVKWOC+6EOIttNFkr0kpV10JfQqFoupT2TvufaVcIxiFEHFAB6C20+Fm\nwFPAFAN0XRGklB/j5Rf7UspRqKUMFAqFwaiajBtCiHvROr8D0JrQClLoGNpiZgqFQqFQAOUbwjwB\nGA0EAxfQHFUXYC1QdHlEhUKhUBSLr/MJVvZ6UHmcTAzwbsFkmFLKfCnlr2jfjigno1AoFGWgqk+Q\nWR4ncwGor/9/WghRsBrm/9BmZVYoFAqFl6j1ZIryNbBSCFEHzbG8LoRIAZ4EPK/0pFAoFIpqSXlG\nlz2FNuz3EjAJ+A7tC/iLwN+Mk6ZQKBRVHzW6zA39e5I++m66EKI50BbYr38xr1AoFAovqeI+xuu5\nyyylmBwDQoQQIVLKXb7LUigUiupBZe+49xVvazI7KX1aGZNuE+iTIoVCoVBUGbx1MrdWqAqFQqGo\nplTxiozXc5f9VNFCFApv2bTC2OWXa9fUBll+PO8Jzl/MNyTO6Ie8mq3Ia+JjQ1n7Um+6v7CKzfvt\npf/AS/786AHD4qooKmBhTMPj9QXV8a9QKBSKCqM835FcTVT161MoFAqFH1E1GYVCofAjqrlMoVAo\nFBWGWk/GA0KIbsADQDMpZTchRAAwWEr5uaHqFAqFoopT1Z1MmftkhBBD0OYvCwM66YebAm8LIR4x\nUJtCoVAornLKu57MfVLK3uijAKWUWWhLEz9joDaFQqGo8phMJp+3ykx5mstaAV/q/zsPNf8v0Nxn\nRQqFQlGNqOrNZeVxMjlAQ+Cw23ELcMpnRQqFQlGNqOQVEZ8pj5P5DnhfCPEMgBAiFEgBXgWWG6hN\noVAoFFc55XEyzwBLga36/lG02RpWAU8bpEuhUCiqBVV9FuYyd/xLKY9LKW8BEoF7gH5AnJSyl5Ty\nmNECqxOZmZn079OTJpFhWFrGMHH8c+Tne55L640F8xFCEBFan263dMaallYYdv78ecaMHkXL2KY0\nbRTO0CGDsNls1UrnoQNZPPbAQG66rhl3dGzLa7MmF6vxzJnT/PWvf6VVZB327ZEucSS1DC+yXde0\nHocOZPmsMTq8Dv/++21kvjuE7QsGMv3eJI9NJyYTPHy7ttrGt9O6s35ObwbcGOti0yKyHj/N6sme\nt+72WZczWZmZ9O/bi6aNwhGtYplUUl4v1PK6YVgIt3XtgtXqltd/G0Wr5tFEN47g3iGDDcvrrMxM\nBvbtRbPG4bRpHcvkCcVrXKRrjAwL4Y5bu5DupBFg7549dLmxPS2aNfb4+4ogwICtMlNufVLKzVLK\nf0kpl6s1ZIxh6OABREU1Yceufaz6+nuWLf0PC+bNLWK3csVyZkyfSmpqKlkH/6BHr94M7NeLM2fO\nADB18kSs1jTW/LKeLTt24XA4GPnoQ9VK59jh9xHZKIqv123lvc+W8d+vl/PRe28UsfvzyGH63dGZ\nwMCiK1RENW2GdW+Oyzbj1UW0S0yhcZNonzV+Mu5WDh07y/VPfEmfWd/Su30z/ta9bRG7R24X9G7f\nDIC7XljNtM/Tee9vXbi2mRmAm69txOqpd5GVc9pnTe7cc/dAoqKi2C73snL1dyxb+hUL53vO65nT\nXyA1NZXMA0fo0bMXA/v1LszrF6ZMJN1qZc3P69i8Xep5/bAhGu8dMpDGUVFs3bmX5au+Y/nSr3jD\ng8ZVThr3HzhC9569GNT/ssY1P/7AXXd0pVlMrCG6vMVk8n2rzJTnO5l8IURecVtFiKwOpG3axJYt\nm5k5ew4hISG0at2aJ54cx/uL3yliu/jdt3lg2EN07NiR4OBgxj39LJhMrFyxnEuXLvHhksWMnziZ\n6OhoQkNDmTZ9FqtWruDQoUPVQue2zVbkjq08NWE69eqHENOiFQ+MGMO/P1lSxNZuz+Hvk2cybdq0\nUuM9c/oUr82azPjpr/g8bDSxRRjXx5iZ8mkaJ89dZO+RUyxYuYOHbiu6PmBiizC26DMv5zvga+sB\n7KdzuU53MqF1g+gz81u+th7wSZM7aWmb2LplMzNfvJzXY8Y+xfvvvVvEdvF773D/sAcL8/qpp5/F\nZDKxqjCv3+f5CZNoquf1C9NnsnqV73lt1TXOcNL4+JNP8f7iohrfd9M4dpyucaXWlWy321i+6ju6\n9+jpkyaFK+WpyYx228YAbwC/A8a8mlRD0q1pxMTGYjabC48lJCaxS0pOnTpVxDYhMalwPyAggPj4\nBNI2bWTf3r2cOHHCJVzExREcHFykaaCq6tyxNYOopjGENLisse118fy+dzdnTrtqjGt7PXd07+VV\nvEvemk9CSkeuT0zxSSfHSH4AACAASURBVB9AQvMwMo+e5viZC4XHNv9ux9IkhLq1XbtKv7EeILFF\nOAA1Ak30SI4muFYga387AsBX/5eJPHTCZ03uFJvXuzzndWKCa163i08gLa1i8zrdmkZMTFGNuz1p\nTE8jwV1juwSsmzYCMGDgYOLatPFJT3kIMJl83iozZe74l1K+5em4EOL/ASOBD30VVR2x2200cHoo\nAoSGhgJgy8mhXr16hcdtNptLoQIwm0Ox5eQUtnO7hzcwm7Hl5FQLnceP2ajfoIHLsQKHc8xuo07d\nep5+ViJnTp/isw/eZvHnK3zSVkBYvSAXBwNw7HSuHlab0+cvN30t25jF7QlNeOg2C2tm9eLM+YuM\nWLSWg7azhmgpDrvNQ16bPee13WajgbmorS3Hht1eQl7bfMtru73oec1l0GgODTW0v7I8VHIf4TNG\nTpD5M9qos0qFECIWrZbVRkq5089ySsbh/TJKjlJsSwv3iatAp9HxLv3iU1qLtsRd286wOL19ttzT\npQXdk5oC0G3SSkLrBbF4zM0cyDmDdV/FPiDLko5XQ15XaLkoJ1X9Y0wjByb0AS4aGF+1Ijw8Apvd\n9YFhs9kwmUyER0S42kZEFHn7stttRDRsSIRu6x5+zG4nomHDaqEzNDScE8dcV488fsyOyWQiNCy8\nXHF+u+I/dL2zh0+6nMk5eZ7QekEux0LrBZGf7yDn5HmX4yP/0oal/5cJwIVL+XyTfpCftx3hni4t\nDdPjifCIiMJaSAE2e/F5bbcVtY1oGEF4uGbrHn7Mbiciwre8Dg8vel57GTTabbbCe1FRMZSn4/+w\nEOKQ23YcbaoZNQtzOUlKTiE7K4scp6aitE0badO2LXXr1nWxTU5OcWnLzsvLIyPdSvsOHWneogVm\ns9klfPu2beTm5pKU7HtfwtWg89r4JA4fzOaY/bLGbZuttLTEcU2duiX80jPHj9mwblxPp5tv80mX\nM9Z9NqLD6xDm5GiSWoaz8+BxzuRecrENDDAR4Pa6W6tmxQ9cTUoqJq/bFM3rpKQUrOmueb053Ur7\n9pfz2loBeZ2UnEJ2dlGNccVoTHfTmJFhJaVDR580+EpV75Mpz536FvC22/Yi0E9K+biB2gxHCGEW\nQqTqjvK0EGKlECJWCBGgHxvmZr9UCPGW/n+8EOK/QojjQoijQoh5QoiaZTm/qYQtMTGRlJT2TJ7w\nPKdOnmTXzp3Mn/caI0Y8hgmIvy6OdWvXYgJGjHyMTz5OZcOGDZw7e5Y5s2dRKyiIHj16UiMwkEce\nHcHLs2dxIDsbu83GlEnj6dd/AI0iI0vU4M1WGXTWrhlQ4paYmMj1CcnMf+kFLp4/zcH9u/no3YXc\n9+BwatcMoE/XZLZZNxTa16pxuZAG1Sga3/9v77zjo6i2APwtLXRICEV6MweQEpq9PRuCoCKiYnk2\nuiCKBRCRLlhAKfpUbA/F3pBmfTbsJBQbB0TpopIgvYXs++POhk0IKWQ3u4Tz8dsf2Zk7d87e2Z0z\np9xzf1/xM+np6TRq1CDXcwdererH5fjyAcvXb2Va71M4RarRpX1d7ri4Oe8lradV/Th+nNqN685q\nTKv6cSSv2kzXk+sD0KRWJXqeL/yrxXH8sn5Lpj7rxJejRHFfrufOKkdO17ptu/bcN/zgtZ425RF6\n9emLD0hs3pSvvgxc67689OILGdf6Qe9ad/Su9U039+KhifdnXOuR997DJZfm7TtZzHf4V0DGkfcO\nZcf2bazU5Uyf6mQs5oM2LZryzVcLKeaDXllkfGjieGK872Nwn4F7dk7nzfoqCEU9hRm/35+vV0JC\nQtv8HhPJV0JCQv2EhAR/QkJCk4SEhDcTEhI+TEhIqJqQkFAxISHh9YSEhO+8dtMTEhLeDjquXEJC\nwu6EhISzExISyiYkJPyRkJAwLCEhoVRCQkKDhISEZQkJCffkR5b0dH+OrFu3zt+xY0d/mTJl/NWr\nV/ePHDnSn+4dBPgXLFiQ0fbxxx/316lTxx8TE+M//fTT/T/88EPGvr179/r79+/vj42N9VeoUMHf\no0cP/z///JPzyfPB0SBnXmUcO3asPyYmxl+qVCk/4C9VqpQ/JibGP3bs2Iy+XnrpJX+5cuVCItfR\nRlG61rnJeP755/tjYmL8JUqU8AP+mJgYf0xMjP+zzz7LixhHfI8a99FKf0FfBTl/uF8+dx3yjohs\nA2JV9aiYExMU+D8NWAicoqrfevuaAT8BDYE6wAIgXlV3i0g3YBpurZxuwDRVrRHU73XAPaqa55zH\nvWmELOroA0qVgH1phK7TMBAOOdenhjarqlQJH3XiyrAudTf70kIj5fWTPw5JPwESalbi2VvP5Kap\nn7MihOnKH43NW/p2XvABMSV97N3vD+l3MpR9OUvYx54Qy1im5JHbE+M//rXAogw/t3HU2jNHkl32\nKnC7iExS1Wi+v2XFj/uO/RK07Vfv//rAZ8BW4AJcllxX4HVVTReRRkA1EQmOyPqAvfkVINT4w9Rv\nqAmlnHv2Z18y5MhxXuN9af6Q9b10dWrujY6AFRu3hrTvo+E7mR7CzgKuLX+I+y0IvjznGR6dHImS\niQe6AENEZA2QKdlfVU/N9qjIk9NXyu8pkzeBS0VkPtAZCKQT7QZ+UtUW4RbSMIxji0ikMItIPeBx\n4GRgB/AKMExVD/uUJSK1gOXAJFUdlddzHYmS+QfnVjraKOX93wT4LuhvgFXe/68DbwDn4ayar4P2\nNxSR8qq6A0BEqgD7VNXW0DEM44iJ0DyZt4AkXKigGjAP+BOYnMMxU4F8h0mOZMb/jfk9Jkr4C3gf\nGCsiPXCWzXjgE1Vd57VZCKQBw4DXgtyB7+OWNHhYRO4GygIv4bR6/8L7CIZhGAVDRNoBrYDzVHUr\nsFVEJgO3cRglIyKdgGZAvkte5DmFWUTCW8OicLgeZxoux8VmtgLdAzs9U/FN4Ayc+RjYvh+4BGgK\nbAKWACtxa+sYhmEcMT6fr8CvfNIWWJ1laZZkQETkkJpLIlIGmI57oE7Luj838mPJHJXRKVVdTWbZ\nu+XSfiCu6GfW7UuBs0IqnGEYxzwRcJdVAbKu/RXIJokHsoYA7gO+VtVPss4lzAv5UTJRkothGIZR\ndIjQZMo8ndWb5tETOOKkp/womRIi0ouchfOr6qELORiGYRjRwt84ayaYKjhD4u/ABhHxAf8BRqnq\npiM9Wb6UDK6ETE74AVMyhmEYeSQCtccWAXVFJF5VA0Xf2gM/B7JnPeoCZwIniMhob1t5IF1ELlbV\nNuSB/CiZPapaNh/tDcMwjFwo7JiMqi4Wke+BiSIyGKgJDAYmAYjIcpyL7GtcJZRgJgPrgQfzer5Q\nridjGIZh5JMIxWQuB57CZctuwxU+ftzbJ0B5r3RYpjW9vSzjbflxnxX57DLDMAwjM6q6noMVTbLu\nO+y9XlVvyO+58qNkXshv54ZhGEbOFCviz+95VjKq2jucghiGYRyLRP16MAXEYjKGYRgRJEK1ywqN\n8K/hahiGYRyzmCVjGIYRQSIwT6ZQMSVjGIYRQYq4jjElYxiGEUmKuiXj8/ut7mVhsWNv6BZ8LeaD\nsqWKsWtfetQsI5sd4ZCzeIgjpeFY9/0Iyq/n3B8QUwL2poW2Um1s+wEh6yuxSW2+fnkop/SYyJLl\n63M/II9s+X56yPoK1ziWLnHkecjPfLe2wKLcfGLdqNVUZskYhmFEkCJuyJiSMQzDiCRFPcXXlIxh\nGEYECbVrNdoo6krUMAzDiCBmyRiGYUSQom3HmJIxDMOIKEU9hdmUjGEYRgQp2irGYjKGYRhGGDFL\nxjAMI4IUcW+ZKRnDMIxIYinMRqGxds0aul3ambo1q9Ls+AaMGD6U9PT0bNs+Pn0aIkKN+Mqc/68z\nWZyclLFv9+7dDLnzdqRRXWpXj+PiThfw808/HlNyrl2zhq6XdKZ2jXikcX3uHTbksDI+Nn0qIkK1\nKpU49+wzSA6SEWDVr79y2sntqV/nuJDIFmDNmjV0vfgialWvQkKjegzPScZpTsaqcRU556zTSU46\nKOOePXsY2L8vjerXpnaNeHpceTkpKSkhkbHucbG8NbUv6z95AJ0/hnG3XpLtTbFEiWL0uvwMAP73\n3GAWPDmQ+rWqZOxfPm80W797lC3fPJLxev3RPiGR8WgYx5woFoJXNBPt8h1TXHPV5dSsWYsffvmV\ndxd8wJzZ7/DYtCmHtJs/bw7jx45i5syZ/L7uDzpe1Jnul13Mzp07ARhxzxC++nIhH32yEP1tHXXq\n1uPqK7odU3JedUU3atasyU+6inkLPuTd2e8wfeqjh7SbN3cO48Y4Gdes30SnizrT7dIuGTJ++sn/\nuOC8s6lXr35I5AqmR/fLqFmzFj+v+I35733Eu7PfZtqU7GUcO2YkM2fOZO2GP+nUuQvdLu2cIePI\nEcNJTk7i0y++ZtnPK/D7/fTpeWNIZHz54V5s/GsrzbqMpFPfaVx8TisGXvOvQ9rdeeMFdDyzOQAd\ne0/lqyW/8fojfTIppM79HiP25NszXt1vezIkMh4N43gsY0omSkhOWsQPy5YyZvxEKlWqROPGxzNw\n0G0898yMQ9o++/RTXPvvGzjppJMoU6YMtw2+E5/Px4J5cwCoWLES4yc8SJ26dSlXrhy3DBzEqlW/\n8sfGjceEnEmejOPuf8DJePzxDLztdp59+lAZn3n6Ka67/qCMt99xFz6fj/lznYypKSnMW/AhHS+6\nqEAyHSLjokUsW7aUcRMOynjroME8+8xTh8o440n+ff2NGTIOvuMu8PmYN3cOaWlp/Pe5Zxg2fAR1\n6tQhLi6O0WPGM3/eXDYWcBzbNKtLy4Ra3DvlHbbt2MOqtX8z9cX/cdNlpx7StvNZLXj3k6UA7N2f\nxrgn5hMfW54TW9QvkAy5cTSMY274fL4Cv6IZUzJRwpLkJOrVq09sbGzGtlaJbVi5Qtm+fXvmtouT\nSWzdOuN9sWLFaNmyFUlJiwC4b/RYzjz74NPm+vXrKF26NLFxcceEnIuTk6hXP7OMia3bsCIbGRcn\nJ9E6sU1mGVslkpT0PQCXXd6dJk2bFkiefMmo2cuY2DqzjK1aJZK06Ht+W7WKrVu3ZtovTZpQpkyZ\nTK7JI6F10zqs2ZjKP9t3Z2xb8ss6pEENypeNOaR9cEV3v9/Pth27aSm1MrYNuPpsfnp3JH8tfJiX\nHrqZqrHlCyQfHB3jmBu+ELyiGVMyUUJKaiqVg34oQMbNNmXz5kzbU1NSiK2cpW1s3CHtALZs2cLd\ng2/j1tvuoHTp0seEnKkpKVTOct642MPLmPXzxMXGkbI5vL741NRsZDzMOKakpGS6icLBcQzEDLLu\nrxwbm+0454e4SuX4Z/uuzHJvc66lKpUzK4j5n//IpecmAlCyRHF6dz+D2tVjia1YDoCly9fz/U9r\nOPHKCbTpNo7YimWZ9dDNBZIPjo5xzA2zZI4CROQbERkVaTkKSn7W9slL201//EGnC86hZWIi94wY\nWRDR8n3u/LQNh5yhljEshFDGsH2GPN7AJj3/IZ99vxKAd6b3p1b1ynyRvJIDBw4AcOUdM3j42Q/Y\nuXsf6//8h9smvsYZbY+nQe34gst4NIzjMUyRUDJFgfj4eFKzZLKkpqTg8/mIr1o1c9uqVUlJzdI2\nNYWq1aplvP9t1SrOPes0Tjn1NJ6b+RLFixc/ZuSMr1qV1CznTUk9vIxZP09KagpVq2VuF2ri4w8d\nm5ScxjHrmHvjWNVrm3X/ltTUTON8JGzesoMqlcpm2lalUjnS09PZvCWzK2rvvjQenfkRABf1ncbI\n6XOoVS2WDX9tzbbvNRtTAahZtVKBZDwaxjE3LLvMKBTatG3HunVr2RxkmicnLaJJ02aUL5/ZNdG6\nTVuWJCdnvD9w4ABLliymffsTAdi8eTOXdr6Q666/kclTpodMwRwtcrZp0451azPLmLToe5pmI2Ob\nNu1IXnzQ537gwAGWLk6mffuTQiLLYWVsexgZmx0qY9u27TLFBQ4cOMCSxcm0P/EkGjRsSGxsbKb9\nP/34I3v37qVN23YFkjH557XUqRFHlcrlDspyQj1++W0TO3fvy9Q2sUlt2p5QL+N9zaqVaNqwBt8s\n/Y26x8Uy5Z4rKVXy4LS8Jg2qA/D7+oK5oo6GccwNc5eFEREZIiJrRGSXiKiIXOttv0BEkkRku4hs\nEJHRWY4bISJ/iMhmERmRZd/zIjJNRCaLSKqI/C0idwftjxORF73jt4vIbBEXnRSRYiIyydu3U0SW\nikgHb19ZEfmviPzlHfeViLQN1Vi0SmxNm3btGXnvMLZt24bqcqZPeYSevfsC0KZlM776ciEAPXv3\n5aVZL/DNN9+wa9cuHpp4PzGlYujQ0WVAjRpxD+3an8jQ4SMOe76iLGdi69a0bdeeEcOHOhmXL2fa\nlEfo2cfJmNi8aYaMvfr05aUXD8r4wITxlIqJ4cJOoc0mO6yM9xyUceqUyfTq3Q+AVs2b8OXCgIz9\nmPXizENk7NjpIooXL85NPXvz4ITxrFu3jpSUFO67dxiXdL2M6tWrF0jGpbqepJ/WMPbWS6hQrjQJ\n9atz67XnMON1J9eSt+7l1MSGADQ/vhajB3QBoGyZUjx6z5XM/XQZqzek8FfqDjqf1YIHBnelbOlS\nHFe1Eg/e2Y25n/3Axr+zt3TyytEwjrlR1AP/EZvxLyKnAoOAk4F1wPnAWyLyBfAmcBvwLNAc+FpE\nFqnqHBG5ABjmtU8ChgAtgPeCuu8B3AFUB64FZojIC6r6B/A8kAY0Aw4A/wGeAy4ArgLO8/rbAvwb\nmCkitT15qgONgL3eeWcAB9NRcsFHzi7ul15+jQH9+9C4Xk0qVKxIz1596NO3Hz4frFyh7Nq5g2I+\n6NDhQkaPG88VV1zBX3/9Rdt27Xnr3bmUK1sGgBf++xzFixfn3UpvZ+p/+n+e5OprrsuruFErZ15+\nVC+98joD+vehQZ3jqOjJ2Ldvf3zAihXKzh078OFkHDvu/kwyvjN7HmXLOBk7d+rAwi8+58CBA6Sl\npRFbwW2fO/99Tj/jzALJ+PKrb3BLv97Ur13Dydi7L337eTKqG8eDMk7IJOPsd+dnyDhy1Bh2bN/O\nSW1bkZaWRseLOjNt+n/yJENik9o57h/35HyG9ryQtR9PYOfuvbz90RK+XfYbiU1qIw1q0FJqsWvP\nPn5cuYHvflhNxzOaM+fxAXyxaCUPzfw4o/+7H36TW687h9Uf3Q/AZ9+vYMoLH+d6fsh9LKNhHI3D\n44tUoEtEOgFPAYmqutnbVkxV00WkErBdVdO97V8BH6rqSBH5D1BNVbt5+0oCfwJTVXWUiDwPnKCq\n7b39ZYGdwNnAL17bpqq63NtfB1gLHAdcBtwMnKGqu7LI9CDQFOiqqmki4gNQ1TwPoN/v90e7aWsY\nxhFxxD/s2T9sKvBN+JIWNaL2xhLJ2mUfA4uBNSLyEbAAeAGnEK4AbheR+jiXXingc++42oAGOlHV\n/SLye5a+fw/av0tEAMoADb3NS7xtAQ4AdYBXcNbLBhH5AJjrbUsHHgfeB9aLyHvAO8Ds/HzgXfv8\n+HyhUeo+H5QpWYzd+9Pzk1xT6IRDzlCvv+EDYkr62LvfT6iGMtQPEz6gVAnYl0bIZAQ4+7qJIesr\noX51/jvhRq4f9hwrVv8Zsn4/fWFoyPoK1zjGFOBOWqyI20oRUzKquhfoIiKtgIuBAcCdInIHzoV1\nNfC2p0S+CDo0hkPlzhpbyr5wEQRmldVS1cNNhDjZc+V1AcYA/UXkDFVdLSLNgH95+57EueIuz+2z\nBvCTr2zLHAl8YL8f0qNYyYRDznAZg35Ce+MJB6GWccny9SHszbFi9Z8h7Tcc1ySarnVRd25EMiZT\nEiijqkuBpSIyAefOag6oqr7mtSuNc1N96R26EWd1BPqJ4aCFkhurcQqoJfBJkBxVVXWjd65iqvoV\n8JWIjMO511qJiAL7VPUj4CMRmQysFpEqOSgswzCMY5pIusvuBC4SkatUdT1OkcThlEBtL1ayD5iA\nUyyB+hQLgGdE5ETgB2AEecySU9WtIvIK8ICIXAakAGOBjiLSHJgCVBGRvt6+tl7fa3HJCCtE5B6c\nS+9Ur82Wgg2DYRjHMr4i7i6LZArzZJySWCwiO4HXcBlbD+MUyc/A18A8YBzQVUQe8No9CszBZaXt\nBb7Jx3kHAr8CP+GUVzPgEi+APxQXn1kJbMcpnR6q+jfQC2gMbMAplgHApYHkBMMwjCPB5yv4K5qJ\nWHbZsciOvaGLnhTzQdlSxdi1Lz26YzJhkLN4sdAH1UuX9LEnygP/MSVgb4gD1rHtB4Ssr8Qmtfn6\n5aGc0mNiSGMyW76fHrK+wjWOpUscuTny3k9/F1iUC0+oGrWqxmb8G4ZhGGHDll82DMOIINHu7ioo\npmQMwzAiiCkZwzAMI2xYdplhGIZhHCFmyRiGYUSQECdLRh2mZAzDMCJIUXeXmZIxDMOIIEU98G8x\nGcMwDCNsmCVjGIYRQcxdZhiGYYQNC/wbhmEYYaOoWzIWkzEMwzDChlVhLkT2pIWu8Gu4qsmuS9kV\nwt4gpoSPulXKsDZlN3vTQiNpfIWYkPQToLgPKpYpzrbdBzgQosEsU6p4aDryCNf1Tg9hCW+31LaP\n3fv9IV0SvPq/Z4asr1b141g4sQunD53D0tWpIet3+yvXH7E5snDllgKP1unHx0atOWTuMsMwjAgS\ntdohRJiSMQzDiCDFivhEGYvJGIZhGGHDLBnDMIwIUrTtGFMyhmEYkaWIaxlTMoZhGBHE5skYhmEY\nxhFiloxhGEYEKeLJZaZkDMMwIkkR1zGmZAzDMCJKEdcyFpOJItasWUPXiy+iVvUqJDSqx/BhQ0hP\nT8+27WPTpiIiVI2ryDlnnU5yUlLGvj179jCwf18a1a9N7Rrx9LjyclJSUkIm54Z1a+l1zWWc2LQO\nZ7dtwkNj7z2snDt37ODaa6+lXnxZVq3Uw/b53xmPkVCjHOvXrgmJjOvWruHKy7rQqE41WjZpyKh7\nhx5Wxicen4aIULt6LB3PO5Mli5OybTd/7rvElSvBws8/DYmMR8P1XrtmDZdd0pk6x8XT5Pj63HvP\n4WV8fLqTsXqVSpz3rzNYnJx5HFf9+iunn9KeBnWPC4lsAerEl+ONu89lzYwr+WlaN8Zc3SZbF5TP\nBzedlwDAB6M78vUDXbjslPqZ2jSsXoHPxl/Er09cEVIZj2VMyUQRPbpfRs2atfh5xW/Mf+8j3p39\nNtOmPHpIu3lz5zB2zEhmzpzJ2g1/0qlzF7pd2pmdO3cCMHLEcJKTk/j0i69Z9vMK/H4/fXreGDI5\nB9zcg+rH1eTjb3/k+dfn8uGCOTz/1PRD2v256Q86n3saxYvnXMfrz01/8Mx/poRMPoB/9+jOcTVr\nsfjHlbw9933mzZnNf6Yfeo735s/h/nGjmTlzJitXb+TCjp3p0e2SjLEMsHPnToYPuYNy5cqFTMaj\n4Xr3uLIbNWvW5Mflq5g7/0PmzH6H6VMPlXH+3DmMGzOKmTNnsnr9Jjpe1JluXbtkyPjpJ/+jw/ln\nU69e/ZDIFcyswf9i45ZdtLj1LS4e/wFd2tfllo7NDml383lCl/Z1Abhw1AJGv7qYp285gxPqxgJw\n5gk1WDDyQtZu3hFyGXPCF4J/0YwpmSghadEili1byrgJD1CpUiUaH388tw4azLPPPHVI22dmPMm/\nr7+Rk046iTJlyjD4jrvA52Pe3DmkpaXx3+eeYdjwEdSpU4e4uDhGjxnP/Hlz2bhxY4Hl/GFJMst/\n+oG77h1LhYqVqN+wMTf2GcirLz53SNvUlL8ZNnIco0ePzrHP8ffeRY9/9yywbAEWJy/ixx+WMmrs\nBCpWqkSjxsfTf+BtzHzu6UPaPv/MDK697vqMsRx4+534fD7emz83U7sHxo/mzLPPIa5KfEhkPBqu\nd3LSIn5YtpSx9x+UceCg23numRmHyvj0U1x3/Q0ZMt4++C58Ph/z580BIDU1hbnzP+TCThcVSKas\ntG5YhRb1YrnvpSS27d7Pqk3bmTbvZ248NyHbtsu8opjpfngveT2pO/bS3FMyceVjuHjcB7yXvD6k\nMuaGz1fwV34RkXoiMk9EUkRkjYg8ICLZ6gMR6SsiKiI7RGSJiFySn3OZkokSFicnUa9+fWJjYzO2\nJbZuwwpVtm/ffkjbxNZtMt4XK1aMVq0SSVr0Pb+tWsXWrVsz7ZcmTShTpswh7osj4adli6lVpx6V\nKh+Us1mLRH7/dQU7dmSWs+kJLbmgU5cc+/vs4/fRX37k5n6DCixbgKWLk6lbrz6Vg8ayZWJrVq44\ndCyXLk6mVWLmsWzeshWLk77P2Pbzjz/w2suzuG/0+JDJeDRc78XJSdSrl42M2Yzj4sVJJGYZx5Yt\nnYwAl3XrTpOmTQskT3YkNqjCmr938M/OfRnblv6eSkKtSpQvnTnk/H7yelo3dA8JJYr76NS2DmVK\nFWfhL5sAeOfbNejGrSGXMTd8IXgdAW8BG4CGwHlAV+C2rI1EpBswEbgJiAWmAa+JSMO8nsiUTJSQ\nmppC5aAbN0BcXBwAKZs3Z9qekpKS6YcPEBsbR8rmzRm++Kz7K8fGHtLPkbBlSyoVK1U+pG+ALfmM\nA+zZvZsx99zBffdPolRM6Mr3ZzeWsbFuLFNTNmfTtvIhbVO9z+L3+xk8qD/D7htNlfjQWDGHkzHa\nrndqakomRR04b3YypqZk0zYuLqSxwOyoUiEmk4IB2LJjr7evdKbt736/ltnfuZjfp+M78+zAM+j3\nxJdsCPHyFtGOiLQDWgFDVHWrqq4EJgO9s2leBhimql+q6n5VfQbYDpyc1/Md09llInIm8AFQSVX3\nRlqe/CzCkds6QGFdJyhEfT/+6AM0b9WG0846NyT9BZOfz59T25nPP0N6ejr/vuHmUIiV9cT5aBqZ\n6x2qcQwneX2Sv+qMhnRsUxuAc+6dR1yFGJ4ZeCbrN+8k+bfwKsMcKfyQSltgtapuCdqWDIiIVFDV\nDDNVVV8MPlBECHa+9AAAIABJREFUKgMVcFZQnjimLRlV/VxVS0eDgomPr0pKauYvekpKCj6fj/iq\nVTO3rVr1kCfE1NQUqlarRlWvbdb9W1JTqVqtWoHljKsSzz9bMi/29M+WVHw+H3H5eNJftVJ57cXn\nuGfMAwWWKStV4quSmnro+Ph8PqrEZxnL+KqkpqYe0ja+alU2//03E8aMZNKjj+EL8Yy5o+F6x8dX\nzbDogs97OBkPaZuSkiFfuNi8bQ9xWRaxi6sQQ3q6n83b9mTa3qdDU2Z/6yyZfWnpvL94A5//uImr\nzmgUVhlzIwKB/yrAlizbAj+Cw/6IRcQHzAC+VdXP8nqyY1rJRBNt2rZj3dq1bA5yQyQt+p6mzZpR\nvnz5TG3btm2Xyd9+4MABlixOpv2JJ9GgYUNiY2Mz7f/pxx/Zu3cvbdq2K7CcLVq1ZuOGdZncTssW\nJ9E4oQnlypXP4cjMzJ/9Jtu3b6PLOSdxYrO6nNjMZf10veA0ZkyfXCAZW7duy/p1azO5dBYnLUKa\nHDqWiW3asmRxcsb7AwcOsGzJYtq2P4kP319AamoKXbt0oHHd6jSuW50N69dxzZWXMeSOgsWQjobr\n3bptO9atO1TGJk0PlbFNm3YsXpxFxiVOxnCS/FsKdeLLUSVI0bRpFM/yDf+wc29aprbFi/koVizz\nDblUycjfAiMR+Cef9pOIlAReBE4Auufn2MiPMCAiQ7wMh11eFsO1InK2iPhFpHRQu1dE5Hnv7xtE\n5EcRmSQiO0Wkpoh8KiL3i8ir3rZ1ItI16PjVIjJcRH4Tkf9kPUd2cgQd20pEPhaRf0TkbxGZ4g18\nSEhs3Zq27doz4p6hbNu2DV2+nKlTJtOrdz8AWjVvwpcLFwLQq08/Zr04k2+++YZdu3bxwITxlIqJ\noWOniyhevDg39ezNgxPGs27dOlJSUrjv3mFc0vUyqlevXmA5m7VIpEViWx4efx87tm9j1UrluSen\n0eP6XgB0OL01i779Ktd+buwzgI+/+ZF3P/o64wUwY9Zb9Li+YJlmLRNb07ptO0bfdw/btm1jhS7n\n8WmPclOvPgCc1PoEvvnKjeWNPfvwyksvZIzlpAfvp1RMDBdc2IlLLrucxT//ymdfJ2W8ahxXkymP\nPcWwe0cVSMaj4XonJjoZ77v3oIzTpjxCr959AWjdoilffelk7Nm7Ly+9eHAcH5w4npiYGC7sGNps\nsqwsW51K8qoURvdoQ4UyJUmoWZEBnZrx9IduTlbSpEs5RZxFNz9pXUYKc/FiPs5pWZOzmx/H3EVr\nwypjFPI3zpoJpgpuZe+/szYWkTLAPKAecIaq/pmfk0U8JiMipwKDcIGkdcD5uMyHm/JweE1gN1BZ\nVfeLCEBf4Drv1Rt4VURqqmrgcawHcAGwCjgrNzlE5ANgB/AeMBXoCNQCZgN3Affn9bPm9ujw8qtv\ncEu/3tSvXYOKFSvSs3df+vbrjw9YocqunTvwAR06XMjYcRO44oor+Ouvv2jbrj2z351P2TJlABg5\nagw7tm/npLatSEtLo+NFnZk2/T95enSJKZF7qyefn8XQwQM4tWVDKlSowDXX9+SmXn3w+Xz8/usK\n9u/ZSUwJH1MnTWT65AcyfPWXnHsyPp+PAYOHcOsdQ6kSW+mQvmseVyPb7cEUz8MHeWHWawwa0Jem\nDWtRoUJFburZm959+uHzwcoVyu6dOyjuc2M5asz4jLFs07Ydb7w9h/Jl3VhWKFc2U78lihenWtV4\nqsTFZnfaDPIy1tFwvXN7Cp71yusM6N+HhnWPo0LFivTs1Yc+/frj88GKFcrOnTvw+aDDhRcyZtz9\nmWR8e/Y8ynrj2KVTBxZ+8TkHDhwgLS2NuIpu+5z573P6GWfmKEOr+nE57r//9SXcfVlLfnvySnbu\n3c/sb9fw3Yq/aVU/joRalWhRL5Zde9P4YPF6WtaLpcuJ9Xh/VEc2pu7kobeXsXXnPlrVj2PyTSfT\nqkEcxYv5KFG8GJtfcM+Yg5/9hqW/p+YoQ0GIwCyXRUBdEYkPui+2B35W1UyThDwX2SvAfuA8Vc3s\ng8wDvkgF6wKISCfgKSAx8IG9fO0zgU+AMoEPJiKvAHtU9QYRuQF4BohV1W3e/k+Bnap6UVA/fwGD\nVHWWiKwGXlfVu7z9ZwfOAZyTnRyqmi4i3YFpqlojSO7rgHtUNc95mX4//qJeDM8wjlGO+Je9dN32\nAt+EW9WpkF/31zfAj8Bg3MP6fGCSqj4mIsuBnqq6UESuAUYDLVX1iNLwIm7JAB8Di4E1IvIRsAB4\nIY/HbgkomCAyapd4CmIdbhADHK5uyeHk2Ak0AqqJSLAW9wH5ShjYdyA/rXPGB5QqAfvSnI0bKv7c\nujuEvUHJ4j6Oq1yaP/7Zw/4DoZG0ctlSIeknQDEflC9dnB17DpAeosGMKZlzlYP8Eq7rnR7Ch0wf\nULqkjz37/SGV8fwRc3NvlEcSalbi2VvP5Kapn7MihHNiFk7MeT5YTkRoxv7luIfqTcA24AngcW+f\nAIGg201AfSDV8xQFeEFVe+XlRBFXMl5mVxcRaQVcDAwA7gTuyKZ51l9uWh7a+Mj8u8zumMPKISJt\ncS65n1S1RS4fJ0fCYTP6Q9zv3rTwWLb7D/hD1neIdNUhpPtD13e4/AOhvt4hdWR490p/iPtdujr0\nrqoVG7eGpd+jBVVdD3Q6zD5f0N8Fnl8QcSXjBc/LqOpSYKmITAB+AZp7TcoCAQuiEbAsly4z8hE9\nd1kdINc6ETnIcR4uftNQRMoHfJYiUgXYF5xTbhiGkV+Kugs9GrLL7gTmi0ht731TIA74CDgAXC4i\nJUTkepzCyI1TReQ8ESmFs0bK4yZcHqkcq4D3cVkXD4tIRRGpAbwOhH6Sh2EYxxQRKitTaESDkpkM\n/AAsFpGdwGu4cgffAkOAccBmIBF4NQ/9vYjLKtsC3A10V9W82MWHk2OJqu4HLsEpnk3AEmAlTjEZ\nhmEcOUVcy0TcXebFQvp5r6z7JgGTDnPc88Dz2ezaparZLgahqvWzvP+UzJcoWzm8tksJSnk2DMMw\ncifiSsYwDONYJtrXgykopmQMwzAiSFEP/BcpJaOqZ0daBsMwjPxQxHVMVAT+DcMwjCJKkbJkDMMw\njjqKuCljSsYwDCOCWODfMAzDCBtFPfBvMRnDMAwjbJglYxiGEUGKuCFjSsYwDCOiFHEtY0rGMAwj\nghT1wL/FZAzDMIywYZaMYRhGBCnq2WU+f0iXxjMMwzDyw6q/dhf4JtyoWpmoVVVmyRiGYUSSqFUP\nocFiMoZhGEbYMEvGMAwjghT17DJTMoZhGBGkqAf+TckYhmFEkCKuYywmYxiGYYQPs2QMwzAiSRE3\nZUzJGIZhRBAL/BuGYRhho6gH/i0mYxiGYYQNUzKGAYhIEX+eNKIVXwhe0YwpGaPQCL6Ri0hUffdU\n1Q8gIqUiLYtxbOHzFfwVzViBzCKOiJRQ1bRIyxFARM4BlqvqxkjLkhURuQZoBQwF/AHFYxjhZP2W\nfQX+ntWOLRW1qiaqniaN0CIiQ4DLIy1HABFpAzwN1I60LFnxLKu2wCmqmq6q/qPFhXa0yJmVLJZt\n1CUhRZu1fbRig1hEEZHmQBfgvUjLAiAiLYBbgI9U9btI31Sy3phVNR24F6grIsO8bVFpyQRkF5Fa\nIhIPVM26L9oREZ+nyDuJyHPAByKSEC3yi0gx7zuBiLQQkQYiUjcc5yrq7jJTMkUQEbkSWAbsBtIj\nLE6Ak4A2QEcRaaGqaZG8oQTFYGoGnlhVdRcwBmgpItUjJVtOBN2cLwfmA58Cz4rIrRC9ijEr3me4\nDHgFWA38V1VXBF2XiH03vDEOKJgHgVdxY/2ciPQL9fks8G8cdajqq8CbwLnAuSJSvLBlyMZSeBoY\nB6wHRopI00i7pESkJ87Se0JEjvM2/w8XlznVaxNVv2FvzM4DZgAjgK5AEvCo98QdVfIeDhGpA9wH\ndFfV0cCrInKciFwjIidH8rsRpOjuBK4AOgD/Av4EpotItVC60sySMY4aRCRWRGoCqGp34B3gSeDM\nwvQvBz1tnyUid4rICBGJUdU3gSlALDBCRJoU5s0kmzF4A3gCaAgsEZEHgOI4a2a4iFSLJssgaJzO\nA55Q1XeB/UAfYKSq/hAx4fLPNmATUMJz+U0G3gJGAQtF5MooGPuWwHBVXQeciXM/d1LVv4CyoTqJ\nLwT/ohlTMkUEERkDzAH+JyKzRKSOql4GfAO8RCEqmiB3zhygCXC7J1cTVX0FF/yvDtwjIicUxs0k\ni4+9u+f26Kaqj6vqeTiroBaQDFwClAKaBo4Nt3x5pIb3fyOglIiUBBYBM1R1rIhUACaLSJNos2iC\n4kilRaSyqm7FWbVDce6yeNznOB53LboXpgWedbxEpAzOoq0oIqcDzwNXqer7XnxxWqgtmqKKpTAX\nAUTkXqA30Av4FReP+Ri4xLvhzwFaAz2B98N9UxeResD7wC2q+rGXhLAMFz/opaqrPH/8UGCp125f\nOGUKkm0yzgWyGDgZWAtcq6q/eDftk3FK8Sxgmar+qzDkyg0RaQlMAK7GuW5mABVxFsxEr80pwDTg\nXO8mHhUEWbaXAP2AZkBnYCcQB9RQ1TlB7cfirN2BEXgAiQVKqupfItIXGO/JcrqqfuW1uRzo6z2c\nFJhN2/YX+DPWqFgyqh4qgjEtfBQjIj4vltAB91T+PtAAV5Pu0cAPVFW74J4a+xeSCyIeqOYpGAE+\nAh7EPYm/ICKNcErnKWBMuBSMiLwQHMD3bg6dgMbemByPS4x4S0Tqqep+Vf0CuA43pn4ROSMcsh0B\n5YAzcArkHeA1nLtpUVCbljj3WVT9rj0FcykwC+cS66Wqy1R1lap+r6pzRKSYOAYA/XHuwMJQMMFB\n/nHA28AvIlIbF597C/gNOBB02EXAX6GyFot64N8smaMcESmL+zHcgnOjPI8Lps7zzPxOqnqP1zbj\niS3MMlXHWVVTgM+A91T1HhE5FfgCWIW7uZ+iqlvCJEN54CFV7Re0rS9woapeKiKlVHWfd6NYAvyh\nqhcGtS0LLABmqepT4ZAxv4jICOAqnJVVCReL6Ydz8W3xtp+rqskREzIbRCQOpxSnqepsz61XD7gY\nWIOLHU7GWThVcJZl2D9DwMLy/p4IXOvJVEJVv/O21wduxSm+Jbh7ejmgtaruD+7jSPlre8EtmWoV\noteSiboJUEbeEJHzgY3AClyq8gygMc6iWeA1a4WLiQBuLkgoFE2Q+8Pn9esXkYZASWCfqv4OjBM3\n+bIM8Jh3aDowFfgOWBQuBePJtAN3A8Z7Op7vyddYREp6CiZGVfeKSH/gFRFpAqh3/C4RWY27mUcE\n7wa3X1U3eJtm47LezlLVN71EhbnApbjMp6GqujwiwuaAqqaKyG5cangyMBGoi7tZ1weOA8birLB0\nVf07nPKIyKfAEFX91nvfCDgNaOO5yep5GXxXAu/iXJCv48b+H1y6dZpEWTWNaMWUzFGIiJyAezKc\nBQwDbgQ+AX5U1QVBiuRsXMwhgxBZMjWAP7z+/CLSHbgf51LYICIrVbUvsBeXrXUF8AjQDtiuqi+H\nQIY8ISKlgYHAzcBNwGDgZeByVd3rNfsDWOfJFniybQecCDxQWLIGyVwcVxVhEfCViLyvqo+p6jIR\n+QWX+vumqqYAn3uvqCG7hxBcjO5a4C6cYnzUU5QdgeE499iOQpCtLPBGQMF47AKqAZeIyAbcw0lV\n3Pd3KnC/qs4Avg7qp3ioFEy0Z4cVFHOXHWWIyAQgDXfjboC7YQ7BuRk+wbmi/sA9tTcEWoXyaUtE\n7gdOVdWzvfdn4fzWV+NcYdfjLJczcO6FKbhg+n7cU+u/VHVJqOTJo8yxOEsGnJJ5HlgO3INTLg/i\nxrJjkH++NFBZVTcVopyZXC8iciJwPnAnsBCXjv4p7gHjE1V9qLBkyytBCuZCXEmjEjjZZ+EC6HWC\nb/Aichfu+3F1kNIvLFnvA37ylN1gXGJMVZxiWaiqn4jIUECAm8IVI/p7R1qB+61avkTUaipTMkcR\nItILl2HUEpen3xrnJnsTGIRLu+2HSw/eAdznmfXFVfVA9r3m6/wVce6DL1V1uHcjvgw4TVVvEZEG\nuJTp6ao61jumJHABUAd3Y9SCynGEssfinqbTgLuB/wAVgK24ORvneD72wOz/QquUkMXteB4usFwJ\neE1V3xORWrgsp1q4G3Uy7lr3UtX9hSVnXvGyyGbhxng9zop9GhjguSlL4jLkmuOssrNUdWkhyBWc\nRdYQGADcBpzvJanUBGI8d2/gmFnABlW9O1xybQ6Bkok3JWOEAhF5CKivqt2DnhjPx/nqXwOGqeof\nXtvA/pAoGK/P0sB0oCYudnEOzqVTGzfzfDXwpKqO8ALvU4DHoiUQHaRo/DiLqywuE2uxqh6ItI9d\nRLoCL3iv6rhYy2RVvVNEYnDB8v4496gPaOC5zKICT1nG4wL5j6jqG17Qfx0u1fphr90wXJZfcVz6\n+uJCkC1YwYzEje8IXL26gcClqjrX2y9AC9z4J+IF+cMlW1FXMlGV6mjkynpc4Lp9UHryh7in3O64\nyY2Ngg8IlYLx+tqDy7Sp5v3/CnAHLqttBzBVVUd4zavifqBhC+7nFy/RoAPue/8KkKqqizwFEzIf\ne14QkaEiMi3ofS3cTe9yVe2nbiJtV+BWEZmgqnvV1fa6DedCaxZNCgYyYi+7cEo82fsursQpyodF\nJM6z1CbhHlAuLAwF48kWUDAn4yyoh73xG4ML7L8jIp285jfhrJySHMwiC9vEUCsrY0QUETld3Azu\nSjhXVRrQ08uECqC4YOqVOL9yOAsl+nGunCRc0cvuwLPAT7jJdQHOxfnjd2btIJIEKZpdwDwRCVQw\nLmyTfjVwi4iM997vwyVObBU3/6m4qs7GXdMhInJB4EBV/U5V1xeyvNmSzVyRGJw1czcuff2xoAeP\nXsDdqrpP3ZykbYUpp5cw8zEHXXWB78NoDiqaM1V1CO77e5WnYEqE8mEtK1ZWxogY4manv4ObBzMF\nFzwfiPuRDBKRDl7TDrgbfS/cDal5uGRS1d3ACZ4MpXBujx24oPTtIvKbiHyAy8q6SV2dp6hCVVNx\ncY99eOnVhRmD8c73Ci6edZeXilwGZyG28h4QfN7N7W1cQkf7wpQvLwS5ZE8WkVtE5BRvbG8HrgF+\nV9X7gg6phku5Lyz5Mu5vqupX1Z9wGW57ga4ikuDt+wdXM2068KmInKaqBwIZcuG2cIu6JWMxmShF\nRFrhgqcXAB1xFsNunBlfC5cRJcAenHXTAqeEvgD6aCEUSxSRxjjltw+XfPAJLqNoB/CZqv4abhkK\ngog8gpsfcba6Mv+RkOEyXCn5YbhrORW4OBAf8Nq8jhvP6ZGQMSuSeRLjpbhCo4txi77dDTyDm9T4\nnPf3D7gkiztx5Vl+KgQZg2MwZwGVga9U9W8RuQj3UPQGLknlV69dLK7aw+OF6TrdsutAgW/CsWWL\nR62qMSUThXjplLWBn9WVyA9k7PTHuXlux2VEJeKC1+97Zv1AXLbMaYWVeutl6UzDBaIDlYGjHi/D\naRgwuzAym3KRpRsuRnQPTkE/hku1Xop7kLgPODPSSltEyqnqzsANXFyNujHAK+rmZ/XHWQPDcHGX\nk3GTLH04V+AdhZG+nkUJPoJL8tiCS/MfpaqPBima13AuvVVZ+ii0JBBTMkah482FuRtX0uTfQdsD\nBQZ34EqmBGYsv4AryXEc0Lmws7m81OWZuFnnN6rq9sI8/5ESysy7ghKkaIbjLNY7gL9wN+hehXFz\nzgkRuRlXUWKKqm7yLO1BuAKXPYA9nnupDy51eYSqjheRUjhLu4yqFmp8zotjDQf+jZuUPBI3v+wl\nVR3nWWGP4uI09+nBygqFyj+7C65kKpcxJWPkARFJxM3bKI0LPE7F1XF6KajNxbgspA/V1QOrhJvZ\nXxL4XlXXFLrgZFReTle39oZxBAQpmrtxLrTdkBGcjihe3KgTTr4ncRN9J+BWO71ZVd8KatsbZ9E8\nCDxQWA8dWVxkPYALgfWqOjyozRBczbdhqvqqiNyIc+11K+y4XICtu9MLfBOuVKaYKRkjZ8Qt83oJ\nLgD8BW4mfxPcD/UaDSrFIq4y8JeR+lEY4cObK/MmMFZVR0ZanmDEzX7vhstyfAinaB7AuWwfUtX3\ngtoOxGVtHV8YqdZZXGRVcVmWt+Os64tUdW1Q2xm4ZJnGGlQBXAqpgGxWirqSseyyKEBE7sYF9s/A\nZeXUxwX4FVcy5gXvyQwAVf3C84kX+rLKRnjxsskuxlkMUYEXvwLnEl2Jmwx6F27tohG4+ODt4krJ\nAKCq04CGEVAwA4F3VHUCLh5UDBjoWdoBnsItk5Cp+GmkHtqKenaZKZnooB1uXZW/cGVDTsE9HfbB\nZe30BWaJK5WfQbTEE4zQoqpzVfWXSMsRwEsq6YFbF+g3nHVwO+4BaDnOYtkFDPDcuQEKZeG0IAVz\nIy7Z4CFv+zTgv7iJn0NFpLV3yBBcRfDNhSFfbth6MkZYEVcV9mvcDPqyuGyXi3BPhwtwk/ZK4L5L\nJxdmaqVhAIhbwOt9XBzjXW/bGJx79w1c6f4TcGvCpALXF0aQXw5d0fIxXLr/iOB0bxG5E5eZWRG3\nKFkl4Dp1yzxExEUWzPa9BXeXVYgxd5lxGLz5Geeq6me4iWIPqurnXjbR97iCk7/jKRgRseUZjMKm\nJC6LLMMy8SZZzsUlKdyFc+0OAm4rrCyyIAVzCu6hbATwAXC/uKUaAu0exqVU/+m97vQUTKlIK5hj\nAVMyUYCqbhZXfLIpLpc/wDpgkqpeobZIkhE5NgDfAteISLXARi9r61eclTAAt55RoZa7EZFzcBWe\nh+Cs/jtxVtebItI2SNbHcJObOwC9RKS+hmnZ7/xiZWWMQkFd8ckXgd4iMlxE3sLVV9oQ1MYUjBFW\nArXIRKSxiLQTkerezfh13FysW8Qtrx3gM5yL9xUNX728YPmy3rO+xymVDjiLaqP3/7c4RdMm0FBV\n78elhvcArouWxJmiHvi3mEwUIa48fm9cTas/OVigr8DriBtGbgTVIuuOS00ugXvImaKqr4jIHcBV\nwM+4TLOWwA04d2+hBtGDJ9J6v5tRuKD/XFzaf13cPJ6uuGXIVwa51wYB72rQujGRZNe+gt+Ey5aK\nXlVjSiYKEbd2yD7vB28uMqPQEJEzcUVZe6jq+yLyLm7BuQdV9WURuQ5X36sxLkbTS1UXFYJcwWnK\nN+KWt2ikrmBrQNGMxGWSzcLN5G+AU4Kj1C3nEPEgf3ZEQsl4Kd2P4xTzDlzK/LDsxkdEbgVuwVUU\nWQYMUtWkvJ7LlEwUYxaMUdiIyG1Agqr2F5F4XED/D1z1gYe9WfJVcKViihVGNYIsWWQVcRbURKA8\ncEqQoimFmyjaFOd6HhF0XNSUEMrKrv0hUDIl861kknDLddyFq449D1d7cHKWdl1wVuuFOAVzK64+\nYuO8JnhYTCaKMQVjhJugGExJL0ZRBdjnzZpfhpvJ3xx3rxguIh/jCnbuKCQF4wtSFGNwsaFXcTXJ\nfMB33jQAvNjRq7hszP0ErREUrQoGCj/w72XetQKGqOpWVV2JSz/vnU3zPsBzqvqtp8wfwo1rl7ye\nz5SMYRyjBMVguuCe/JNxy2ffhqtA8ZWqTvSa/8/b/yfwdGHdtINcZA/iVqwcBVzhpfxfiauu/JWI\n1PAOaYVTNKO9zxa1sYoAEQj8twVWZ3lISMatPF0hm7YZBXc9hb+EfKxvZHMuDOMYxbsJd8PNir8N\nt2REIAW5BtA+yGW7C5cq/E1hxwjFLS52Om5BtxQRqeNVv+iBiyX0BVaKyK9AOdw8GP/R4m4uXaLQ\nc5CrcOiy6Kne//HA9jy0jc/ryUzJGMYxiucSG4SrQPy+iJT1AsKX4lxOvwDzRWQRbh7MMxFKQtmF\nixucKSK7cC6cergVLjvgStxUx8VoHvfmlEVtDCZKyI9iK5ASNCVjGMcu+3FVvyt4Cmc8bi5MLdxq\np/NwFs2pwL+CKxkXMhuB53Fp1fG4JTAeV9WPRGQScIGqDgo0NgWTK3+TedI33nu/ty8vbX/M68ks\nJmMYxyjq1rZ/G3gYWIGr7fWEqjbALZucoKpX40rlR2zRNC8OMBE4HzhRVUep6kfe7hpkdu9EdZA/\nSlgE1PWyBwO0x63EuyObthmVE7zkkDa4ya55wiwZwzi2mQTMAaqq6v+yBMo3e2X+90ZGtIN4bro1\nACJyPNAItyxGS9zyykYeUdXFIvI9MFHcUu81cct9TwIQkeVAT1VdiFvl9BUReQmXbXgn7vswL6/n\nMyVjGMcwqroX+AHAq5/XXEROBoYBZ6nq/kjKlxWvQGw/3JP330Abi8EcEZdzcF2dbcATuMmZAIKL\nb6Gq74nIMFzpoGq4Mj6dAnOT8oJNxjQMAwDvZnIFcADorarJuRwSEbx5MWnAfquKEf2YkjEMAwAR\nqQwUB/yqmppb+2jgaElTPpYxJWMYhmGEDcsuMwzDMMKGKRnDMAwjbJiSMQzDMMKGKRnDMAwjbJiS\nMQzDMMKGKRnDMAwjbJiSMQzDMMKGKRnjmEdEmoiIX0TO9t5/ICIzC1mGTSIy6jD7zvbka5LHvm7w\n2pcugDwF7sMwwGqXGVGIiHwKnIErRQ9uPYudwIfAfaqq4Ty/ql6Q17YiUhu4UFWfDqNIhnHUYpaM\nEa28rqqlvVcMkAiUBBaKSKUIyxZMV6BnpIUwjGjFLBnjqEBV14rIIGAtbhGtBSKyGreYVQfc2idV\nRKQYMBS4Drd64mbc8sKjAlV6ReRS4H6gPm7xpcnB5/IsqU2qepX3/jyv/Qm4yr/PAmOBB3El0n0i\nsgfooqofikhXT4amOGtsPjBYVf/2+msKPAm0Bv4C7s3PWIhIdU/mjkAMsBq4X1VnZWl6log8DDQE\nVgK3qeqnXh+lgXE4JVkTWA9MU9Wp+ZHFMHLDLBnjaCLwUBRcfv5m3E26qvd+JG7N92tx5cq7AjcC\nowBEpC4yNWxyAAADu0lEQVTwBvAyEItTRoMPd0IRaQ7MxSmFWKALbiniO1X1LuAF4DvP4vpQRM71\n+n4EiANa4W7ib3n9+XALhW0D6uBK1l8MVM7HODyNUxyNcQuNTQNmikizLO0GARd5Y7MQmBu0UNWT\nwHlAJ2+c+gLjReTmfMhhGLlilowR9Xg35nq4ZXdXAF8G7U5W1Y+9dsVwCmCUqiZ5+5NE5FHgVmAE\nrpT9dmCCVx5eReQR4KXDnP5mYIWqPuO9/0FELgfSD9N+ADBPVV/x3q8XkSHA9yLSEKd4BLjWW5kS\nEbkDuCqPw4H3GUqo6nbv+Odwi0udCPwc1G5iYMlkERkJ9Ac6iMgCnBK+NCi+9bGI/Be3ANgzGEaI\nMCVjRCvdPbdWgE3AZ8D5WRZMWhX0d1XcTXySiDwUtN2Hc2mVAuoC67KsP5LTeuXHA78Fb1DVz3No\n3wQ43nOfBXMAaAAE4kkZfarqRhHJT2n9Jjir40SgAm5tdoCsmWBLg86RIiJbcJ//eJwX4w0RCS7D\n7sONs2GEDFMyRrTyeiAmkgv7gv4OKJ9rVPX17BofJiU3J7fxgVz2Z2U38KSq3nKY8199mOPydA4R\nqQh8BPwPSFTV9d6669kt2pXV2vIBezg4Tqer6vd5Oa9hHCkWkzGKDKq6Dfck3jZ4u4hUF5Fy3tt1\nQB1vGd8ArXLodgUugB/c37kicuXhxMjm/GVF5Lig84OzagL765L3mEwznLX2kKqu97adfJi2JwSd\nozouprQWZ/2lZSNnbRGJyaMchpEnzJIxihqTgXu9DLGPgEa49cm/AfoAs4HRwJ1eLKYRLkB+OGYA\nt3lxk8dwGWnPe3+Dm79TU0Sq4CyER4BvvPZP4FxYU4HWXhLBt8Afnow34NKyJ+MsjLywGqcgzhSR\nZFziwBDgH5wrLJihItIPl2QwxmvznqruFJGngBEishhYhMt0exu3zvuEPMpiGLliloxR1JgMTMLd\n4Hfh4jgf4ikSVV0GXA3cAGwBZuIy0rJFVVcA5+Ky0LYA7+FSmB/0mryAe1hbjwukf4cLzF8HpAC/\nA6WAjqqarqr7cKnHNYCNwHfAOxy0cHJEVTcBtwC345TGWOA27/PeLiLjvKbpwHTgY1zadXugs6ru\n9PbfCbzunXs38CYueeCBvMhhGHnFll82DMMwwoZZMoZhGEbYMCVjGIZhhA1TMoZhGEbYMCVjGIZh\nhA1TMoZhGEbYMCVjGIZhhA1TMoZhGEbYMCVjGIZhhA1TMoZhGEbYMCVjGIZhhA1TMoZhGEbY+D8s\noZ4DLGFKzwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f118e171860>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "YEjGb1_sGFRy"
      },
      "cell_type": "markdown",
      "source": [
        "## Final Words\n",
        "You have learned how to perform neural-based emotion recognition using RNNs. There are many things you can do after you have completed this tutorial. You can attempt the exercises outlined in the \"Outline\" section of this notebook. You can also try other types of neural architectures such as LSTMs, Bi-LSTMS, attentions models, and CNNs. In addition, you can also store the models and conduct transfer learning to other emotion-related tasks. \n"
      ]
    },
    {
      "metadata": {
        "id": "gSYXLGiZGFRz"
      },
      "cell_type": "markdown",
      "source": [
        "---"
      ]
    },
    {
      "metadata": {
        "id": "QATOWiPRGFRz"
      },
      "cell_type": "markdown",
      "source": [
        "## References\n",
        "\n",
        "- [Introduction to what is a Tensor](https://www.youtube.com/watch?v=hCSjWCVrphc&t=1137s)\n",
        "- [Deep Learning for NLP](https://docs.google.com/presentation/d/1cf2H1qMvP1rdKUF5000ifOIRv1_b0bvj0ZTVL7-RaVE/edit?usp=sharing)\n",
        "- [Enable Eager Execution on TensorFlow](https://colab.research.google.com/github/zaidalyafeai/Notebooks/blob/master/Eager_Execution_Gradient_.ipynb)\n",
        "- [Basic Text Classification](https://www.tensorflow.org/tutorials/keras/basic_text_classification)\n",
        "- [Deep Learning for NLP: An Overview of Recent Trends](https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-trends-d0d8f40a776d)"
      ]
    }
  ]
}