Text Classification - FYP-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification Engine for Sensor Fusion\n",
    "\n",
    "First, let us load and preprocess text transcriptions.\n",
    "\n",
    "**Note**: Running this python script require `nltk` libraries to be set up in prior. Therefore, please download the following dependencies by uncommenting the line `nltk.download()`.\n",
    "\n",
    "1. `perluniprops`\n",
    "2. `punkt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "# from sklearn.externals import joblib\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from nltk.tokenize import word_tokenize \n",
    "from sklearn.utils import shuffle\n",
    "from nltk.tokenize.treebank import TreebankWordDetokenizer as Detok\n",
    "from nltk.tokenize.treebank import TreebankWordDetokenizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from tensorflow.keras.layers import *\n",
    "import matplotlib.pyplot as plt\n",
    "# from tensorflow.keras.utils.np_utils import to_categorical\n",
    "from tensorflow.keras.regularizers import l2\n",
    "# from tensorflow.keras.utils.vis_utils import plot_model\n",
    "# from yellowbrick.classifier import ClassificationReport\n",
    "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
    "from tensorflow.keras.models import load_model\n",
    "import os.path\n",
    "from imblearn.over_sampling import ADASYN, SMOTE, RandomOverSampler\n",
    "from imblearn.pipeline import make_pipeline\n",
    "from imblearn.keras import BalancedBatchGenerator\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.utils.multiclass import unique_labels\n",
    "from sklearn.utils import class_weight\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the dataset and shuffle it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                  sentence  operation\n",
      "68             What is the location of cup          1\n",
      "110                          Detect laptop          1\n",
      "546  Can I get information about the chair          2\n",
      "92            Can you find the pen for me?          1\n",
      "566      What do you know about the bottle          2\n",
      "..                                     ...        ...\n",
      "336     Give me information about the hand          2\n",
      "690                            can we hide          3\n",
      "435           Give an account of this hand          2\n",
      "325                   I can’t find my hand          1\n",
      "603          Can you describe this monitor          2\n",
      "\n",
      "[709 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "# nltk.download()\n",
    "filepath = 'data/fyp_dataset.txt'\n",
    "df = pd.read_csv(filepath, names=['sentence', 'operation'], sep=', ', engine='python')\n",
    "df = shuffle(df)\n",
    "sentences = df['sentence'].values\n",
    "y = df['operation'].values\n",
    "print(df)\n",
    "\n",
    "# df.to_csv(r'data/dataset.txt', sep=', ', index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to **preprocess** the text utterences in the dataset that we possess. This includes:\n",
    "\n",
    "0. Shuffle the dataset: This is required as this is a manually created dataset according to predefined schema.\n",
    "1. Removing stopwords.\n",
    "2. Removing duplicates.\n",
    "3. Shuffling up data (jumbling up the order)\n",
    "\n",
    "## Removing Stopwords\n",
    "\n",
    "`NLTK`, or the Natural Language Toolkit, is a treasure trove of a library for text preprocessing. It’s one of my favorite Python libraries. `NLTK` has a list of stopwords stored in 16 different languages.\n",
    "\n",
    "However, we use a specific set of stopwords which is apt for our context.\n",
    "\n",
    "```python\n",
    "stopwords_list = [\"i\", \"me\", \"my\", \"myself\", \"we\", \"our\", \"ours\", \"ourselves\", \"you\", \"your\", \"yours\", \"yourself\", \"yourselves\", \"he\", \"him\", \"his\", \"himself\", \"she\", \"her\", \"hers\", \"herself\", \"it\", \"its\", \"itself\", \"they\", \"them\", \"their\", \"theirs\", \"themselves\", \"which\", \"who\", \"whom\", \"these\", \"those\", \"am\", \"is\", \"are\", \"was\", \"were\", \"be\", \"been\", \"being\", \"have\", \"has\", \"had\", \"having\", \"do\", \"does\", \"did\", \"doing\", \"a\", \"an\", \"the\", \"and\", \"but\", \"if\", \"or\", \"because\", \"as\", \"until\", \"while\", \"of\", \"at\", \"by\", \"for\", \"with\", \"against\", \"into\", \"through\", \"during\", \"before\", \"after\", \"above\", \"below\", \"to\", \"from\", \"up\", \"down\", \"in\", \"out\", \"on\", \"off\", \"over\", \"under\", \"again\", \"further\", \"then\", \"once\", \"here\", \"there\", \"when\", \"why\", \"how\", \"all\", \"any\", \"both\", \"each\", \"few\", \"more\", \"most\", \"other\", \"some\", \"such\", \"no\", \"nor\", \"not\", \"only\", \"own\", \"same\", \"so\", \"than\", \"too\", \"very\", \"s\", \"t\", \"don\", \"should\", \"now\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords_list = [\"i\", \"me\", \"my\", \"myself\", \"we\", \"our\", \"ours\", \"ourselves\", \"you\", \"your\", \"yours\", \"yourself\", \"yourselves\", \"he\", \"him\", \"his\", \"himself\", \"she\", \"her\", \"hers\", \"herself\", \"it\", \"its\", \"itself\", \"they\", \"them\", \"their\", \"theirs\", \"themselves\", \"which\", \"who\", \"whom\", \"these\", \"those\", \"am\", \"is\", \"are\", \"was\", \"were\", \"be\", \"been\", \"being\", \"have\", \"has\", \"had\", \"having\", \"do\", \"does\", \"did\", \"doing\", \"a\", \"an\", \"the\", \"and\", \"but\", \"if\", \"or\", \"because\", \"as\", \"until\", \"while\", \"of\", \"at\", \"by\", \"for\", \"with\", \"against\", \"into\", \"through\", \"during\", \"before\", \"after\", \"above\", \"below\", \"to\", \"from\", \"up\", \"down\", \"in\", \"out\", \"on\", \"off\", \"over\", \"under\", \"again\", \"further\", \"then\", \"once\", \"here\", \"there\", \"when\", \"why\", \"how\", \"all\", \"any\", \"both\", \"each\", \"few\", \"more\", \"most\", \"other\", \"some\", \"such\", \"no\", \"nor\", \"not\", \"only\", \"own\", \"same\", \"so\", \"than\", \"too\", \"very\", \"s\", \"t\", \"don\", \"should\", \"now\"]\n",
    "stopwords_set = set(stopwords_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before removing the stopwords, we need to tokenize the sentences. Afterwards, we remove the stopwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_stop_words(sentences):\n",
    "    filtered_sentences = []\n",
    "    detokenizer = Detok()\n",
    "    for sentence in sentences:\n",
    "        tokenized_sentence = word_tokenize(sentence)\n",
    "        filtered_sentence = [word for word in tokenized_sentence if word not in stopwords_set]\n",
    "        filtered_sentence = [] \n",
    "        for w in tokenized_sentence: \n",
    "            if w not in stopwords_set: \n",
    "                filtered_sentence.append(w)\n",
    "        filtered_sentences.append(filtered_sentence)\n",
    "    return filtered_sentences\n",
    "        \n",
    "filtered_sentences = filter_stop_words(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The filtered results are as follows (vertically shortened for brevity).\n",
    "\n",
    "```python\n",
    "[['What', 'properties', 'cup'], ['Make', 'aware', 'mouse'], ['locate', 'body'], ['Is', 'book'], ['Discover', 'book'], ['Show', 'where', 'bottle'], ['Describe', 'mobile', 'phone'], ['Give', 'account', 'this', 'mobile', 'phone'], ['Help', 'find', 'keyboard'], ['I', 'can', '’', 'find', 'chair'], ['Where', 'I', 'keep', 'chair'], ['Highlight', 'book'], ['What', 'properties', 'monitor'], ['I', 'can', '’', 'find', 'keyboard'], ['Please', 'locate', 'chair'], ['find', 'bob', 'behind'], ['Where', 'I', 'keep', 'monitor'], ['Give', 'account', 'this', 'monitor'], ['look', 'bank'], ['I', 'can', '’', 'find', 'monitor'], ['I', 'need', 'find', 'book'], ['Show', 'where', 'keyboard'], ['please', 'locate', 'pin'], ['Explain', 'properties', 'this', 'book'], ['Discover', 'chair'], ['Explain', 'features', 'this', 'laptop'], ['can', 'hide', 'pain', 'finally'], ['Search', 'pen'], ['What', 'features', 'bottle'], ['Please', 'locate', 'pen'], ['Show', 'properties', 'cup'], ['Show', 'properties', 'this', 'chair'], ['Can', 'help', 'find', 'chair'], ['Explain', 'properties', 'this', 'cup'] \n",
    " \n",
    " ...\n",
    " \n",
    "['Is', 'chair'], [\"n't\", 'guns'], ['Spot', 'cup'], ['Point', 'mobile', 'phone'], ['I', 'can', '’', 'find', 'bottle'], ['Show', 'properties', 'this', 'mouse'], ['Show', 'properties', 'monitor'], ['Search', 'hand']]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us detokenize the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detokenize(filtered_sentences):\n",
    "    detokenized_sentences = []\n",
    "    for sentence in filtered_sentences:\n",
    "        detokenized_sentences.append(TreebankWordDetokenizer().detokenize(sentence))\n",
    "    return detokenized_sentences    \n",
    "\n",
    "detokenized_sentences = detokenize(filtered_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "188"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = []\n",
    "for i in detokenized_sentences:\n",
    "    for w in i.split(\" \"):\n",
    "        words.append(w)\n",
    "len(set(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us assign the detokenized sentences back to the `pandas` dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                  sentence  operation  \\\n",
      "68             What is the location of cup          1   \n",
      "110                          Detect laptop          1   \n",
      "546  Can I get information about the chair          2   \n",
      "92            Can you find the pen for me?          1   \n",
      "566      What do you know about the bottle          2   \n",
      "\n",
      "                     filtered_sentence  \n",
      "68                   What location cup  \n",
      "110                      Detect laptop  \n",
      "546  Can I get information about chair  \n",
      "92                       Can find pen?  \n",
      "566             What know about bottle  \n"
     ]
    }
   ],
   "source": [
    "df['filtered_sentence'] = detokenized_sentences\n",
    "print(df.head())\n",
    "df.to_csv(r'data/detokenized_dataframe.csv',index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The field ```filtered_sentence``` is what we feed into the model as inputs. The labels remain the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the model.\n",
    "\n",
    "Here, we will test out several models and test the accuracy parameters, in order to arrive at a final model. We will also use a grid-search methodology for obtaining the best hyperparameters for the chosen model.\n",
    "\n",
    "However, before this step, we need to understand the distribution of the dataset. For this, we will use `matplotlib` to plot the dataset w.r.t the labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD1CAYAAACrz7WZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANi0lEQVR4nO3df6zddX3H8edLiswNI7DeNV1bvMR1MTWbhd1VjMvCJI5fJsVkI+UPaQhJ/aNkmPnHqv/gkpHUZEpmtpHVwCzGiQQ1NELmOsZizCJ4QVYoHfGKZW1T2qsgP8KGaXnvj347D+W259577rmHfng+kpvzPZ/v93vP++Ykz3vy7Tm3qSokSW1526gHkCQtPOMuSQ0y7pLUIOMuSQ0y7pLUIOMuSQ1aMuoBAJYuXVrj4+OjHkOSTiuPPPLIT6tqbKZ9b4q4j4+PMzk5OeoxJOm0kuSZk+3zsowkNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KD3hQfYlps41vuG/UIQ7V361WjHkHSiPnKXZIaZNwlqUHGXZIaZNwlqUHGXZIaZNwlqUHGXZIaZNwlqUHGXZIaZNwlqUHGXZIaZNwlqUHGXZIaZNwlqUF9457kV5I8nOQ/k+xO8pfd+gVJHkoyleTrSd7erZ/V3Z/q9o8P90eQJJ1oNq/cXwU+XFXvB9YClye5GPgccGtV/RbwPHBDd/wNwPPd+q3dcZKkRdQ37nXMy93dM7uvAj4M3NOtbweu7rbXd/fp9l+aJAs2sSSpr1ldc09yRpLHgMPATuDHwM+r6kh3yH5gRbe9AtgH0O1/Afj1hRxaknRqs4p7VR2tqrXASmAd8N5BHzjJpiSTSSanp6cH/XaSpB5zerdMVf0ceBD4IHBOkuP/B+tK4EC3fQBYBdDtfxfwsxm+17aqmqiqibGxsXmOL0mayWzeLTOW5Jxu+x3AR4A9HIv8n3SHbQTu7bZ3dPfp9v9bVdVCDi1JOrUl/Q9hObA9yRkc+2Vwd1V9O8mTwF1J/gr4IXB7d/ztwFeSTAHPARuGMLck6RT6xr2qdgEXzrD+NMeuv5+4/r/Any7IdJKkefETqpLUIOMuSQ0y7pLUIOMuSQ0y7pLUIOMuSQ0y7pLUIOMuSQ0y7pLUIOMuSQ0y7pLUoNn84TDpTWV8y32jHmGo9m69atQjqAG+cpekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBvWNe5JVSR5M8mSS3Ulu6tY/m+RAkse6ryt7zvl0kqkkTyW5bJg/gCTpjWbzh8OOAJ+qqkeTvBN4JMnObt+tVfXXvQcnWQNsAN4H/Cbwr0l+u6qOLuTgkqST6/vKvaoOVtWj3fZLwB5gxSlOWQ/cVVWvVtVPgClg3UIMK0manTldc08yDlwIPNQt3ZhkV5I7kpzbra0A9vWctp9T/zKQJC2wWcc9ydnAN4BPVtWLwG3Ae4C1wEHg83N54CSbkkwmmZyenp7LqZKkPmYV9yRncizsX62qbwJU1aGqOlpVrwFf4peXXg4Aq3pOX9mtvU5VbauqiaqaGBsbG+RnkCSdYDbvlglwO7Cnqr7Qs76857CPAU902zuADUnOSnIBsBp4eOFGliT1M5t3y3wI+DjweJLHurXPANcmWQsUsBf4BEBV7U5yN/Akx95ps9l3ykjS4uob96r6HpAZdt1/inNuAW4ZYC5J0gD8hKokNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KD+sY9yaokDyZ5MsnuJDd16+cl2ZnkR93tud16knwxyVSSXUkuGvYPIUl6vdm8cj8CfKqq1gAXA5uTrAG2AA9U1Wrgge4+wBXA6u5rE3Dbgk8tSTqlvnGvqoNV9Wi3/RKwB1gBrAe2d4dtB67uttcDd9Yx3wfOSbJ8wSeXJJ3UnK65JxkHLgQeApZV1cFu17PAsm57BbCv57T93ZokaZHMOu5Jzga+AXyyql7s3VdVBdRcHjjJpiSTSSanp6fncqokqY9ZxT3JmRwL+1er6pvd8qHjl1u628Pd+gFgVc/pK7u116mqbVU1UVUTY2Nj851fkjSD2bxbJsDtwJ6q+kLPrh3Axm57I3Bvz/p13btmLgZe6Ll8I0laBEtmccyHgI8Djyd5rFv7DLAVuDvJDcAzwDXdvvuBK4Ep4BXg+gWdWJLUV9+4V9X3gJxk96UzHF/A5gHnkiQNwE+oSlKDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KD+sY9yR1JDid5omfts0kOJHms+7qyZ9+nk0wleSrJZcMaXJJ0crN55f5l4PIZ1m+tqrXd1/0ASdYAG4D3def8fZIzFmpYSdLs9I17VX0XeG6W3289cFdVvVpVPwGmgHUDzCdJmodBrrnfmGRXd9nm3G5tBbCv55j93dobJNmUZDLJ5PT09ABjSJJONN+43wa8B1gLHAQ+P9dvUFXbqmqiqibGxsbmOYYkaSbzintVHaqqo1X1GvAlfnnp5QCwqufQld2aJGkRzSvuSZb33P0YcPydNDuADUnOSnIBsBp4eLARJUlztaTfAUm+BlwCLE2yH7gZuCTJWqCAvcAnAKpqd5K7gSeBI8Dmqjo6nNElSSfTN+5Vde0My7ef4vhbgFsGGUqSNBg/oSpJDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktSgvnFPckeSw0me6Fk7L8nOJD/qbs/t1pPki0mmkuxKctEwh5ckzWzJLI75MvC3wJ09a1uAB6pqa5It3f2/AK4AVndfHwBu624lifEt9416hKHau/WqUY/w//q+cq+q7wLPnbC8HtjebW8Hru5Zv7OO+T5wTpLlCzWsJGl25nvNfVlVHey2nwWWddsrgH09x+3v1iRJi2jgf1CtqgJqrucl2ZRkMsnk9PT0oGNIknrMN+6Hjl9u6W4Pd+sHgFU9x63s1t6gqrZV1URVTYyNjc1zDEnSTOYb9x3Axm57I3Bvz/p13btmLgZe6Ll8I0laJH3fLZPka8AlwNIk+4Gbga3A3UluAJ4BrukOvx+4EpgCXgGuH8LMkqQ++sa9qq49ya5LZzi2gM2DDiVJGoyfUJWkBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWrQkkFOTrIXeAk4Chypqokk5wFfB8aBvcA1VfX8YGNKkuZiIV65/1FVra2qie7+FuCBqloNPNDdlyQtomFcllkPbO+2twNXD+ExJEmnMGjcC/iXJI8k2dStLauqg932s8CyAR9DkjRHA11zB/6gqg4k+Q1gZ5L/6t1ZVZWkZjqx+2WwCeD8888fcAxJUq+BXrlX1YHu9jDwLWAdcCjJcoDu9vBJzt1WVRNVNTE2NjbIGJKkE8w77kl+Lck7j28Dfww8AewANnaHbQTuHXRISdLcDHJZZhnwrSTHv88/VdU/J/kBcHeSG4BngGsGH1OSNBfzjntVPQ28f4b1nwGXDjKUJGkwfkJVkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQcZdkho0tLgnuTzJU0mmkmwZ1uNIkt5oKHFPcgbwd8AVwBrg2iRrhvFYkqQ3GtYr93XAVFU9XVW/AO4C1g/psSRJJ1gypO+7AtjXc38/8IHeA5JsAjZ1d19O8tSQZnkzWAr8dLEeLJ9brEd6y/D5O321/ty9+2Q7hhX3vqpqG7BtVI+/mJJMVtXEqOfQ/Pj8nb7eys/dsC7LHABW9dxf2a1JkhbBsOL+A2B1kguSvB3YAOwY0mNJkk4wlMsyVXUkyY3Ad4AzgDuqavcwHus08Za4/NQwn7/T11v2uUtVjXoGSdIC8xOqktQg4y5JDTLuktQg4y71SPLeJJcmOfuE9ctHNZNmL8m6JL/fba9J8udJrhz1XKPgP6guoiTXV9U/jnoOzSzJnwGbgT3AWuCmqrq32/doVV00yvl0aklu5tjfs1oC7OTYp+IfBD4CfKeqbhnheIvOuC+iJP9dVeePeg7NLMnjwAer6uUk48A9wFeq6m+S/LCqLhzpgDql7vlbC5wFPAusrKoXk7wDeKiqfnekAy6ykf35gVYl2XWyXcCyxZxFc/a2qnoZoKr2JrkEuCfJuzn2/OnN7UhVHQVeSfLjqnoRoKr+J8lrI55t0Rn3hbcMuAx4/oT1AP+x+ONoDg4lWVtVjwF0r+A/CtwB/M5oR9Ms/CLJr1bVK8DvHV9M8i7AuGtg3wbOPh6IXkn+ffHH0RxcBxzpXaiqI8B1Sf5hNCNpDv6wql4FqKremJ8JbBzNSKPjNXdJapBvhZSkBhl3SWqQcZekBhl3SWqQcZekBv0f2PRMmB8YTokAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "df['operation'].value_counts().plot(kind=\"bar\")\n",
    "\n",
    "def plot_history(history):\n",
    "    plt.title('Loss')\n",
    "    plt.plot(history.history['loss'], label='train')\n",
    "    plt.plot(history.history['val_loss'], label='test')\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we test the Multinomial Naive Bayes model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.898876404494382\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.84      1.00      0.91        89\n",
      "           2       0.98      1.00      0.99        62\n",
      "           3       1.00      0.33      0.50        27\n",
      "\n",
      "    accuracy                           0.90       178\n",
      "   macro avg       0.94      0.78      0.80       178\n",
      "weighted avg       0.91      0.90      0.88       178\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vect',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=None)),\n",
       "                ('tfidf',\n",
       "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                  sublinear_tf=False, use_idf=True)),\n",
       "                ('clf',\n",
       "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags = ['1', '2', '3']\n",
    "\n",
    "sentences = df['filtered_sentence'].values\n",
    "y = df['operation']\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    sentences, y, test_size=0.25, random_state=1000)\n",
    "\n",
    "def nb_train(X_train, X_test, y_train, y_test):\n",
    "    nb = Pipeline([('vect', CountVectorizer()),\n",
    "               ('tfidf', TfidfTransformer()),\n",
    "               ('clf', MultinomialNB())])\n",
    "    nb.fit(X_train, y_train)\n",
    "    y_pred = nb.predict(X_test)\n",
    "    print('Accuracy: %s' % accuracy_score(y_pred, y_test))\n",
    "    print(classification_report(y_test, y_pred,target_names=tags))\n",
    "    return nb\n",
    "    \n",
    "nb_train(X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll try a Linear SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.9606741573033708\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.94      1.00      0.97        89\n",
      "           2       0.98      1.00      0.99        62\n",
      "           3       1.00      0.74      0.85        27\n",
      "\n",
      "    accuracy                           0.96       178\n",
      "   macro avg       0.97      0.91      0.94       178\n",
      "weighted avg       0.96      0.96      0.96       178\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vect',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=Non...\n",
       "                ('clf',\n",
       "                 SGDClassifier(alpha=0.001, average=False, class_weight=None,\n",
       "                               early_stopping=False, epsilon=0.1, eta0=0.0,\n",
       "                               fit_intercept=True, l1_ratio=0.15,\n",
       "                               learning_rate='optimal', loss='hinge',\n",
       "                               max_iter=5, n_iter_no_change=5, n_jobs=None,\n",
       "                               penalty='l2', power_t=0.5, random_state=42,\n",
       "                               shuffle=True, tol=None, validation_fraction=0.1,\n",
       "                               verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def svc_train(X_train, X_test, y_train, y_test):\n",
    "    sgd = Pipeline([('vect', CountVectorizer()),\n",
    "                ('tfidf', TfidfTransformer()),\n",
    "                ('clf', SGDClassifier(loss='hinge', penalty='l2',alpha=1e-3, random_state=42, max_iter=5, tol=None)),\n",
    "               ])\n",
    "    sgd.fit(X_train, y_train)\n",
    "    y_pred = sgd.predict(X_test)\n",
    "    print('Accuracy %s' % accuracy_score(y_pred, y_test))\n",
    "    print(classification_report(y_test, y_pred,target_names=tags))\n",
    "    return sgd\n",
    "\n",
    "svc_train(X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vect',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, vocabulary=Non...\n",
       "                ('clf',\n",
       "                 SGDClassifier(alpha=0.001, average=False, class_weight=None,\n",
       "                               early_stopping=False, epsilon=0.1, eta0=0.0,\n",
       "                               fit_intercept=True, l1_ratio=0.15,\n",
       "                               learning_rate='optimal', loss='hinge',\n",
       "                               max_iter=5, n_iter_no_change=5, n_jobs=None,\n",
       "                               penalty='l2', power_t=0.5, random_state=42,\n",
       "                               shuffle=True, tol=None, validation_fraction=0.1,\n",
       "                               verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd = Pipeline([('vect', CountVectorizer()),\n",
    "                ('tfidf', TfidfTransformer()),\n",
    "                ('clf', SGDClassifier(loss='hinge', penalty='l2',alpha=1e-3, random_state=42, max_iter=5, tol=None)),\n",
    "               ])\n",
    "sgd.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 1, 1, 1, 3, 1, 2, 2, 1, 2, 2, 2, 2, 3, 2, 3, 1, 1, 1, 3, 2,\n",
       "       1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 2, 1, 1, 2, 1, 1, 2, 2, 2, 2,\n",
       "       1, 1, 1, 2, 3, 2, 1, 1, 1, 3, 2, 2, 2, 2, 1, 1, 3, 1, 1, 3, 2, 1,\n",
       "       1, 2, 1, 3, 2, 2, 1, 1, 3, 3, 3, 1, 2, 2, 2, 1, 1, 1, 1, 2, 2, 2,\n",
       "       1, 2, 2, 1, 1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 1, 1, 1,\n",
       "       1, 2, 1, 2, 2, 3, 1, 1, 2, 1, 2, 2, 1, 3, 1, 1, 1, 1, 2, 1, 3, 1,\n",
       "       1, 1, 1, 2, 1, 1, 2, 2, 3, 2, 1, 1, 1, 1, 2, 3, 1, 3, 2, 1, 2, 1,\n",
       "       1, 1, 1, 2, 2, 1, 1, 3, 2, 1, 1, 1, 2, 2, 2, 2, 1, 2, 3, 2, 1, 1,\n",
       "       2, 1])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Third, we'll try an LSTM. This is based on [Multi-Class Text Classification with LSTM](https://towardsdatascience.com/multi-class-text-classification-with-lstm-1590bee1bd17) by Susan Li. We'll visualize the LSTM using `plot_model` and `pydot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pydot in /home/dasun/anaconda3/envs/dm/lib/python3.6/site-packages (1.4.1)\r\n",
      "Requirement already satisfied: pyparsing>=2.1.4 in /home/dasun/anaconda3/envs/dm/lib/python3.6/site-packages (from pydot) (2.4.0)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install pydot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 164 unique tokens.\n",
      "Shape of data tensor: (709, 250)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dasun/anaconda3/envs/dm/lib/python3.6/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 624 samples, validate on 70 samples\n",
      "Epoch 1/150\n",
      "624/624 [==============================] - 3s 4ms/step - loss: 1.6549 - accuracy: 0.3846 - val_loss: 1.7274 - val_accuracy: 0.0000e+00\n",
      "Epoch 2/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.5550 - accuracy: 0.3846 - val_loss: 1.7203 - val_accuracy: 0.0000e+00\n",
      "Epoch 3/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.4676 - accuracy: 0.3846 - val_loss: 1.6346 - val_accuracy: 0.0000e+00\n",
      "Epoch 4/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.3930 - accuracy: 0.3862 - val_loss: 1.4756 - val_accuracy: 0.0000e+00\n",
      "Epoch 5/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.3205 - accuracy: 0.4359 - val_loss: 1.4353 - val_accuracy: 0.0000e+00\n",
      "Epoch 6/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.2520 - accuracy: 0.4343 - val_loss: 1.4384 - val_accuracy: 0.0000e+00\n",
      "Epoch 7/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.1805 - accuracy: 0.5897 - val_loss: 1.5234 - val_accuracy: 0.0000e+00\n",
      "Epoch 8/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.1067 - accuracy: 0.5978 - val_loss: 1.4290 - val_accuracy: 0.0000e+00\n",
      "Epoch 9/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 1.0190 - accuracy: 0.6923 - val_loss: 1.2952 - val_accuracy: 0.0000e+00\n",
      "Epoch 10/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.9150 - accuracy: 0.6939 - val_loss: 1.3165 - val_accuracy: 0.0000e+00\n",
      "Epoch 11/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.8166 - accuracy: 0.6923 - val_loss: 1.2133 - val_accuracy: 0.0143\n",
      "Epoch 12/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.7327 - accuracy: 0.7548 - val_loss: 1.1789 - val_accuracy: 0.1143\n",
      "Epoch 13/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.6575 - accuracy: 0.7756 - val_loss: 1.0681 - val_accuracy: 0.3143\n",
      "Epoch 14/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.6119 - accuracy: 0.8381 - val_loss: 0.9809 - val_accuracy: 0.7571\n",
      "Epoch 15/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.6686 - accuracy: 0.8333 - val_loss: 0.9001 - val_accuracy: 0.8143\n",
      "Epoch 16/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.5395 - accuracy: 0.9183 - val_loss: 1.1002 - val_accuracy: 0.6571\n",
      "Epoch 17/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.5022 - accuracy: 0.8862 - val_loss: 0.9871 - val_accuracy: 0.7000\n",
      "Epoch 18/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.4759 - accuracy: 0.9119 - val_loss: 0.8352 - val_accuracy: 0.7571\n",
      "Epoch 19/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.4509 - accuracy: 0.9295 - val_loss: 0.7538 - val_accuracy: 0.7571\n",
      "Epoch 20/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.4103 - accuracy: 0.9279 - val_loss: 0.7744 - val_accuracy: 0.7571\n",
      "Epoch 21/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.3825 - accuracy: 0.9439 - val_loss: 0.8024 - val_accuracy: 0.7571\n",
      "Epoch 22/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.3541 - accuracy: 0.9359 - val_loss: 0.6470 - val_accuracy: 0.8000\n",
      "Epoch 23/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.3400 - accuracy: 0.9439 - val_loss: 0.6466 - val_accuracy: 0.7571\n",
      "Epoch 24/150\n",
      "624/624 [==============================] - 3s 4ms/step - loss: 0.3185 - accuracy: 0.9343 - val_loss: 0.6394 - val_accuracy: 0.8143\n",
      "Epoch 25/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.3011 - accuracy: 0.9535 - val_loss: 0.6008 - val_accuracy: 0.8286\n",
      "Epoch 26/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2929 - accuracy: 0.9471 - val_loss: 0.6863 - val_accuracy: 0.7571\n",
      "Epoch 27/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2761 - accuracy: 0.9439 - val_loss: 0.5468 - val_accuracy: 0.7857\n",
      "Epoch 28/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2674 - accuracy: 0.9551 - val_loss: 0.5508 - val_accuracy: 0.7857\n",
      "Epoch 29/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2508 - accuracy: 0.9535 - val_loss: 0.6224 - val_accuracy: 0.7857\n",
      "Epoch 30/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2457 - accuracy: 0.9487 - val_loss: 0.5364 - val_accuracy: 0.7857\n",
      "Epoch 31/150\n",
      "624/624 [==============================] - 3s 4ms/step - loss: 0.2400 - accuracy: 0.9535 - val_loss: 0.5334 - val_accuracy: 0.8000\n",
      "Epoch 32/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2331 - accuracy: 0.9583 - val_loss: 0.6058 - val_accuracy: 0.7857\n",
      "Epoch 33/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.2194 - accuracy: 0.9583 - val_loss: 0.5050 - val_accuracy: 0.8000\n",
      "Epoch 34/150\n",
      "624/624 [==============================] - 2s 3ms/step - loss: 0.2154 - accuracy: 0.9615 - val_loss: 0.6494 - val_accuracy: 0.7571\n",
      "Epoch 35/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2075 - accuracy: 0.9647 - val_loss: 0.4869 - val_accuracy: 0.8000\n",
      "Epoch 36/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2124 - accuracy: 0.9535 - val_loss: 0.5471 - val_accuracy: 0.7857\n",
      "Epoch 37/150\n",
      "624/624 [==============================] - 3s 4ms/step - loss: 0.2008 - accuracy: 0.9599 - val_loss: 0.5924 - val_accuracy: 0.7857\n",
      "Epoch 38/150\n",
      "624/624 [==============================] - 2s 4ms/step - loss: 0.2053 - accuracy: 0.9551 - val_loss: 0.5572 - val_accuracy: 0.7857\n",
      "178/178 [==============================] - 0s 1ms/step\n",
      "Model: \"sequential_8\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_8 (Embedding)      (None, 250, 32)           1600000   \n",
      "_________________________________________________________________\n",
      "spatial_dropout1d_8 (Spatial (None, 250, 32)           0         \n",
      "_________________________________________________________________\n",
      "lstm_8 (LSTM)                (None, 100)               53200     \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 3)                 303       \n",
      "=================================================================\n",
      "Total params: 1,653,503\n",
      "Trainable params: 1,653,503\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Test set\n",
      "  Loss: 0.222\n",
      "  Accuracy: 0.933\n"
     ]
    }
   ],
   "source": [
    "# The maximum number of words to be used. (most frequent)\n",
    "MAX_NB_WORDS = 50000\n",
    "# Max number of words in each complaint.\n",
    "MAX_SEQUENCE_LENGTH = 250\n",
    "# This is fixed.\n",
    "EMBEDDING_DIM = 32\n",
    "\n",
    "def init_tokenizer():\n",
    "    tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~', lower=True)\n",
    "    tokenizer.fit_on_texts(df['filtered_sentence'].values)\n",
    "    word_index = tokenizer.word_index\n",
    "    print('Found %s unique tokens.' % len(word_index))\n",
    "    return tokenizer\n",
    "\n",
    "tokenizer = init_tokenizer()\n",
    "\n",
    "def lstm_train(df, tokenizer):\n",
    "\n",
    "    X = tokenizer.texts_to_sequences(df['filtered_sentence'].values)\n",
    "    X = pad_sequences(X, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "    print('Shape of data tensor:', X.shape)\n",
    "    \n",
    "    Y = pd.get_dummies(df['operation']).values\n",
    "    \n",
    "    X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.25, random_state = 42)\n",
    "    \n",
    "    # Oversampling the minority class\n",
    "    smote = SMOTE('minority')\n",
    "    X_train, Y_train = smote.fit_sample(X_train, Y_train)\n",
    "    \n",
    "#     cw = class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=X.shape[1]))\n",
    "    model.add(SpatialDropout1D(0.2))\n",
    "    model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, kernel_regularizer=l2(0.01)))\n",
    "    model.add(Dense(3, activation='softmax'))\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "    epochs = 150\n",
    "    batch_size = 100\n",
    "    \n",
    "#     history = model.fit(X_train, Y_train, \n",
    "#                         epochs=epochs, batch_size=batch_size, class_weight=cw, \n",
    "#                         validation_split=0.1,callbacks=[EarlyStopping(monitor='val_loss', patience=3, min_delta=0.0001)])\n",
    "    \n",
    "    \n",
    "    history = model.fit(X_train, Y_train, \n",
    "                        epochs=epochs, batch_size=batch_size,\n",
    "                        validation_split=0.1,callbacks=[EarlyStopping(monitor='val_loss', patience=3, min_delta=0.0001)])\n",
    "    \n",
    "    \n",
    "    \n",
    "    accr = model.evaluate(X_test,Y_test)\n",
    "    \n",
    "    y_pred = model.predict(X_test)\n",
    "    \n",
    "    print(model.summary())\n",
    "    print('Test set\\n  Loss: {:0.3f}\\n  Accuracy: {:0.3f}'.format(accr[0],accr[1]))\n",
    "    plot_model(model, to_file='model.png')\n",
    "    return model, history\n",
    "\n",
    "model, history = lstm_train(df, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEFCAYAAAD+A2xwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3yV9d3/8ddZOSd7T7IT8g0QCCGsIKCIggMRF47aOqitba1Ve9+t7a/33bu2d9u7Wmu1trZatVqtC0XcKHuPhJWEXCEhg+y9yck45/dHgDKySXIyPk8feeTkmu8T4idXrus7dHa7HSGEEGOX3tEBhBBCXBop5EIIMcZJIRdCiDFOCrkQQoxxUsiFEGKMk0IuhBBjnBRyMa4ppSKVUk2OziHEcJJCLoQQY5zR0QGEcASllCfwPDATsAOfAT/VNK1DKfUL4CagDagG7tU0rbSn5Q55A0KcQ67IxUT1LF3FeDowG0gE/kMpFQY8AszRNG02sAGY19NyhyQX4gJSyMVEdS3wJ03T7JqmWYEXTi8rBg4DaUqpp4BDmqat62W5EA4nhVxMVHq6bqmc+7VJ0zQbcDlwL11X7H9QSv2up+UjmliIHkghFxPVF8BDSimdUsoMfAv4UimVCKQDxzRN+w3wB2BOT8sdlF2I88jDTjERuHbTBPF64NvAUcAJ+Bz4X03T2pRS7wAHTu9zCnhY07TD3S0fubcgRM90MoytEEKMbXJrRQghxjgp5EIIMcb16x65Umoe8H+apl1xwfKvAT8EOoGXNU37y5AnFEII0as+C7lS6kfA14HmblY/BUwDmoBMpdRbmqbV9nSs1NRUM11P+kvpKv5CCCH6ZgCCgf3JycnWC1f254o8F7gZeL2bdUcAT6AD0HF+u9zuzAG29+OcQgghLrYI2HHhwj7vkWuathZo72F1OpAKZAAfa5pW18fhZFwKIYQYvG5r6KDbkSulZtDVFjeKrlsr/1RK3aZp2ru97NYJkJCQgNlsHvA5U1NTSU5OHkzcETUWckrGoSEZh4Zk7J3VaiU9PR16uCV9Ka1W6unqFHFK07ROoALwvoTjCSGEGIQBX5Erpe4C3DRN+5tS6q/ADqVUG1330l8d4nxCCCH60K9CrmlaPjD/9Os3z1n+Al2jxgkhhHAQ6RAkhBBjnBRyIYQY46SQCyHEGDdmhrFtaq2juC0NXX4VLmZPXM1euJo9cTV74mRwRqfTOTqiEEI4xJgp5NXNxdR05lJTlHvROqPeCRezJ+4Wb5LCl+HnHuqAhEKIicJqtbJ+/Xpuu+22Prd9//338fT0ZOnSpcOWZ8wU8gjfaSjz9UTFTaLZWk+ztZ6WtrrTr7s+N5yqpKa5lBuTfoDF5OboyEKIcaqyspJ33323X4X85ptvHvY8Y6aQAzjpXQjyjO5x/dGiraTmf8b27He4auq96HTyCECI8e5HH6Xy3uGCIT3mrYkR/O6GnntxvvDCC+Tk5BAfH8+CBQtoaWnhf//3f1m3bh3p6ek0NzcTExPDb37zG5577jn8/PyIjo7mxRdfxGQyUVRUxHXXXcd3vvOdIck7pgp5XxImLaKsPpfi2mzSi7czPfRyR0cSQoxDDz74INnZ2SxatIj6+np+9rOf0dTUhIeHB6+88go2m43rr7+e8vLy8/YrKSlh/fr1tLW1sWjRIink3dHp9CyKW836g8+Slv8FgR6RBHhEODqWEGIY/e6G5F6vnodbVFQUAGazmZqaGh577DFcXFxoaWmhvf388Qbj4uIwGo0YjUYsFsuQZRh39x4sJjcWqzsAO1u1f2Ftb3F0JCHEOKPX67HZbGdfA2zbto3S0lKefvppHnvsMVpbW7lwTuThal037go5QJBnNInhS2m21rHj+HsXfTOFEOJS+Pr60t7eTmtr69llM2bM4OTJk6xevZqHH36YsLAwKioqRiTPuLq1cq4ZYVdSVn+CkzWZHCvdxdSQyxwdSQgxTpjNZj788MPzlvn7+7N27dqLtj136Nt58+adfb1z584hyzNmrsg7bTaKm9r6vb1ep2exugOLyZUDeZ9S1VQ0jOmEEMJxxkwhf+9wITetz+GN1BP93sfFyYNFcbdjs3eyNetftHW09r2TEEKMMWOmkKdE+mMx6Hh03QEqm/pfkCd5xzE99AoaW6vZlfO+3C8XQow7Y6aQh3u78mBiANUtVn64/sCA9k0Kv5oA9wjyq45wvHz/MCUUQgjHGDOFHOD2OB9mh/nyRmoeX2SV9Hs/vd7AYnUnJoOZQ4VfYbN3O+2dEEKMSWOqkBv0Ov62ej4GvY7vrt1Ds7W9751Oc7N4ERMwi5a2BopqsoYxpRBCjKx+FXKl1Dyl1JZuls9RSm1XSu1QSr2nlBq6rko9SAzx4YeXTyW/pplfbDgyoH3jguYCkF22bziiCSEmCKvVyrvvvjugffbv309W1vBcRPZZyJVSPwJeAiwXLNcBLwL3aZq2EPgcGJH+8P+9fAYxvu78Yesx0oqq+72fj2sw/u7hFNVm09RaN4wJhRDj2ZnRDwdi7dq1w9ZBqD8dgnKBm4HXL1geB1QDjyilpgOfaJqmDXG+bjmbjPz51nks/+tXfOudPez5wbUYDf27SxQXNJfKxkKOl+8jKWLZMCcVQgy3/Xmfkl81sL/O+xLpN4M5Udf1uP7M6Id/+tOfyM7Opra2FoCf/exnKKV4/PHHKSwsxGq1smbNGsLDw9m+fTsZGRnExsYSEhIypHl1/WmOp5SKBN7SNG3+OcsuA74CkoHjwMfA7zRN29jTcVJTUyOBvEuL/G+/2F3MJ3n1PJwUwN1T/Pq1j83ewbHWj9BjJN5yvQx1K8QYV9p+mPrOoe3w52kIJdiU2OP6yspKnnvuOaZMmYKfnx9XX301paWl/PWvf+XHP/4xP/7xj/nVr34FwNGjR7nssst44YUXSElJITGx5+P2Q1RycnL+hQsvpYt+NZCjaVomgFLqc7qKeo+F/IyEhATMZvOAT5iamnped9dX4hOY9rsPeTG9mu9fu4BoX/d+Haczt4ys0t0ERLkS7jt1wDkGmnM0koxDQzIOjUvLODLv7dyMRUVFuLq6Ul9fT15eHhkZGQDYbDYWLlzIL3/5S9577z2amppYuXIlycnJ+Pr6EhsbO6j3abVaSU9P73H9pVyOngDclFKxp79eBGRcwvEGzNfVzNM3zuFUeyffeW9vvzv7xAXOASC7bO9wxhNCjFNnRj+Mjo7m3nvv5fXXX+eZZ57hhhtuoKKigoyMDJ5//nn+9re/8eSTT9LR0YFOpxu2DokDviJXSt0FuGma9jel1BrgzdMPPndpmvbJkCfsw51Jkfwz9QRfZJXwRloedyf3PIPQGT5uIfi5h5196Olm8RqBpEKI8eLM6IfNzc189tlnvPPOOzQ1NfHQQw/h7+9PZWUlq1atwsXFhfvvvx+j0UhiYiJPPfUUoaGhxMTEDGmefhVyTdPygfmnX795zvJNwNwhTTRAOp2OP98yj+lPrueHHx7gGhWCn1vfrSBV4FyqGk9yvHw/SRFXj0BSIcR40d3oh+d64oknLlp2xx13cMcddwxLnnHxpC/Sx40nrplJVbOV76zt3y2WSP9ETAYzx8v3S09PIcSYNi4KOcDDi+JZHB3A+0cKeWlvTp/bmwxOxAQkne7pOSKtJoUQYliMm0Ju0Ot57a6FeDs78ei6/WSW9d3hJy5QenoKIca+cVPIAcK8XXnx9hROtXdy1z+3c6q9o9ftzzz0LK7VRqyn56m2Jmx224icSwgxMYyrQg5w0/RwHlwQx9HSOn70UVqf26vAudixj8jwthUNBby7/7dsyvyHjIsuhBgy466QAzy1MplpQZ78eafGh+kne912pB56Wttb2Kq9ic3eQVGtJuOiCyGGzLgs5M4mI2/evQiL0cA3395FUV1zj9uaDE5E+3c99CwepoeedrudHcffpdlaT3xwCiaDhX15H9PUWjss5xNCTCzjspADJAR78/sbZ1PT0sY33txJp63n+9Lq9PC22jA99Mws2cHJmmMEe8YyN/oG5kXfQEdnGzuPv4dd7pcLIS7RuC3kAN9Omcyq6WFszS3ntxt7HqfAxy0EP7fheehZ0VDIgfzPcDa5s1jdjl6nJyZgFmE+Uyitz0WTYQKEEJdoXBdynU7Hi6tTCPNy4RcbjrAzr+exgOOChv6hp7Wj67643W5nsbodZyf3s7lSYm/CyejMgbxPaWzt/5jqQghxoXFdyAF8XMy8/rWF2O1w9xs7qDvV1u12UacfemaV7qawOvOSz2u329mR/R7N1joSw64k2Cv2vPUuTh7Mj76RDls7O7LlFosQYvDGfSEHWBQdyH9dPZ3C2mbu+9dObLaLm/6ZDE7MjryO9k4rm469xpasN2hpaxj0OTNLdnKyJpMgz2gSw5d2u02UfyLhvtMob8jjWMmuQZ9LCDGxTYhCDvDTq6ZzZWwQ6zOK+OmnB7vdRgXP44aZD+PvHk5+1VE+SH2a7LJ9A75abrHVkJr/GRaTG4vVneh7mLxCp9OREnMTZqMrqQVfUH+qcsDvSwghJkwhNxr0vHPPYpS/B09uzuClPce73c7bNZDrZjzI/JgbATu7ct7n86MvUt/SvyJr7ThFYdtubHYbi9XtuDj1PtmFs5MbKbE30mlrZ2f2e9LrUwgxYBOmkAN4u5hZ/80l+LqY+d7avWzMLu12O51OT3xwCqtmPUaYz1TKG/L48OAzHC7cSKft/G7/Nnsn1o4Wmq111LWUszP7XdrtLSSGLSHEa3K/ckX6zSDSbwYVjQVkFu+45PcphJhYLmWqtzEp1s+DtfddzrIXvuK2f2xl58PXMiXQs9ttXc2eLJ36DQqq0tlz4kMOFn5JVtkejHoT7Z1W2jutFxV2AFe9P4nhVw0o1/yYGymrP0FawQZCfeLxcgkY1PsTQkw8E+qK/IxF0YG8eHsK9a3t3PDSJiqbWnvdPsIvgZtm/RAVNB+brZNOWwdmowteLkEEeUYT5jOFaP+ZqKB5JIYtJdwppcf74j2xmFxJiVmFzd7Brpy1MhaLEKLfJtwV+Rl3J0eTU9nIL788ws2vbOHLB6/GYjL0uL2T0UJK7CpSYlf1eezUqtRBZYrwSyDMZwona45RVp97UZNFIYToTr8uG5VS85RSW3pZ/zel1G+HLNUI+fnyGdyRFMmu/ErWvL1rVFwFJ4ZdCcCRk1scG0QIMWb0WciVUj8CXgK6nQhTKfVtYPoQ5xoROp2Ov9++gAWR/rx1MJ8nNhxxdCT83MMI9oqltD6HysbeR24UQgjo3xV5LnBzdyuUUil0Tcr816EMNZIsJgPv33cFUT5uPLHhCP9MPeHoSMwIXQLA0ZObHZxECDEW6PpzO0EpFQm8pWna/HOWBQOvAjcBq4F4TdMe7+04qampkUDe4OMOn7x6K2s25GHttPPcknBmBbo6LIvdbifXuolT9homm5dj0Xs4LIsQYlSJSk5Ozr9w4aU87LwN8AM+BYIAF6VUlqZpr/a1Y0JCAmazecAnTE1NJTk5ecD79Ucy4BMWxXUvbuLxXaVsf2g5U4O8BnWsocgZUO3CpmOvYfOoIFktuaRjdWc4v5dDRTIODck4NByZ0Wq1kp7e8wiug25+qGnas5qmJWuadgXwW+DN/hTx0ezKycG8uDqFulNtrHhpE6UNLQ7LEuYTj5dLICcqD9PYWuOwHEKI0W/AhVwpdZdS6lvDEWY0+PrsaH557UwKaptZ+ffNNFnbHZJDp9MzPfQK7NjIKN7mkAxCiLGhX7dWNE3Lp+uhJpqmvdnN+leHNJWD/WRpAvk1Tfx9bw63v7aND+9fgtEw8n2novxncLDgS7LLDjAjbGmf47YIISamCdmzsy86nY7nb5nH8vgQPs8q4Xvv73VIG3O9zkBC6GJs9g4yS2QMFiFE96SQ98Bk0PP21xeTNMmHl/bk8H+ben7QMJxiA5NxNrmjle7B2nHKIRmEEKObFPJeuFtMrF+zhHBvV/7fp4d4wwFtzI16E9MmLaS900pW6e4RP78QYvSTQt6HEE8XPv7mlXhaTKx5ezebc8pGPIMKmo+TwUJm8U46Orufqk4IMXFJIe+HaUFerL3vCgBueWULR0pqR/T8JqOZKSELsHY0k93H5NB1LRWcqDiEzdY5QumEEI4mhbyflsQG8fIdC6hvbeeav31FblXjiJ5/SshlGPUm0ou2XTQGen1LJYcLN7Iu7Q+sS3uabdlvcejkxhHNJ4RwnAk7jO1g3DUrippmKz9Yt5/lf/2KbQ8tJ8TTZUTObTG5Ehc0l8ySnZyoPESgRyT5VUfJqzpCbXPXTEd6nYEwn6lUNxWRXrSVaP+ZMkGFEBOAFPIBemhRPDUtVn6x4QjX/O0rtnxvOT4uAx9uYDCmTVpMVukedud8gM3edeukq3hPIdJvBmE+U3AyWiiozmDzsdfZk7uO5QkPoNPpRiSfEMIxpJAPwn8tm0HNqTae257FDS9t4otvX4Wb2TTs53U1e6KC55NVuptQb9VVvH2nYjY6n7dduM9UQr3jKarN4kTlIWICkoY9mxDCcaSQD4JOp+PplbOpabHyRmoet7y6lfVrlmA29jzD0FCZG7WCOZHXodf3fC6dTse8mJWUpuWyP+9jQn0UZuPI3AISQow8edg5SHp916QUK6aG8lV2KV9/YwedNtuwn1en0/VaxM9wt/iQGLaU1vZm0vK/GPZcQgjHkUJ+CUwGPW99YxGLowNYe6SQ77znmK78PZk2aSFeLgFoZfuobCx0dBwhxDCRQn6JnE1G1t2/hFmhPvx9bw4/+eSgoyOdZdAbmR+zCrCf94BUCDG+SCEfAp7OTnz6wFKUvwdPbs7gn8eqHB3prCDPaGIDkqlpLuVYiXTxF2I8kkI+RPzdLHz+7auY5OnCswcr+Ffa6JnRbnbUtZiNLhws3ECztd7RcYQQQ0wK+RAK93blkweuxNWk5763drHpeKmjIwFgMbmRHHkNHZ1t7DvxsaPjCCGGmBTyITY92JsnF4ehA255deuIj8vSk8mBswlwj6Cg+ihFNZqj4wghhpAU8mEwO9CVV+5cQENrO9e/uJHC2mZHR0Kn0zM/dhU69Ow98aE8+BRiHOlXIVdKzVNKbelm+Z1Kqb1KqV1KqReUUvKL4bQ7kqJ48oZkShpOcf2LG6ltsTo6Ej6uwUydtJDG1hry27ZR1Vjk6EhCiCHQZ+FVSv0IeAmwXLDcGfgVsETTtAWAJ7BiOEKOVY9ePoWHF8WTWV7Pza9sobXd8VfBM8OvIsxnCs22Kj4+/Ce2ZL1Jw6lqR8cSQlyC/lxB5wI3d7PcCizQNK3l9NdGoHWogo0HOp2O36+cza2JEWw7UcE9/9qJzebYDkMmgxNLp95DlNPl+LmFkl91hHVpT7M3dz2t7U0OzSaEGBxdf3oiKqUigbc0TZvfw/rvA9cB12ma1uMBU1NTI4HR0y5vhFg7bTy8uZCDFS3coXx4LDnI0ZEAsNvt1HcWUd5xlDZ7M3qM+BkV/sY49DoZhkeIUSgqOTk5/8KFl/R/6+l74r8D4oBbeivi50pISMBsHvjQr6mpqSQnJw94v5HWXc4vE2aw+E9f8JZWw2wVxaOXT3VQui6pqanMnj0bmE2nbQXZZfs4fHIjFe0ZNOoKSYldRbjvNIdnHO3/3pJxaEjG3lmtVtLTe54A/lIfTv6Vrnvnq865xSK64e1i5tMHlhLi4cx/fpTKB0dHz9gnBr2RKSELuDn5P0kMW0pbZyvbst+mpa3B0dGEEP0w4EKulLpLKfUtpdQsYA0wHdiklNqilLppyBOOI2HerqxfcyUuJiNff2MH+wpHT1d+ACejhaSIq5kbtYKOzjYZNVGIMaJft1Y0TcsH5p9+/eY5q6S54QAlhfrw5tcXcdPLW7jx75vZ/YNrifRxc3Ss80wOmkNW6W5yKtKID0nBzy3U0ZGEEL2QQuwAK6aG8syq2VQ0tXLDS5uoO9Xm6Ejn0ev0zI1eAdjZd+KjUTU0rxDiYlLIHeR7C+PPtjFf/Y+ttHcO/6QUAxHsFUu4z1QqGgrIrzrq6DhCiF5IIXegp1Ymc8O0UDYeL+O7o2xSCoDZUdej1xk4kP8pHZ3tjo4jhOiBFHIHMuj1vPG1hcwK9eHlfTn8blOGoyOdx8PZl6khC2m21pFRvM3RcYQQPZBC7mCuZhMf3r+EMC8XfvrpQd45lO/oSOeZEbYEi8mNo0VbZCxzIUYpKeSjQIinC+vXXIm72cS9/9rJ7vxKR0c6y8loYVbEMjps7aQVSHNEIUYjKeSjxIwQb97+xmI6bHZueXULFY2nHB3prNjA2fi4BpNbkUZl48kROWdpXS5tNuljJkR/SCEfRZbHh/Db62dR3tjKvW/tcvgAW2d0NUe8AWBEmiM2ttbwRfpL5Ldtx2YfXa15hBiNpJCPMo8snsIyFcIXWSU8u/2Yo+OcFeQZTYRvApWNheRVHh7WcxVWZwB2rPYGcsoPDOu5hBgPpJCPMnq9jlfvXECAm4XHPzlIWtHoGSt8dtR1p5sjfkZ75/B1Yuoq5Dp0GEgr2EB7h+Mn5RBiNJNCPgoFujvzyp0LaO+08bV/7qDJOjracLtbfJg2aREtbfXD1hzxVFsj5Q0FBHpE4G+Mp7W9iaPFW4blXEKMF1LIR6lr4ifx6OVTyK5s4JF1+x0d56wZoUtwNrlztGgLtc3lQ378kzXHADvhvtPwN8bh4uRBRvF2mlrrhvxcQowXUshHsf+9LolZoT68si+Xtw/mOzoOACajmfkxN9Jp62Cr9uaQ9/gsqO7qFBXuOw29zsisiOV02jpIK/h8SM8jxHgihXwUMxsNvHH3IlydjDz43h7yqhsdHQmACL8EVNB86lrKOZD/yZAdt62jldK6HHxcg3G3+AAQE5CEr+skTlQeGrGmj0KMNVLIR7k4fw+evWkuDa3tfP2NnXSMksG15kRdj5dLIFmleyio6nnmkoEoqs3CZu8kwjfh7DKdTs+c6OsB2J/38agbj0aI0UAK+Rhwz5xobp8Zye6CSp7YcMTRcQAwGkxcru7CoDeyM2ctzdZLv4ddeM5tlXMFeUafHYmxoHpofmkIMZ5IIR8DdDodf7l1HpE+rvx641G25JQ5OhIA3q6BzI26gbaOU2zT3sJm7xz0sTps7RTVaLhbfPFyCbxo/eyo69Dp9KTmf0anreNSYgsx7kghHyM8nZ144+5F6HU6vvHmTupHyWQUcUFzifBNoLwhn8OFmwZ9nNLa43TY2ojwnYZOp7tovYezH1OCU2hsreFYya5LiSzEuNOvQq6UmqeU2tLN8huUUvuVUruVUg8MeTpxnvkR/vzsqukU17fw358fcnQcoOuvhQWTb8bV7MWRk5soqz8xqOP8u7VKQo/bJIYtxcnozOGTm2htbx7UeYQYj/os5EqpHwEvAZYLlpuAPwDLgMuBbymlgoYjpPi3Hy9NIM7fgz/vzB41vT7NRhcWqzsAHdu0twdcZG32Tk7WHMPZyR1/957nBzWbXJgZtpT2zlYOFX51iamFGD90fbUCUErdAhwBXtc0bf45y2cAv9M07ZrTX/8B2KVp2rs9HSs1NTUSyBuC3BPavrImHtpUyFQfC39fFoVBf/GtCEeoaM+kvCMDD30I4U4Lur1F0p2mzgry2rbiY4hhktOsXre12W0ct35Bm72ZyeZlWPQeQxFdiLEiKjk5Of/Chca+9tI0ba1SKrKbVR7AuTMNNAKe/UmSkJCA2Wzuz6bnSU1NJTk5ecD7jbThzpkM7KjdzlsH80lr9+DBBXEDPsZwZLTZk9iQ/hJl9Sdwm9ROfHBKv/bbm7seSmH2lCsI8ZrcZ8aAahc2H3udVtdCLpt6z5DlH4yx8DMpGYeGIzNarVbS03tusXUpDzsbAPdzvnYHpB/1CHlqZTLuZhP/79ODo2bscr1Oz6K42zEbXdh34hOqm4r73Mdut1NYk4GT0Zkgj+h+nSfcZyoB7hGcrDlGXUvFpcYWYsy7lEJ+DJislPJRSjkBi4HdQxNL9CXYw4VfXptI3ak2fvxxmqPjnOVq9mRh3Gps9g42H3sDa3vvk0NUNxXTbK0nzDsevd7Qr3PodDqmhFwGgFa655IzCzHWDbiQK6XuUkp9S9O0duAx4Au6CvjLmqb1fQkmhsx3FiiSJvnw2oETbMsd+gGsBivMJ57EsCtpstawLfutXieHONPB58JOQH2J8J2Gs5M7ORWptHfKMLdiYutXIdc0Lf/Mg05N097UNO1vp19/pGnaHE3TkjVNe344g4qLGQ16nr9lLjodfG/tXtpHSfd9gMTwq5jkHUdxbTaHe2lhUlidgUFvYpL3wO7z6/UGVNA82jut5FYcvNS4Qoxp0iFojJsX4c83500ms7yeZ7aOnhmF9Do9i+PuwM3sw+GTmyiszrxom7qWCupPVTLJOw6jwWnA54gLmotOpyerdLeMwSImNCnk48Cvr0/Cz9XME18eprB29HSUMZtcWDLlbgx6I9uz36bhVNV568+MrRIxwNsqZ7g4eRDhm0BdSznlDdKqVUxcUsjHAR8XM/+3IpmWtk4e/XD0TEIB4OsWwoLYW2jvtLLp2Ovn3c8uqM5Ap9MT6hM/6OPHB3d1bciSh55iApNCPk58Y3Y0C6MCWHf0JJ9kFjk6znliApKID06hrqWcncfXYrfbabbWUd1URLBnDGajy6CPHegRhZdLIAXV6bS0NQxhaiHGDink44Rer+P5W+Zi0Ot4+IN9tLSNrhEC50RdT4B7BPlVR8gs2dHjkLUDpdPpiA9OwW63kV22byiiCjHmSCEfRxKCvXlk8RTya5r55SgZt/wMg97IFfFfw9nkzoG8z8go3gl0de65VDEBSZgMZrSyvdhsgx9KV4ixSgr5OPPzZTOI9HHl91szR82gWme4mD24Iv5roIMmaw3+7uG4mC99rBSTwUxsQDKn2hoprMkYgqRCjC1SyMcZV7OJv96WQqfNzgNv7x5VbcsBAj0jmRvVNXVbtH/ikB1XnX7oeaxEOheLiUcK+Th0VVww986J4VBJLU9vubj9tqNNCbmMm5P/A9XPQbX6w8slgGDPWMob8qhtHh0zKAkxUqSQj1NPrUwm0N3CLzYcRquo73uHEascYv0AACAASURBVObh7IdeN7Q/ftIUUUxUUsjHKW8XM8/eNBdrh41vv7sHm23893wM852Cq9mT3Mo02jpaHR1HiBEjhXwcu2VGOKumh7H9RAV/23Pc0XGGnV5nIC5oHh2dbeRWjJ4RIYUYblLIxzGdTsdzN83F02Li8Y/TKKobPd33h0tc4Fz0OgNZpXtk/BUxYUghH+dCPF14cmUyjdZ2vrt277gvbs5ObkT6Taf+VAVl9bmOjiPEiJBCPgHcPzeWK2OD+CSzmLcP5Ts6zrA7M8VcVqk0RRQTgxTyCUCn0/HCbfNxNhl4ZN1+qprG94NAf/dwfFxDKKw+1q/p5oQY66SQTxAxfu48cc1MKpusPLb+gKPjDCudTsesyOXYsbH52D/7nG5OiLFOCvkE8vCieGaH+fJGah47ixsdHWdYhXorEsOW0mStZVv2W9h7mW5OiLGuz0KulNIrpV5QSu1WSm1RSsVesP4/lFKpSqn9Sqmbhi+quFRGg54XV6fgZNDzxJ4SiuvH95VqYvjSs9PNHSrc6Og4Qgyb/lyRrwIsmqalAI8Dvz+zQinlBTwMpADLgGeGI6QYOjNCvHlqZTK11k7uen07HaNsLJah9O/p5rw5fHITRTVZQ3Jcu91GYXUmX2a8Qnm7DNIlHE/XV3M0pdTTwD5N0946/XWxpmmTTr82AZuAlYArsF3TtKiejpWamhoJyJxcDma32/npzmI2Fjbwjam+PDQz0NGRhtUpWy251k3oMDDZfBVOerdBHafT3kFtZz7VHcdpszedXqoj3nI9Jp3z0AUWomdRycnJ+RcuNPZjRw/g3ME6OpVSRk3TzsxccBLIBAzAb/qTJCEhAbPZ3J9Nz5OamkpycvKA9xtpYyHn/+uwUdBi57XMam5JmcGKqaGOjnSRofw+BpZ5sjNnLZXGQ1w347sYDaZ+79tsreNYyS5yyvbR1tmKXmdkcuBszCZX0ou2YgloZWb4wiHJORzGws+jZOyd1WolPT29x/X9ubXSALifu885RfxaIBiIAsKBVUqpuYPMKkaQm8nAO/csxmzUc++bOymoaep7pzFsctAc4gLnUtNcyu7cD/rsGGW326loKGBL1pu8t/93pBdvQ683MjP8Km6b8ziXTb6VxNAr0WMku2w/NrtMaCEcpz9X5DuBG4B3lFLzgaPnrKsFTgFWTdPsSqk6wGvoY4rhkBjiw7M3zeXb7+7hzte3s+V7y3AyGhwda9jMjbmB6uYScivSCHCPQAXPO2+93W6joqGQguqjFFRn0GytA8DbJYipkxYS7T8Tg/7f/8uYjGa8DBHUtOVSVJN1ydPWCTFY/SnkHwBXK6V2ATrgPqXUY0COpmnrlVJXAXuUUjZgB/Dl8MUVQ23NvFi2nSjnjdQ8Hv8kjadvnOPoSMPGqDexJP5rfHToOfaeWI+PWzC+rpMoazhBQVU6hdWZnGrvapZpMpiJ9p/J5MDZBHnGoNPpuj2mjzGams5ctLK9UsiFw/RZyDVNswEPXrA465z1Pwd+PsS5xAjR6XT8+ZZ5pBXV8MdtWSyMCuTmGeGOjjVs3CzeXK7uZEPGy3yV8Q/AjrWjqxmm2ejK5MA5RPgmEOwVc97Vd0+c9V4EuEdQXHucxtZq3C2+w/wOhLiYdAgSuJlNvPONxbg4GVjz9i5yq8Z3Z6EQ78nMiliOtaMZg95IfHAKyxMe4PZ5P+WyybcQ6qP6VcTP6Jpmzo5Wum/4QgvRCynkAoCpQV78+Zb5NLS2c/tr22htH98P72aEXcGtsx/ntjmPMz/mRoK9YtDrBvd8IMIvAbPRhePlB+i0dfS9gxBDTAq5OOvrs6O5f24sB4tr+MG6feN+yFs3ixe6IZhuzqg3ERs4G2tHMwVVPTcRE2K4SCEX53n25jnMDPHmpT05PLd9aHpCTgQqqKvVbVaZzBcqRp4UcnEeZ5ORdfcvIcjdmR+uT+WTzCJHRxoTPJz9CPGaTEVDPrXNZY6OIyYYKeTiImHerqy7/wqcDHru+ud2jpbWOjrSmKCCutqla2V7HZxETDRSyEW35oT78epdl9Fk7WDl3zdT1nDK0ZFGvTDfKbg4eZBbkUZ7p9XRccQEIoVc9Oi2xAh+ee1MCmubufmVLZxqlxYZvdHrDMQFzaW900pe5WFHxxETiBRy0aufLE3ga8lR7C2s4v63do37liyXanLgHHToySrdMyq+V2kFGyhuSx0VWcTwkUIueqXT6XhxdQqXRfrzzqECnthwxNGRRjVXsydhvlOoaS6hqumkQ7PkVhzkyMlN1HSeoKIh36FZxPCSQi76ZDYaWHvfFUT5uPHEhiO8mXZpQ8o3tLbxl50aNS3j8z5yfNB8ALRSxz30rG+pZHfOB2d7qKYXb3NYFjH8pJCLfvF3s7B+zRI8LCbWvLWLXXkVgzpOSX0LVzy/gYfe38ej68bnJNDBXjG4W3zJqzrskImfOzrb2ZL1Bh22NhZOXo2zzoeTNceoaxncv5kY/aSQi36bGuTFW19fTKfdzsq/b+aDo4UD2j+rvJ6Fz33O4ZJaXJwMvJmWR05VwzCldRydTo8KmkenrYOcitQRP//+vE+obSlDBc0jyn8G/iYFQEbx9hHPIkZG/0cGEgJYHh/Cy3cs4Nvv7OHWV7fywPzJ/H5lMq7m3mfc2ZVXwY0vb6ampY1fXTuTGD937nx9O7/5Kp2/37FghNKPnNjAZNIKNnDk5GaKarTTw+Dq0OlAx5nXOvQ6Ay5OHriavXA1e5797OzkPqixX/KrjqCV7cHbJYg5USsA8NBPwt3iS25FGkkRy3Bxcu/jKONLp62Djw49h797GJdNvtXRcYaFFHIxYHcnR5M0yYe7/7mDF/ccZ/uJcv75tUUkhfp0u/26o4V87Z87aLfZ+PvtC7h3bgw2m51fBh7h9dQT/Ozq6UT5jq/iYjG5Eh88n8ySHZTW5wx4fx16XMzuuJq9UUFzifZP6nFM9DMaW6vZeXwtRr0Tl8ffdXY6O51Ox7SQRezJXUdW6S5mRSwf1Hsaq05UHqKupZz6lkqSIpaPy19kUsjFoEwL8mL3D67lJ5+k8ez2LFKe/YzfXJ/EDxZNQa//d8F5YVc2339/H84mA2vvW8I18ZMA0Ot1/PSq6dz9xg5+uymdv96W4qi3MmzmRq9gdtR1YLdz5r+u15xd1mlrp6WtgWZrHc3W+n9/bquj2VpHZUMBFQ35ZJftZ37MKrxdu58ou9PWwZasf9HeaWVR3Gq8XALOWx8bMIuDBV+SVbqH6aFLMBmchv8bMArY7XYyS3Z2vcZGTnkqM8KucGyoYSCFXAyaxWTgD6vmsDw+hPv+tYv/WJ/KF1klvHLnAoLcnfnvzw/x66/S8Xcz89GaK5kT7nfe/qtnRvDEhiP8Y/8Jfrp0OhE+g5vdfjTT6/Rd82r1yIKzkzu+bpO6XdvYWsO+Ex9zsiaT9Yf+yNSQhcwMX4rJcP7k5an5n1HdVERsQDIxAbMuOo7R4ER88HwOn9xITvl+poRcdgnv6t9sdhvHy/YT6Bl10S+P0aCs/gS1zaVM8o6jrP4Ex8v3Mz308j7/uhlr5GGnuGTXxE/i0H+s4Jr4EL7MLmXmUx9z0ytb+PVX6cT4urPz+9deVMQBDHo9P71qOu2dNn63OcMByUc/d4sPS6d+g6VT78HVyZOM4m18kPo0+VVHz3byKazOJLNkJ57O/syLubHHY00JScGgN5JRvGNIJou22+3sO/ERu3M/YEP6S7S2N1/yMYdaZskOABLDriLSdzqNrdWU1Z9wcKqh12chV0rplVIvKKV2K6W2KKViL1h/rVJqz+mPPyulxtevOtEvge7OfPzNK/njqjk0Wtv5KKOI2WG+7Pj+cmL8er4neWdSJDG+7ry8N4eiutFXCEaLMJ8prJr1GIlhV9La3sSWrDf4MuMVSuty2HH8XQx6I1fEf63XWyYWkxuxAck0WWspqLr0X5zpxVvJKt2NyWChpa2BHdnvDksP0vbONoprs7HbbQPar+FUFSdrsvB3DyfAI5zJQV3z0R4v3z/kGR2tP1fkqwCLpmkpwOPA78+sUEq5A08CKzRNmw/kAxdfeokJQafT8dCiePY+ch2/vX4WG79zNQHuzr3uYzTo+clVCbR12nhSrsp7ZTSYSIpYxqpZjxLiNZmSumy+SH+Jto5TzIteibdrUJ/HmDZpEaAjvXjbJRXd3Io0UvM/x9XsyapZjxDsFUtRbdbZK+Ch0mnrYGPmq3yZ8TJHi7YOaN+ue+N2poYsBCDQIwoPZz/yq9LPztM6Xuj6+sdUSj0N7NM07a3TXxdrmjbp9OvlwL1AGxANvKRp2j96OlZqamokcGndAsW402Gzc8tHOVSf6mDdjbH4OffelFF03dZosBVT1n4UV70/k0zJ/b7vW2DdRYOtmCinK3Az+A/43E2d5eS1bUePkRjzEix6T9rtreS0bqCTNqLNV+Ki774F00DY7XaK2vdR19nVX0GHnsnm5Zj1fT9L6bS3kdX6MQadE8p83dmZoCrbNco6jhBsSsLPGNvHUfqv1dZAdcdxPAyTcDf0/Qv1EkQlJyfnX7iwPw87PYD6c77uVEoZNU3roOvqewkwE2gCtiuldmualt3bARMSEjCbzb1t0q3U1FSSk5MHvN9IGws5R1vGn3d68uC7e9hQbeT3N3blGm0Zu+PYjLOBnu+Jn3FhxrAGPz498hfaXctInnbNgM5Y3VTC50fXo9fpWZZwH0Ge0WfXhdcGsCHjZSp0B7kh8WGcjJZ+H7e772Na/hfUFRXi7x6OCprHjuPv0mg5TkrCmj5/aaUXbcOW38msiCtICJ1zdvmptjje2Z9Oq6mUWUmrB/TQs7uMdrsdrWwvmXkb6bR1UNN5gkneccyOvK5ffyH1l9VqJT2952kE+3NrpQE49yan/nQRB6gG9muaVqZpWhOwja6iLsSA3DM7mjAvF/66O5vyRhn7fDgFeEQQ4B5BUW0WdS3l/d6vqbWWrzJfOd3E8fbzijhAiPdkpodeTmNrDbty3r+kWzda2V6OFG3G3eLL0qn3EBMwi1BvRWl9DrkVab3ua7N3cqx0F0a96ex98TOcndwJ95lCbUsZ1U3Fg84H0NrezKZjr7Endx1GvVPXJN6esRTXZrP+4B/ZeXwtLW0j03O5P4V8J3AdgFJqPnD0nHWpQIJSyk8pZQTmA5lDnlKMe05GAz9emsCp9k6e3iI/QsNtWuhioP/d9q3tLXyZ8TKn2hqZE7WCKP8Z3W6XFHE1Ae4R5FcdGfRDxaKaLPbkfIjZ6MrV0+7DYnJFp9MxP2YVRr2J/Xmf9NpCprA6g2ZrHbGBszEbXS5aPzmwa37V7Et46FlSe5wP057hZM0xgj1jWJn0A+KDU1iWsIarpt6Hp4s/x8v38/6BJzlU+BXtnW2DPld/9KeQfwC0KqV2AX8AHlVKPaaUWqlpWiXwE+ALYC/wvqZpMo24GJT758YyydOFP+/SqGxqdXSccS3cZwoeFj9yKw72edXYYWtn47HXqD9VybSQhUybtLDHbfU6A4vVnTgZndl7Yv2A5y+taipiS9ab6PUGrpp2Dx7O/2474WbxJiliGdaOFvbnfdLjMTKKux64TgnpfuiHEO/JuDh5kld5aMAFttPWwf4TH7Mh4+9YO1qYHXktyxLW4Gr2BLoe+If6KFYm/YCU2JswGswcKvyK91OfJLtsP7YBtrzprz7vkWuaZgMevGBx1jnr3wLeGuJcYgIyGw38aMk0frBuP89sO8bN3XdiFENAp9MzbdIidud+wLGSXSRH/vteeYetnbaOVto6TtHW0UpG8VYqGvKJ9JvR1VO1D24WLy6bfCubj73OVu1NViQ+hLEfPUkbW2vYmPEPOmztLJlyN/7u4RdtMyVkAScqDpFbkUZMwCxCvM5/YFnZWEhlYyGh3vF4Onf/IFev0zM5cDaHT24kv+oIkwNn95kNuh5ofnL4eWqaS/Fw9mOxugM/t9AezmFABc0j2m8m6cVbSS/ezq6ctTRb60iKuLpf5xsI6dkpRpU182P5zcZ0/rQji6UrYhwdZ1yLCZjFwcINZJbspKA642zhttkvntIv0COKRXGrz7b+6EuE7zSmBC/gWOku9p5Y3+dgVZ32Nr7KeJVT7Y3Mi15JhO+0brfT6wykTL6JTw49z+6cD7gx6ZGzY8oAZBZ3dcfv7a8G6BrU7PDJTRwv39+vQn6i8hA51i+xW23EBc5lTvSKfg1zYDKaSYpYRlzQPLJKdxPiPbnPfQZDCrkYVZxNRv5zyVR+uD6VJw+UkpzUhqfzxBgXZKQZDSYSw64iNf9T2jpO4WS04Gb2xslo6fowOONk7BpCYHLgnLOTVPTX7KjrKG/I53j5AZydPPByCcRkMGMyOGE0OGHSmzEanDDojRS07aTZVsW0SYt6vCVyhp9bKFNCLiOzZAdHijadHQSs2VpHftVRvF2CCPLs/SLA3eJDiFcsJXXHqWup6HV4Aa10L7tz16HHyBXxdxHhlzCg7wN0zRx17l89Q00KuRh1vpUSx8v7cthQUE/8bz/k19cncc/smPMG4xJDY0pIClNChmfAsq7epnex/tCzHDm5qc/tI/2mMzvy2n4dOyniagqq0zlatJUov0S8XYM4VrobOzamTlrYr2aFcUFzKKk7zvHy/cyJur7bbdKLtnEg/1MsJldC9QsGVcRHgoy1IkYdFycj+x65nu8mBtDU1s43397NZc99xr7CKkdHEwPk4ezHjUk/YFHcaubHrGJ25LUkhi1lWshC4gLnEuWfSJjPFHwNsSwcwK0bk8HM/Jgbsdtt7M75gPYOK9ll+7CY3IjyT+zXMcJ8pmI2upJTnkan7fzbSXa7nYMFX3Ig/1NcnDy4Zvq3cdZ7Dfj9jxS5IhejksVk4N5pfvx45UJ+/HEabx3MJ+WPn3HvnBh+fX0SgX10/Rejh7vFF3eLb6/bpKamYtQPrEdvmM8UIv2mk191lC8zX6Gt4xSJYUv7fRyD3khsQBIZJTs4WZNJpF9Xk0q73c7+vE/ILNmBu8WHZQnfxN3iAzh2Mu3eyBW5GNVCvVx54+5FbP7uMmYEe/Pq/lzif/shz2zNpL1zeJpyibFjbvQNmAwWKhry0esMxAfPH9D+k4NOtykv62pTbrPb2JXzPpklO/ByCeDa6Q+eLuKjmxRyMSYsjglk/6PX8aeb52LQ6fjh+lSm/PZDfr85g+pmq6PjCQdxcfI4+xAx2n8mzgOc/cfLJYAAjwhK6nJoOFXF9uy3OV6+H1/XSVwz/du4mD2GI/aQk1srYswwGvR85zLF6pmR/M8Xh3l5bw4/+jiN//r8ELfPjOQ7lynmdjPuuRjfVNA8XM2eBHpEDWr/yYFzqGgo4NMjf6G1vZkA9wiumnbfgMaKcTS5Ihdjjq+rmedunsvJn9/C71cmE+blymsHTpDyx8+Y98ynvLIvh5a2i9tCi/FJp9MR5jNl0IU30m8GJoOZ1vZmgj1juTphzZgq4iBX5GIM83Ex88jlU3l40RS+Ol7KX3ZqfJxZzDff3s1/rk/lnjkxPDB/MvGBno6OKkYxk8GJ+TGrqG0uZWbE1QN+6DoaSCEXY55er2OZCmGZCqGwtpkX92Tz0p4cntl2jGe2HWNhVABr5sdy64wIXJzkR15cLCYgCUhydIxBk59qMa6Ee7vyy2uT+NnVM/gw/SQv7TnOxuNl7Mir4JEP9nPXrCi+OX8yMyeN/pYIQvSXFHIxLpmNBlbPjGT1zEjyqht5eV8Or+zL5S+7svnLrmxmh/myZl4s986JwclocHRcIS6JPOwU416Urzu/vDaJ/J/dzAf3XcGKqaGkFdXwnff2MvsPn7A7v9LREYW4JFLIxYRhNOhZmRDGh2uWkP9fN/PggjgyyupZ9KfP+f77+2hoHd7B/4UYLlLIxYQ0ydOF52+Zx9bvLUf5e/DnnRrTf/cRH2cWOTqaEAMmhVxMaAujA0j74Qr+e9kMyptaufHvm7njtW0yb6gYU/p82KmU0gN/BhIBK/BNTdNyutnmE+BDTdNeGI6gQgwXs9HAz5cncmtiBN9+Zw/vHi7gq+xSfndDMvfNjRnQTOtCOEJ/rshXARZN01KAx4Hfd7PNrwBpzyXGtGlBXmx7aDnP3TSXdpuNB97ZTdLvP+blvTm0tnc6Op4QPepPIV8IfA6gadoe4Lx5kZRStwI24LMhTyfECNPrdXx3oSL9P1dyR1IkmeX1PPDObiJ/tZaff36I0oYWR0cU4iI6u93e6wZKqZeAtZqmfXb660IgWtO0DqVUAvAEcCvw30BZb7dWUlNTI4G8IcouxLArb27n3eM1rMuppaHNhlEPV4d7cke8D1N8ZEx0MeKikpOT8y9c2J8OQQ3AuWND6jVNOzMi0TeAScAmIBJoU0rla5r2eW8HTEhIwGw29yf0eVJTU0lOTh7wfiNtLOSUjP133WL4k7Wdf6bl8ey2Y3yWX89n+fUsjApgxSQnHrvxcgz60dtuYLR8H3sjGXtntVpJT0/vcX1/CvlO4AbgHaXUfODomRWapv3ozGul1P/QdUXeaxEXYixyNZv4dkocD8ybzJfZpTy7/RifZ5WwIw/+mvkhP1gcz31zY3Ezj70Bl8TY15/LiA+AVqXULuAPwKNKqceUUiuHN5oQo49er2N5fAifPLCUjB+t5KZYb0obTvHIugOEP7GWxz9Oo6iu2dExxQTT5xW5pmk24MELFmd1s93/DFEmIcaE+EBPfjI3mL98/Spe2JXNn3dqPLk5gz9szWT1zEgevXwKs0J7n6tSiKEgg2YJcYn83Sz817IZ/OeSabyZlscz2zJ5My2PN9PymBfux6LoAOZH+pMS4U+QhzwgFUNPCrkQQ8RiMnD/vFjumxvDBq2UP2zNZOPxMvYWVp3dJsrHjXkRfqRE+DM/0p/EEG9MhtH7oFSMDVLIhRhiOl3XffTl8SE0traz/2QVewqq2J1fyZ6CSt46mM9bB/MBcHUysmJqKLfNjODa+ElYTDKkrhg4KeRCDCN3i4krJwdz5eRgAOx2O8erGs8W9Y3ZZbx9KJ+3D+XjbjZxw7RQVs+MYJkKwSzjpIt+kkIuxAjS6XTE+XsQ5+/BPXNisNvtHCyu4Z1DBbx7OP/svXVPi4kbE8K4bWYkV8YGyZW66JUUciEcSKfTMSvUl1mhvvzm+iT2n6zm3dNF/bUDJ3jtwAmcTQYWxwSyLC6Yq1UIUwM9ZSAvcR4p5EKMEjqdjrnhfswN9+P/Vsxib2EVa48U8KVWyhdZJXyRVQKkEuLhzNUqhKvjgrkqLhh/N4ujowsHk0IuxCik1+tIifQnJdIfgOL6Fr7KLuVLrYSvjpfyj/25/GN/LgAR3q5E+bgR6eNGlO/pz6c/gtylueNEIIVciDFgkqcL98yJ4Z45Mdhsdg6X1PJldglfZZeSVdHAltxyyC2/aD+zUU+UuxMrSmBpXDALowJwcZL/7ccb+RcVYozR63UkhfqQFOrDj65MAKC1vZOC2ibyaro+8qtPf65p4khJLU9tyeSpLZk4GfRcFuXP0snBLI0LJjnUZ1QP+CX6Rwq5EOOAxWRABXiiAjwvWrdz736avSaxMbuUjcfL2JxTzuaccn722SG8nJ24LMqfOH8PYnzdifFzJ8bXnXBvV+moNIZIIRdinLMY9VymQlimQgCobGplc04ZG4+X8lV2KZ9kFvMJxeftY9DriPB2JdrXncl+7iyICmBJbCDBHi6OeAuiD1LIhZhg/N0srJ4ZyeqZkUBXYc+tbiSnqpETVY3kVjdxorqR3OpGvsruKvZ/2ZUNwNRAT5bEBnHl5CAujwnE22Xg8wqIoSeFXIgJzt/Ngr+bhfkR/heta7K2c6y8nq255Ww8XsaOvHKe36nx/E4NvU7HrFAflsQGEefvgZ+r+fSxzPi7WvCwmKS9+wiRQi6E6JGb2cSccD/mhPvxH0um0dbRyd7CKjYfL2NTThl7Cqo4cLK6231NBn1XcXftKu4BbhYC3Z0JdLcQ4Hbms4VAdwvtnb1POSl6J4VcCNFvTkYDi6IDWRQdyH8vT6TZ2s6egipO1rVQ1dxKVbOVyqZWKptbqW62UtlkJb+2iSOltb0eVwcEfppPqJcLkzxdCPV0+fdrL1fCvVyI9HGTK/weSCEXQgyaq9nE0rjgPrdrbe+koqmViqZWyhtPUd7YSkVT1+fyxlaOl1TQYDOQXlrX4xW+j4sTc8L9mBfux7yIrh6wPnKPHpBCLoQYARaTgXBvV8K9Xbtdf2ZiY7vdTk1LG0X1zRTVtVBU30JRXTO5VU3sP1l1zlAFXSb7uTM3wo+5YX5M8nLBy9kJL4sT3i5OeDk74WE2odeP/6v4Pgu5UkoP/BlIBKzANzVNyzln/aPAHae//FTTtF8MR1AhxPin0+nwdTXj62omMcTnovWVTa3sLaxiX0EVewur2F9YxRupebyRmtfD8cDDbMLL2YlgD2fCvV2J9HYjwseNCG9XIk9/Pre3a1tHJ9UtVqqauz6qT3/Ozaumxq2ExBBvAkbZ0Af9uSJfBVg0TUtRSs0Hfg/cCKCUiga+BswD7MB2pdQHmqYdGa7AQoiJy9/NwoqpoayYGgqAzWYnu7KB1KJqqpqt1J1qo/ZUG3WnP+pPf13b0saBk9XsKajq9rgBbhZcnYxUt1hpaG3v8fxPp3UNgxDk7sz0YC8SQ7yZEeJNYog3KsCTTpudmhbr6Y+2s59rW6w0WNu5fWYkU4O8hvz70p9CvhD4HEDTtD1KqdnnrDsJXKNpWieAUsoEtA55SiGE6IZeryM+0JP4wIt7tF6o02ajpP4UBbXN5Nc2UVjbTH5NU9fXNU2cau8k6v+3d3chUtVhHMe/65TjS6ukC1EQWUTPRULkRF2UtVERFt50nYReZC8XhVFhKEF005tEBQWmGb1AlFp4ISn0YlZQDZu1VM9qaRZtXciaW5uzrtnFObOeds7sbjrO+f/jhDWQngAABIRJREFU97nZOefMzvz2YfaZc/5n5vznnEHXzDJzZpTpmlmma+Y0utIjhB/37WVo+hx2/TLAV/0DbO/rZ3tf/+jjd3TAsQk+fPPXkaM8vrhysn92g45jEzyzmb0IbHT3renyfuACdx/J3KcDeALodPflzR6rWq3OA/KPgUREInJo+Ch7Bg6z++Bh+gZq7B+sMa00hVlTS8wql5KfU0vMLh//efHc6Zx2cmP251cqlX1jV05mj/wQ0JlZnjKmiU8D1gODwF2TSTJ//nzK5f9+trl+QiR0MeRUxtZQxtaINeO1bXruWq1Gb29v0+2TuSrOx8BNAOkY+df1Deme+DvALndfXh9iERGR9pnMHvlm4AYz+4Tkc/tLzWwFsAcoAdcAZTNblN5/pbt/ekrSiohIgwkbubv/DdwxZvV3mduaZ0pEpEC64LCISOTUyEVEIqdGLiISOTVyEZHItfuiWSWA4eHhE36AWq3WsjCnUgw5lbE1lLE1lLG5TM8s5W2f8JudrVStVq8CPmrbE4qI/L8srFQqO8eubPce+efAQqAf0JeHREQmpwScTdJDG7R1j1xERFpPJztFRCKnRi4iEjk1chGRyKmRi4hETo1cRCRy7f744QmZaALoUJhZD/B7urjX3ZcWmSfLzK4AHnP3bjO7ENhAMs9qL3B3epXLQo3JuADYAuxONz/v7m8Ul250KsP1wDygDDwKfENAtWyS8WcCqqWZlYC1gJF8DHkpySWyNxBOHfMyziagOmZF0cgZZwLoUKQzJeHu3QVHaWBmDwBLgD/TVWuAVe7+gZm9QFLLzUXlg9yMC4A17v5Ucaka3AoccPclZjYX6AG+JKxa5mV8hLBquRjA3a80s26S12MHYdUxL+MWwqrjqFiGVv41ATRw2fh3L8QlwAwz22Zm76VvOKH4Hrgls1wBPkxvbwWub3uiRnkZbzazHWa2zsw6m/xeO70JrM4sjxBeLZtlDKaW7v42cHu6eB7wG4HVcZyMwdQxK5ZGPovjQxYAR80stKOJIeBJ4EaSiTheCyWju28EjmRWdbh7/ZtggySHjIXKyfgZcL+7Xw38ADxcSLAMd//D3QfTf+C3gFUEVssmGUOs5YiZvQw8S5IzqDpCbsbg6lgXSyMfdwLoQPQBr7r7MXfvAw6QfKU2RNmxx07gYFFBxrHZ3av128ClRYapM7NzgfeBV9z9dQKsZU7GIGvp7rcBF5GMRU/PbAqijtCQcVuIdYR4GnnTCaADsoxk7B4zO4fkKKK/0ETN9aTjfgCLCPNCZu+a2eXp7euA6nh3bgczOwvYBjzo7uvT1UHVsknGoGppZkvMbGW6OETyZvhFYHXMy7gppDpmBXHoPwkNE0AXnCfPOmCDme0kOfO+LMCjhrr7gLVmNhX4luSwMTR3As+Z2TDwK8fHK4v0EHAmsNrM6uPQ9wDPBFTLvIwrgKcDquUm4CUz2wGcDtxLUruQXpN5GX8ivNckoItmiYhEL5ahFRERaUKNXEQkcmrkIiKRUyMXEYmcGrmISOTUyEVEIqdGLiISuX8AqlDcrMNTPIQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(history) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model architecture is shown below.\n",
    "\n",
    "<div align=\"center\"><img src=\"model.png\" /></div>\n",
    "\n",
    "## Test the Models\n",
    "\n",
    "Now, let us test the models. We start with the LSTM.\n",
    "\n",
    "### Testing LSTM "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted vector:  [[0.79105943 0.00257285 0.20636767]]  Predicted Class:  Locate\n"
     ]
    }
   ],
   "source": [
    "new_command = ['Track the pencil']\n",
    "filtered_commands = filter_stop_words(new_command)\n",
    "seq = tokenizer.texts_to_sequences(filtered_commands)\n",
    "padded = pad_sequences(seq, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['Locate', 'Describe', 'No_Op']\n",
    "print(\"Predicted vector: \", pred, \" Predicted Class: \" ,labels[np.argmax(pred)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(r'data/dataset.txt', header=None, index=None, sep=',', mode='a')\n",
    "model.save('lstm.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_model(model_path ):\n",
    "    model = model.load(model_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}