DatasetCleanup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Clean up a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Baby Bunny](img/kit.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\david.curran\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\numpy\\core\\__init__.py:29: UserWarning: loaded more than 1 DLL from .libs:\n",
      "C:\\Users\\david.curran\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\numpy\\.libs\\libopenblas.IPBC74C7KURV7CB2PKT5Z5FNR3SIBV4J.gfortran-win_amd64.dll\n",
      "C:\\Users\\david.curran\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\numpy\\.libs\\libopenblas.TXA6YQSD3GCQQC22GEQ54J2UDCXDXHWN.gfortran-win_amd64.dll\n",
      "  stacklevel=1)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "#generic imports\n",
    "import csv\n",
    "import notebook as nb\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "#import json\n",
    "import sys\n",
    "import inspect\n",
    "import math\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn import metrics\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "import random\n",
    "\n",
    "dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) # script directory\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have some example documents with classifications and you want to increase the accuracy of predictions you make with this dataset. This is a workshop to find issues in a dataset with python. \n",
    "\n",
    "We use a bunny farm chatbot. But sentiment analysis, news, email classification or any supervised learning dataset will have similar issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bashful Bunnies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A pet rabbit seller on facebook wants to answer common customer questions while they are offline. They give you previous questions they have been asked. And you help them label these by Topic_Verb. This is a whole other talk.\n",
    "\n",
    "Now you want to quickly go through these labels to find mistakes. Usually 500 questions in 20-50 intents. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tasks\n",
    "<input type=\"checkbox\"> Load Libraries\n",
    "<input type=\"checkbox\"> Label Data\n",
    "<input type=\"checkbox\"> Find Topics, Find Verbs\n",
    "<input type=\"checkbox\"> Load Data\n",
    "<input type=\"checkbox\"> Language specific stuff (spelling, accents, segment words in Chinese etc)\n",
    "<input type=\"checkbox\"> Run KFold test\n",
    "<input type=\"checkbox\"> Save Data to show improvements\n",
    "<input type=\"checkbox\"> Graph questions, Intents and Accuracy\n",
    "<input type=\"checkbox\"> Find Duplicates\n",
    "<input type=\"checkbox\"> Find Wrongly labeled\n",
    "<input type=\"checkbox\"> Find Double Categories\n",
    "<input type=\"checkbox\"> Find Nonsense\n",
    "<input type=\"checkbox\"> Find Bad intents\n",
    "<input type=\"checkbox\"> Look at confusion Matrix\n",
    "<input type=\"checkbox\"> Label new data \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open unlabeled.csv in your spreadsheet viewer.\n",
    "\n",
    "Can you use kmenas clustering? Yes but you have to look anyway so it doesnt speed things up\n",
    "\n",
    "Look for Topics\n",
    "\n",
    "Move fast and talk\n",
    "\n",
    "Then look in topics for verbs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First off load the data. The file needs to be this name, type and in he local directory. And in the format Question, Intent with a header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Intent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Are the baby rabbits certified?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What cert does the kit have?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Does the rabbit have a certificate? And what f...</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Is the kit certified by a vet?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Has the kit been cleared by a vet?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            Question                Intent\n",
       "0                    Are the baby rabbits certified?  RABBIT_CERTIFICATION\n",
       "1                       What cert does the kit have?  RABBIT_CERTIFICATION\n",
       "2  Does the rabbit have a certificate? And what f...  RABBIT_CERTIFICATION\n",
       "3                     Is the kit certified by a vet?  RABBIT_CERTIFICATION\n",
       "4                 Has the kit been cleared by a vet?  RABBIT_CERTIFICATION"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filePath = r'bunny.csv'\n",
    "df = pd.read_csv(filePath,header=0,encoding='utf-8')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ground Truth is utterence (question), intent\n",
    "\n",
    "  - Real questions from real users\n",
    "  - Topic_Verb the utterences\n",
    " \n",
    " 10 topics, 50 intents, 2000 Questions but this process here with 500."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Kfolds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a classifier and get it to classify every quesstion in the training data. \n",
    "\n",
    "What it gets wrong we look at to see if our original classificationw as wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Make five folds\n",
    "#shuffle the data\n",
    "df = shuffle(df, random_state=0)\n",
    "df = df.dropna()\n",
    "numFolds=5\n",
    "size=math.trunc(len(df)/numFolds)\n",
    "#MAke the flds 5 folds\n",
    "Fold1=df[:size]\n",
    "Fold2=df[size:(size*2)]\n",
    "Fold3=df[(size*2):(size*3)]\n",
    "Fold4=df[(size*3):(size*4)]\n",
    "Fold5=df[(size*4):]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### now we have folds define a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')),# tokenizer=LemmaTokenizer() ,ngram_range=(1, 2)  ,stop_words=stopWords ngram_range=(1, 1)\n",
    "                    ('clf',  SGDClassifier(loss='modified_huber', penalty='l2',alpha=1e-3, tol=0.001, max_iter=1000, random_state=42)),#n_iter=20 'hinge'\n",
    "                    #('clf',  MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)),\n",
    "                    ]) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy is 0.7130434782608696\n"
     ]
    }
   ],
   "source": [
    "#This is what makes train smaller\n",
    "train = pd.concat([Fold2, Fold3, Fold4,Fold5]) \n",
    "text_clf.fit(train.Question, train.Intent)\n",
    "\n",
    "\n",
    "df1 = pd.DataFrame(columns=['Question', 'GTIntent', 'Prediction 1','Conf 1','Pred2','conf2','match'])\n",
    "\n",
    "total=0\n",
    "\n",
    "\n",
    "def runFold(test):\n",
    "    i=0\n",
    "    global total,df1\n",
    "\n",
    "    for row in test.iterrows():\n",
    "        row2=[test.iloc[i,0]]\n",
    "        #row2=[row]\n",
    "        predicted=text_clf.predict(row2)\n",
    "        values= text_clf.predict_proba(row2)\n",
    "        best_n = np.argsort(values, axis=1)[:,-3:]\n",
    "    \n",
    "        if test.iloc[i,1]==predicted[0]:\n",
    "                match=1\n",
    "                total=total+1\n",
    "        else:\n",
    "                match=0\n",
    "            \n",
    "\n",
    "        df1 = df1.append({'Question' : row2[0] , 'GTIntent' : test.iloc[i,1],'Prediction 1':text_clf.classes_[best_n.item(2)],'Conf 1':math.trunc(values[0][best_n.item(2)]*100),'Pred2':text_clf.classes_[best_n.item(1)],'conf2':math.trunc(values[0][best_n.item(1)]*100),'match':match} , ignore_index=True)#,'':,'':,'':,'':,'':\n",
    "\n",
    "\n",
    "        i=i+1\n",
    "\n",
    "        \n",
    "train = pd.concat([Fold2, Fold3, Fold4,Fold5]) \n",
    "text_clf.fit(train.Question, train.Intent)        \n",
    "runFold(Fold1)\n",
    "train = pd.concat([Fold1, Fold3, Fold4,Fold5]) \n",
    "text_clf.fit(train.Question, train.Intent)        \n",
    "runFold(Fold2)\n",
    "train = pd.concat([Fold1, Fold2, Fold4,Fold5]) \n",
    "text_clf.fit(train.Question, train.Intent)        \n",
    "runFold(Fold3)\n",
    "train = pd.concat([Fold1, Fold2, Fold3,Fold5]) \n",
    "text_clf.fit(train.Question, train.Intent)        \n",
    "runFold(Fold4)\n",
    "train = pd.concat([Fold1, Fold2, Fold3,Fold4]) \n",
    "text_clf.fit(train.Question, train.Intent)        \n",
    "runFold(Fold5)\n",
    "\n",
    "\n",
    "print(\"accuracy is \"+str(total/(len(Fold1)*5)))\n",
    "df1.to_excel('checkThese.xlsx', encoding='utf-8', index=False)#\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## save data in a csv \n",
    "## Date, Accuracy, intent numbers, question numbers\n",
    "import datetime as dt\n",
    "now=dt.datetime.today().strftime(\"%m/%d/%Y\")\n",
    "\n",
    "with open(\"result_statistics.csv\", \"a\") as myfile:\n",
    "    myfile.write(now+\",\"+str(df['Intent'].nunique())+\",\"+str(len(df))+\",\"+str(math.trunc((total/(len(Fold1)*5)*100)))+\"\\n\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![rabbit eating](img/coco2.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find Errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three kinds of errors\n",
    "\n",
    "Duplicates\n",
    "\n",
    "Confident and wrong\n",
    "\n",
    "Second Confidence High\n",
    "\n",
    "Low confidence nonesense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>GTIntent</th>\n",
       "      <th>Prediction 1</th>\n",
       "      <th>Conf 1</th>\n",
       "      <th>Pred2</th>\n",
       "      <th>conf2</th>\n",
       "      <th>match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>Ba mhaith liom coinin. Ach nil aon airgead agam</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>31</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>has the kit had to go see a vet?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>Did the rabbit have to visit a vet?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>Do you verify that the kit does not have fleas?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>Can you guarantee the rabbit doesn't have worms?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>50</td>\n",
       "      <td>BREEDER_GUARANTEE</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Question         GTIntent  \\\n",
       "111   Ba mhaith liom coinin. Ach nil aon airgead agam  PURCHASE_RABBIT   \n",
       "112                  has the kit had to go see a vet?    RABBIT_HEALTH   \n",
       "113               Did the rabbit have to visit a vet?    RABBIT_HEALTH   \n",
       "114   Do you verify that the kit does not have fleas?    RABBIT_HEALTH   \n",
       "115  Can you guarantee the rabbit doesn't have worms?    RABBIT_HEALTH   \n",
       "\n",
       "             Prediction 1 Conf 1                 Pred2 conf2 match  \n",
       "111         RABBIT_HEALTH     31    BREEDER_REFERENCES    19     0  \n",
       "112         RABBIT_HEALTH    100  RABBIT_SOCIALISATION     0     1  \n",
       "113         RABBIT_HEALTH    100  RABBIT_SOCIALISATION     0     1  \n",
       "114  RABBIT_SOCIALISATION     11        RABBIT_PARENTS    11     0  \n",
       "115         RABBIT_HEALTH     50     BREEDER_GUARANTEE    50     0  "
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Intent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>Any rabbits to sell?</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>Any rabbits to sell?</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Question           Intent\n",
       "111  Any rabbits to sell?  PURCHASE_RABBIT\n",
       "113  Any rabbits to sell?  PURCHASE_RABBIT"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dupes = pd.DataFrame()\n",
    "dupes=df[df.duplicated('Question', keep=False)]\n",
    "dupes = dupes.sort_values('Question',ascending=False)\n",
    "\n",
    "htmld = dupes.to_html()\n",
    "\n",
    "dupes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![dog](img/dog.jpg) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>GTIntent</th>\n",
       "      <th>Prediction 1</th>\n",
       "      <th>Conf 1</th>\n",
       "      <th>Pred2</th>\n",
       "      <th>conf2</th>\n",
       "      <th>match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What happens if the rabbit  was not well?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Do you have a list of other people who bought ...</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>What shots has the rabbit gotten?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Do rabbits need a health cert</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Has the rabbit ever been unhealthy?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>Has the kit been cleared by a vet?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>Would it be possible to get other buyers detai...</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>Do rabbits get health innoculations and vaccines?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>I will buy the kit if you let me veterianian s...</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>Do you verify that the kit does not have fleas?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Much for a rabbit?</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>98</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Question              GTIntent  \\\n",
       "3            What happens if the rabbit  was not well?         RABBIT_HEALTH   \n",
       "5    Do you have a list of other people who bought ...    BREEDER_REFERENCES   \n",
       "30                   What shots has the rabbit gotten?         RABBIT_HEALTH   \n",
       "34                       Do rabbits need a health cert  RABBIT_CERTIFICATION   \n",
       "35                 Has the rabbit ever been unhealthy?         RABBIT_HEALTH   \n",
       "54                  Has the kit been cleared by a vet?  RABBIT_CERTIFICATION   \n",
       "90   Would it be possible to get other buyers detai...    BREEDER_REFERENCES   \n",
       "92   Do rabbits get health innoculations and vaccines?         RABBIT_HEALTH   \n",
       "94   I will buy the kit if you let me veterianian s...         RABBIT_HEALTH   \n",
       "113    Do you verify that the kit does not have fleas?         RABBIT_HEALTH   \n",
       "51                                  Much for a rabbit?       PURCHASE_RABBIT   \n",
       "\n",
       "             Prediction 1 Conf 1                 Pred2 conf2 match  \n",
       "3         PURCHASE_RABBIT    100  RABBIT_SOCIALISATION     0     0  \n",
       "5    RABBIT_SOCIALISATION    100        RABBIT_PARENTS     0     0  \n",
       "30        PURCHASE_RABBIT    100  RABBIT_SOCIALISATION     0     0  \n",
       "34          RABBIT_HEALTH    100  RABBIT_SOCIALISATION     0     0  \n",
       "35        PURCHASE_RABBIT    100  RABBIT_SOCIALISATION     0     0  \n",
       "54          RABBIT_HEALTH    100  RABBIT_SOCIALISATION     0     0  \n",
       "90         RABBIT_PARENTS    100  RABBIT_SOCIALISATION     0     0  \n",
       "92   RABBIT_CERTIFICATION    100  RABBIT_SOCIALISATION     0     0  \n",
       "94        PURCHASE_RABBIT    100  RABBIT_SOCIALISATION     0     0  \n",
       "113  RABBIT_CERTIFICATION    100  RABBIT_SOCIALISATION     0     0  \n",
       "51          RABBIT_HEALTH     98   BREEDER_INFORMATION     1     0  "
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find high confidence wrong\n",
    "highWrong = df1[(df1['Conf 1'] > 95) & (df1['match'] ==0)]\n",
    "highWrong=highWrong.sort_values(by=['Conf 1'],ascending=False)\n",
    "\n",
    "htmlhw = highWrong.to_html()\n",
    "\n",
    "highWrong    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![two at once](img/maxresdefault.jpg) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>GTIntent</th>\n",
       "      <th>Prediction 1</th>\n",
       "      <th>Conf 1</th>\n",
       "      <th>Pred2</th>\n",
       "      <th>conf2</th>\n",
       "      <th>match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>Can you tell me more about why you breed these...</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>50</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Can I meet the parents and what rabbit food br...</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>50</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>I would like to see the mother</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>50</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>How does the rabbit handle strange and new peo...</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>50</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>did a vet ever have to look at the bunny?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>50</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>What breed is the baby rabbits mother?</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>50</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Is the kit certified by a vet?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>51</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>Has a vet certified the rabbit ?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>52</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Does the rabbit have a certificate? And what f...</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>54</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Do you have any kind of guarantee with the kit?</td>\n",
       "      <td>BREEDER_GUARANTEE</td>\n",
       "      <td>BREEDER_GUARANTEE</td>\n",
       "      <td>55</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>44</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Do I need to get innoculations?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>57</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>Will my rabbit need booster vacinations</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>61</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>Can you guarantee the rabbit doesn't have worms?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>BREEDER_GUARANTEE</td>\n",
       "      <td>62</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>Has the rabbit  ever been sick?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>66</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>Can I visit the mother?</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>66</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Is there a partiular food I need for the kit?</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>66</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Do you check the rabbit for fleas or worms?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>71</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>Is the rabbit  under a limited registration? I...</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>72</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>What references do you have?</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>37</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>I'd like to talk to another owner, is that pos...</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>42</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Can you give me any references?</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>37</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>What is your mission statement?</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>40</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>How many vaccines has the rabbit had for its h...</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>78</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Question              GTIntent  \\\n",
       "102  Can you tell me more about why you breed these...   BREEDER_INFORMATION   \n",
       "36   Can I meet the parents and what rabbit food br...        RABBIT_PARENTS   \n",
       "79                      I would like to see the mother        RABBIT_PARENTS   \n",
       "66   How does the rabbit handle strange and new peo...  RABBIT_SOCIALISATION   \n",
       "46           did a vet ever have to look at the bunny?         RABBIT_HEALTH   \n",
       "55              What breed is the baby rabbits mother?        RABBIT_PARENTS   \n",
       "32                      Is the kit certified by a vet?  RABBIT_CERTIFICATION   \n",
       "60                    Has a vet certified the rabbit ?  RABBIT_CERTIFICATION   \n",
       "11   Does the rabbit have a certificate? And what f...  RABBIT_CERTIFICATION   \n",
       "24     Do you have any kind of guarantee with the kit?     BREEDER_GUARANTEE   \n",
       "65                    Do I need to get innoculations?          RABBIT_HEALTH   \n",
       "59             Will my rabbit need booster vacinations         RABBIT_HEALTH   \n",
       "114   Can you guarantee the rabbit doesn't have worms?         RABBIT_HEALTH   \n",
       "81                     Has the rabbit  ever been sick?         RABBIT_HEALTH   \n",
       "56                             Can I visit the mother?        RABBIT_PARENTS   \n",
       "39       Is there a partiular food I need for the kit?           RABBIT_FOOD   \n",
       "14         Do you check the rabbit for fleas or worms?         RABBIT_HEALTH   \n",
       "78   Is the rabbit  under a limited registration? I...   BREEDER_INFORMATION   \n",
       "17                        What references do you have?    BREEDER_REFERENCES   \n",
       "84   I'd like to talk to another owner, is that pos...    BREEDER_REFERENCES   \n",
       "1                      Can you give me any references?    BREEDER_REFERENCES   \n",
       "40                     What is your mission statement?   BREEDER_INFORMATION   \n",
       "97   How many vaccines has the rabbit had for its h...         RABBIT_HEALTH   \n",
       "\n",
       "             Prediction 1 Conf 1                 Pred2 conf2 match  \n",
       "102         RABBIT_HEALTH     50   BREEDER_INFORMATION    50     1  \n",
       "36         RABBIT_PARENTS     50           RABBIT_FOOD    50     0  \n",
       "79         RABBIT_PARENTS     50           RABBIT_FOOD    50     1  \n",
       "66   RABBIT_SOCIALISATION     50  RABBIT_CERTIFICATION    50     1  \n",
       "46          RABBIT_HEALTH     50       PURCHASE_RABBIT    50     1  \n",
       "55    BREEDER_INFORMATION     50        RABBIT_PARENTS    49     0  \n",
       "32   RABBIT_CERTIFICATION     51         RABBIT_HEALTH    48     1  \n",
       "60   RABBIT_CERTIFICATION     52         RABBIT_HEALTH    47     1  \n",
       "11        PURCHASE_RABBIT     54           RABBIT_FOOD    45     0  \n",
       "24      BREEDER_GUARANTEE     55         RABBIT_HEALTH    44     1  \n",
       "65          RABBIT_HEALTH     57   BREEDER_INFORMATION    42     1  \n",
       "59          RABBIT_HEALTH     61   BREEDER_INFORMATION    38     1  \n",
       "114     BREEDER_GUARANTEE     62         RABBIT_HEALTH    37     0  \n",
       "81          RABBIT_HEALTH     66       PURCHASE_RABBIT    33     1  \n",
       "56         RABBIT_PARENTS     66         RABBIT_HEALTH    33     1  \n",
       "39            RABBIT_FOOD     66         RABBIT_HEALTH    33     1  \n",
       "14          RABBIT_HEALTH     71       PURCHASE_RABBIT    28     1  \n",
       "78    BREEDER_INFORMATION     72       PURCHASE_RABBIT    27     1  \n",
       "17          RABBIT_HEALTH     37    BREEDER_REFERENCES    22     0  \n",
       "84            RABBIT_FOOD     42         RABBIT_HEALTH    22     0  \n",
       "1           RABBIT_HEALTH     37    BREEDER_REFERENCES    22     0  \n",
       "40          RABBIT_HEALTH     40       PURCHASE_RABBIT    21     0  \n",
       "97   RABBIT_CERTIFICATION     78         RABBIT_HEALTH    21     0  "
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find intents where second intent is high confidence\n",
    "\n",
    "doubleq = df1[df1.conf2 > 20]\n",
    "doubleq=doubleq.sort_values(by=['conf2'],ascending=False)\n",
    "htmldq = doubleq.to_html()\n",
    "\n",
    "doubleq    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why two questions at once?\n",
    "  -  Customers see one task when business sees two. 'I want to cancel my card and get a new one'\n",
    "  -  People complain about two things at once. 'The food is terrible and the portions are too small'\n",
    "  -  Chatbot is not clear enough to ask one question at a time. This is the big reason to admit you are a chatbot\n",
    "  -  Just asked two questions when they shouldnt have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![rabbit eating](img/0000049339.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>GTIntent</th>\n",
       "      <th>Prediction 1</th>\n",
       "      <th>Conf 1</th>\n",
       "      <th>Pred2</th>\n",
       "      <th>conf2</th>\n",
       "      <th>match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>xxxxxxxxxxxxxx</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>27</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>Ba mhaith liom coinin. Ach nil aon airgead agam</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>31</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Can you give me any references?</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>23</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What is your mission statement?</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>23</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>What references do you have?</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>23</td>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Do rabbits get health documentation</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Do rabbits have parental bloodlines?</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>What food does the rabbit like to eat?</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Do rabbits need a health cert</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>Will my rabbit need booster vacinations</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>Can my vet examine the baby rabbits?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>Has the kit been around other rabbits and people?</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>Do you verify that the kit does not have fleas?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Question              GTIntent  \\\n",
       "36                                      xxxxxxxxxxxxxx        RABBIT_PARENTS   \n",
       "111    Ba mhaith liom coinin. Ach nil aon airgead agam       PURCHASE_RABBIT   \n",
       "1                      Can you give me any references?    BREEDER_REFERENCES   \n",
       "7                      What is your mission statement?   BREEDER_INFORMATION   \n",
       "17                        What references do you have?    BREEDER_REFERENCES   \n",
       "4                  Do rabbits get health documentation  RABBIT_CERTIFICATION   \n",
       "28                Do rabbits have parental bloodlines?        RABBIT_PARENTS   \n",
       "29              What food does the rabbit like to eat?           RABBIT_FOOD   \n",
       "34                       Do rabbits need a health cert  RABBIT_CERTIFICATION   \n",
       "59             Will my rabbit need booster vacinations         RABBIT_HEALTH   \n",
       "76                Can my vet examine the baby rabbits?         RABBIT_HEALTH   \n",
       "92   Has the kit been around other rabbits and people?  RABBIT_SOCIALISATION   \n",
       "114    Do you verify that the kit does not have fleas?         RABBIT_HEALTH   \n",
       "\n",
       "             Prediction 1 Conf 1               Pred2 conf2 match  \n",
       "36          RABBIT_HEALTH     27     PURCHASE_RABBIT    26     0  \n",
       "111         RABBIT_HEALTH     31  BREEDER_REFERENCES    19     0  \n",
       "1           RABBIT_HEALTH     23     PURCHASE_RABBIT    18     0  \n",
       "7           RABBIT_HEALTH     23     PURCHASE_RABBIT    18     0  \n",
       "17          RABBIT_HEALTH     23     PURCHASE_RABBIT    18     0  \n",
       "4    RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "28   RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "29   RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "34   RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "59   RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "76   RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "92   RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  \n",
       "114  RABBIT_SOCIALISATION     11      RABBIT_PARENTS    11     0  "
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find intents that are low confidence and wrong. Might be nonsense\n",
    "\n",
    "verylow = df1[(df1['Conf 1'] < 36) & (df1['match'] ==0)]\n",
    "verylow=verylow.sort_values(by=['conf2'],ascending=False)\n",
    "dflc = verylow.to_html()\n",
    "verylow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"all.html\", 'w') as _file:\n",
    "    _file.write( \"<h1>Duplicates</h1>\"+htmld + \"\\n\\n<h1>Double Questions</h1>\" + htmldq+ \"\\n\\n<h1>High Confidence Wrong</h1>\"+ htmlhw+ \"\\n\\n<h1>Could be Nonsense</h1>\"+ dflc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Find Bad Intents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![rabbit eating](img/bunny2.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wrong = df1[(df1['GTIntent']!=df1['Prediction 1'])]\n",
    "correct = df1[(df1['GTIntent']==df1['Prediction 1'])]\n",
    "\n",
    "wf=wrong.groupby(['GTIntent'], as_index=False).count()\n",
    "crf=correct.groupby(['GTIntent'], as_index=False).count()\n",
    "gf1=wf[['GTIntent','Question']]\n",
    "gf1.columns = ['Intent', 'wrong']\n",
    "\n",
    "\n",
    "crf1=crf[['GTIntent','Question']]\n",
    "crf1.columns = ['Intent', 'right']\n",
    "#crf1.head()\n",
    "result = pd.merge(crf1,gf1,on='Intent',how='outer')\n",
    "result=result.fillna(0)\n",
    "#total questions\n",
    "result['total'] = result['wrong'] + result['right']\n",
    "#calculate percentage we got right\n",
    "result['percent'] = (result['right']/(result['wrong'] + result['right']))*100\n",
    "\n",
    "result=result.round(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Intent</th>\n",
       "      <th>right</th>\n",
       "      <th>wrong</th>\n",
       "      <th>total</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>36</td>\n",
       "      <td>8</td>\n",
       "      <td>44</td>\n",
       "      <td>82.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>16</td>\n",
       "      <td>81.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>75.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PURCHASE_RABBIT</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>60.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>29.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Intent  right  wrong  total  percent\n",
       "6         RABBIT_HEALTH     36      8     44     82.0\n",
       "7        RABBIT_PARENTS     13      3     16     81.0\n",
       "1   BREEDER_INFORMATION      9      3     12     75.0\n",
       "3       PURCHASE_RABBIT      6      4     10     60.0\n",
       "4  RABBIT_CERTIFICATION      4      4      8     50.0\n",
       "2    BREEDER_REFERENCES      2      5      7     29.0\n",
       "5           RABBIT_FOOD      2      4      6     33.0"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lowAcc = result[(result.percent < 100) & (result['wrong'] >1)]\n",
    "lowAcc=lowAcc.sort_values(by=['total'],ascending=False)\n",
    "lowAcc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Low and right should be looked at](img/fixthese2.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you fix these basic misclassifications the next common issue is mixed up intents.\n",
    "\n",
    "                  Actual\n",
    "             Intent1 Intent2 ...     \n",
    "     Intent1  \n",
    "     Intets2      \n",
    "     ...  \n",
    "       \n",
    "predicted     \n",
    "       \n",
    "       \n",
    "       \n",
    "  \n",
    " Open checkthese spreadsheet and select two intents that are mixes up\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>GTIntent</th>\n",
       "      <th>Prediction 1</th>\n",
       "      <th>Conf 1</th>\n",
       "      <th>Pred2</th>\n",
       "      <th>conf2</th>\n",
       "      <th>match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the size of the parents?</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Has the litter been not well?</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>RABBIT_HEALTH</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What does the rabbit  like to eat?</td>\n",
       "      <td>RABBIT_FOOD</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Do you have details on your breeding business?</td>\n",
       "      <td>BREEDER_INFORMATION</td>\n",
       "      <td>BREEDER_REFERENCES</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Do rabbits get health documentation</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         Question              GTIntent  \\\n",
       "0               What are the size of the parents?        RABBIT_PARENTS   \n",
       "1                   Has the litter been not well?         RABBIT_HEALTH   \n",
       "2              What does the rabbit  like to eat?           RABBIT_FOOD   \n",
       "3  Do you have details on your breeding business?   BREEDER_INFORMATION   \n",
       "4             Do rabbits get health documentation  RABBIT_CERTIFICATION   \n",
       "\n",
       "           Prediction 1 Conf 1                 Pred2 conf2 match  \n",
       "0        RABBIT_PARENTS    100  RABBIT_SOCIALISATION     0     1  \n",
       "1         RABBIT_HEALTH    100  RABBIT_SOCIALISATION     0     1  \n",
       "2  RABBIT_SOCIALISATION     11        RABBIT_PARENTS    11     0  \n",
       "3    BREEDER_REFERENCES    100  RABBIT_SOCIALISATION     0     0  \n",
       "4  RABBIT_SOCIALISATION     11        RABBIT_PARENTS    11     0  "
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def get_confusion_matrix(y_true, y_pred, labels, normalize=False):\n",
    "    conf_matrix = metrics.confusion_matrix(y_true, y_pred, labels)\n",
    "    if normalize:\n",
    "        conf_matrix = conf_matrix.astype('float')/conf_matrix.sum(axis=1)[:, np.newaxis]\n",
    "    conf = pd.DataFrame(conf_matrix, index=labels, columns=labels)\n",
    "    return conf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "conf=df1#res[0]\n",
    "cf = get_confusion_matrix(conf['GTIntent'], conf['Prediction 1'], labels=conf['GTIntent'].unique())\n",
    "#df['Diag'] =pd.Series(np.diag(cf), index=[cf.index, cf.columns])\n",
    "cf.tail()\n",
    "\n",
    "conf = cf.to_csv('Confusion.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![rabbit eating](img/baby-bunny.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have a classifier use it to classify new data. Then you can go through this labeled data and agree/disagree with it. This is much faster than labeling data yourself cold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Do the baby rabbits get a certificate?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What cert do the kits have?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Does the rabbit  have a certificate?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Is the kit approved by a vet?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Has the kit been given the OK by a vet?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Has a veterinarian certified the rabbit ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    Question\n",
       "0     Do the baby rabbits get a certificate?\n",
       "1                What cert do the kits have?\n",
       "2       Does the rabbit  have a certificate?\n",
       "3              Is the kit approved by a vet?\n",
       "4    Has the kit been given the OK by a vet?\n",
       "5  Has a veterinarian certified the rabbit ?"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filePath = r'tolabel.csv'\n",
    "\n",
    "label = pd.read_csv(filePath,header=0)\n",
    "\n",
    "#label=label.drop(['Unnamed: 0'], axis=1)\n",
    "label = label.dropna()        \n",
    "label.head(20)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 2), preprocessor=None, stop_words=None,\n",
       "        strip...om_state=42, shuffle=True, tol=0.001,\n",
       "       validation_fraction=0.1, verbose=0, warm_start=False))])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#train = pd.concat([trainSent]) \n",
    "text_clf.fit(df.Question, df.Intent)        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label all the new data. And then go through it by intent agreeing or disagreeing with the label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labeled = pd.DataFrame(columns=['Question', 'Prediction 1','Conf 1','Pred2','conf2'])\n",
    "i=0\n",
    "\n",
    "\n",
    "for row in label.iterrows():\n",
    "    row2=[label.iloc[i,0]]\n",
    "    predicted=text_clf.predict(row2)\n",
    "    values= text_clf.predict_proba(row2)\n",
    "    best_n = np.argsort(values, axis=1)[:,-3:]            \n",
    "    labeled = labeled.append({'Question' : row2[0] ,'Prediction 1':text_clf.classes_[best_n.item(2)],'Conf 1':math.trunc(values[0][best_n.item(2)]*100),'Pred2':text_clf.classes_[best_n.item(1)],'conf2':math.trunc(values[0][best_n.item(1)]*100)} , ignore_index=True)\n",
    "    i=i+1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Prediction 1</th>\n",
       "      <th>Conf 1</th>\n",
       "      <th>Pred2</th>\n",
       "      <th>conf2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What cert do the kits have?</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Does the rabbit  have a certificate?</td>\n",
       "      <td>RABBIT_CERTIFICATION</td>\n",
       "      <td>100</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Is the kit approved by a vet?</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Has the kit been given the OK by a vet?</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Has a veterinarian certified the rabbit ?</td>\n",
       "      <td>RABBIT_SOCIALISATION</td>\n",
       "      <td>11</td>\n",
       "      <td>RABBIT_PARENTS</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    Question          Prediction 1 Conf 1  \\\n",
       "1                What cert do the kits have?  RABBIT_SOCIALISATION     11   \n",
       "2       Does the rabbit  have a certificate?  RABBIT_CERTIFICATION    100   \n",
       "3              Is the kit approved by a vet?  RABBIT_SOCIALISATION     11   \n",
       "4    Has the kit been given the OK by a vet?  RABBIT_SOCIALISATION     11   \n",
       "5  Has a veterinarian certified the rabbit ?  RABBIT_SOCIALISATION     11   \n",
       "\n",
       "                  Pred2 conf2  \n",
       "1        RABBIT_PARENTS    11  \n",
       "2  RABBIT_SOCIALISATION     0  \n",
       "3        RABBIT_PARENTS    11  \n",
       "4        RABBIT_PARENTS    11  \n",
       "5        RABBIT_PARENTS    11  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labeled.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Below if we have time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy of intents graphs and tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>intents</th>\n",
       "      <th>questions</th>\n",
       "      <th>accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10/03/2019</td>\n",
       "      <td>2</td>\n",
       "      <td>13000</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10/07/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>500</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10/07/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>404</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10/07/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>320</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10/07/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>504</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  intents  questions  accuracy\n",
       "0  10/03/2019        2      13000        70\n",
       "1  10/07/2019        3        500        67\n",
       "2  10/07/2019        3        404        66\n",
       "3  10/07/2019        3        320        65\n",
       "4  10/07/2019        3        504        70"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "#date,intents,questions,accuracy\n",
    "filePath = r'result_statistics.csv'\n",
    "plotdata = pd.read_csv(filePath,header=0)\n",
    "\n",
    "#df = df.iloc[1:]\n",
    "plotdata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from random import random\n",
    "colors=[]\n",
    "when = plotdata.iloc[:,0]\n",
    "#make a random list of colors\n",
    "for i in enumerate(when):\n",
    "    nextC=random(),random(),random()\n",
    "    colors.append(nextC)\n",
    "    \n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfZUlEQVR4nO3de7xVVb338c8XkIsXBGWDiJediqZWUmfnJc1Ss9Aueo6SmiWaRh3TrDwl9tRTvbo81uli52gWaYapJWqGWalEUkczE0xNQUU4qCjCRkW5GAj8nj/GWO7Jdl8Wmz3X2hu+79drvda8/uaYc601f3OMMddaigjMzGzL1qfeBTAzs/pzMjAzMycDMzNzMjAzM5wMzMwMJwMzM8PJwGpI0ghJf5a0XNJ3612ejSEpJO1V73JY95L0FUlX17scPYGTQY1ImiHpBUkD6l2WMimZL2l2G7MnAEuBwRFxfhkfxBzzFUkrCo9l3bmNKsvgE0yVJDXmZNuvk+X2lnS9pKWSXpT0oKTPSupbq7JuzpwMakBSI/B2IIAP1HjbHX7ASnA4MBzYQ9JbW83bHZgd3fRNxw727bqI2LbwGNId2+uJtpQToaQ9gXuAp4A3RsT2wDigCdiunmXbXDgZ1MZpwF+BnwHjizMkDZL0XUlP5KudOyUNyvMOk/QXScskPSXp9Dx9hqSzCjFOl3RnYTwkfVLSXGBunvaDHOMlSbMkvb2wfF9JX5A0LzfhzJK0q6RLWzfnSPqNpE93sK/jganA74r7Kqmy75/PV+vvA74AnJTHH8jLbS/pCkmLJD0t6euVE17ez7skfV/S88BXqjj2G+jKcSis/i5Jc3MN71JJqnKbIensvO5ySV+TtKeku3M5pkjqn5d9p6SFuRxLJS2QdGrxOEq6TNLvJK0EjsjH7CpJzfl99EVJfSQNyO+dNxTWb5D0sqThefx9ku7Py/1F0psKyy6Q9Ll8Bb4yvy4jJP0+78cfJA0tLH9w4f36gKR3FubNyPt9V173dknD8uw/5+dl+b1wSBuH8avAXyLisxGxCCAiHo2ID0XEMkm/lXRuq+P+oKTj8/D+kqZJel7SYklfaOe1ancfNnsR4UfJD+Bx4GzgX4BXgBGFeZcCM4BRQF/gbcAAYDdgOXAKsBWwIzAmrzMDOKsQ43TgzsJ4ANOAHYBBedqHc4x+wPnAs8DAPO9zwD+AfQABB+RlDwSeAfrk5YYBq4rlb7WfWwMvAccCJ5CahPoX5v8M+Hph/CvA1a1i/Br4MbANqYbxN+Djhf1cC5yb92NQG2V4TcxW8zf6OBSO6S3AkPzaNANj29nGBmXI694MDAb2B1YD04E9gO2B2cD4vOw78z5+L78P3gGsBPYpHMMXgUNJF3MDgatICXg7oBF4DDgzL/9T4BuFsnwSuDUPvwVYAhxEeu+NBxYAA/L8BaSLmBGk9+cS4D7gzblsfwS+nJcdBTyXX/s+wNF5vKHwnp0H7A0MyuMX5XmN+Rj16+B1exY4o4P5HwTuKYwfkLffPx+XRfn1HpjHD2r9WnW2D5v7o+4F2NwfwGGkBDAsjz8CfCYP9wFeBg5oY70LgZvaiTmDzpPBkZ2U64XKdoFHgePaWW4OcHQePgf4XQcxP0w6SfbLJ4tlwL8W5v+MDpJBPumspnCSJyXDOwr7+WQn+/UVYE3eduVxRzcchwAOK4xPASZ2UIbWyeDQwvgs4ILC+HeBi/PwO0nJYJtW2/pS4RheVZjXNx+z/QrTPg7MyMPvAuYX5t0FnJaHLwO+1qrsjwLvyMMLgFML824ELiuMnwv8Og9fAPy8VazbaElyM4AvFuadTUtSaqTzZPAK7STfPH8A8DwwOo9/B/hh4T30985eq872YXN/uJmofOOB2yNiaR6/lpbmk2GkK5V5bay3azvTq/VUcUTS+ZLmKDVFLSNdkVaq6R1tazLpJE9+/nkH2xwPTImItRGxGvgVrZrFOrE7qRa0KFfTl5FqCcPb2692TImIIYXHEZUZm3AcIF2dVqwCtq2iLBWLC8MvtzFejPVCRKwsjD8B7FwYLx6DYaSr3ydaLT8qD/8RGCTpIEm7A2OAm/K83YHzK8c6H49dW22r2nLvDoxrFeswYGRh+U05fs+1irWB/H6bAnxYUh9SAqi8V6v9LFWzD5utWncublGU2v4/CPSVVPkgDACGSDqA1CTxT2BP4IFWqz9FaqZpy0pSk0zFTm0s82onbW4XvwA4Cng4ItZLeoHUFFLZ1p7AQ23EuRp4KJd3X1IzzmtI2gU4EjhQ0gl58tbAQEnDCsmwzTIWyrGaVIta29Z22linapt4HGppqKRtCglhNzYsU/EYLCVdNe9Oam6qLP80QN7HKaST42LglohYnpd7itSE9I1uKPNTpKvqj3Vh3Wpe0z+Qmh6v7GCZyaQEcCewKiLuLpTtlCq2sSn70Ou5ZlCu44F1wH6kK7IxpBPq/5Cq6utJbbrfk7Rz7sA8ROn202tIHZYflNRP0o6SxuS49wP/JmlrpXvfz+ykHNuRmh6agX6S/i+p/bricuBrkkYreZOkHQEiYiFwL+lDdmNEvNzONj5Caqvep7CvewMLaf+DuBhozFdyROoYvB34rqTBuRN0T0nv6GT/qtXl41AHX5XUPyew9wHXt7VQRKwjXRF/Q9J2+er/s6QkXnEtcBJwah6u+AnwiVxrkKRtJL1XUlfuzrkaeL+k9+T38UClzvBdqli3GVhP6kNpz5eBt0n6T0k7AUjaS9LVkoYA5JP/elKzW7EGewuwk6RPK3WqbyfpoG7eh17PyaBc44ErI+LJiHi28gAuAU5VujXyP0g1hHtJbZ7fInXYPknqyDo/T7+f1CkG8H1Su/hi0tXQNZ2U4zbg96ST9ROk2kixqeF7pBPK7aQO4CtInXwVk4E30nkT0Q+L+5n39Ue031RUOcE9J+m+PHwaqdljNqk9/wY2vppeuUOp+BjOph+HWnmWtO/PkF7bT0TEIx0sfy6ptjifdFV8LekiA4CIuCfP35m0/5XpM4GPkd6PL5BudDi9KwWOiKeA40h3iDWTjuvnqOIcExGrgG8Ad+XmmYPbWGYecAipf+FhSS+S+jBmkm60qLiK9F69urDuclJn8PtJx3YucAStbMo+bA6UO0nM2iXpcNKHqzHXZqwk+VbGqyNii7ga7W6STgMmRMRh9S5Lb7NFZDzrOklbAecBlzsRWE8maWvSXUqT6l2W3sjJwNolaV/SrZkjgYvrXByzdkl6D6lpZzEb9otYldxMZGZmrhmYmVkv+Z7BsGHDorGxsd7FMDPrVWbNmrU0IhqqWbZXJIPGxkZmzpxZ72KYmfUqkp7ofKnEzURmZuZkYGZmTgZmZoaTgZmZ4WRgZmY4GZiZGSUmA0n7KP23auXxUv4J2R2U/ot0bn4e2nk0MzMrU2nJINKfVY+JiDGk//5dRfqHpYnA9IgYTfof2IlllcHMzKpTq2aio4B5EfEE6ffCJ+fpk0l/AGNmZnVUq28gnwz8Ig+PyP9oRUQsyn868hqSJgATAHbbbbcub/jyZ6/r8rpFZ+10UrfE2dxM/tpd3RJn/JcO7ZY49fL/Fv+kW+JcOGKL/MdF6wFKrxlI6g98gHb+tq89ETEpIpoioqmhoaqf1jAzsy6qRTPRMcB9EbE4jy+WNBIgPy+pQRnMzKwDtUgGp9DSRARwMy3/iTsemFqDMpiZWQdKTQb5b+iOBn5VmHwRcLSkuXneRWWWwczMOldqB3JErAJ2bDXtOdLdRWZm1kP0iv8zsC3Pr/92Y7fEOf7AEzYYn3fP090Sd8+DRnVLHLOewj9HYWZmTgZmZuZmoi6bdtut3RLn6PeM7ZY4ZmabwjUDMzNzMjAzMzcT9ThX3f5Et8Q57d27d0scM9syuGZgZmZOBmZm5maiLca1S1/qljgfGja4W+JYdZp/+9VuidPw3i93SxzbfLlmYGZmTgZmZuZkYGZmOBmYmRlOBmZmhpOBmZnhZGBmZjgZmJkZTgZmZoaTgZmZUXIykDRE0g2SHpE0R9IhknaQNE3S3Pw8tMwymJlZ58quGfwAuDUiXg8cAMwBJgLTI2I0MD2Pm5lZHZWWDCQNBg4HrgCIiDURsQw4DpicF5sMHF9WGczMrDpl1gz2AJqBKyX9XdLlkrYBRkTEIoD8PLytlSVNkDRT0szm5uYSi2lmZmUmg37AW4DLIuLNwEo2okkoIiZFRFNENDU0NJRVRjMzo9xksBBYGBH35PEbSMlhsaSRAPl5SYllMDOzKpT25zYR8aykpyTtExGPAkcBs/NjPHBRfp5aVhnMzDoza93LmxzjX/oO6oaS1FfZ/3R2LnCNpP7AfOAMUm1kiqQzgSeBcSWXwczMOlFqMoiI+4GmNmYdVeZ2zcxs4/gbyGZm5mRgZmZOBmZmhpOBmZnhZGBmZjgZmJkZTgZmZoaTgZmZ4WRgZmaU/3MUZmbd4tZFt25yjLEjx3ZDSTZPrhmYmZmTgZmZORmYmRlOBmZmhpOBmZnhu4nMzEqx7sd/2+QYfT9+YDeUpDquGZiZmZOBmZk5GZiZGU4GZmaGk4GZmVHy3USSFgDLgXXA2ohokrQDcB3QCCwAPhgRL5RZDjMz61gtagZHRMSYiGjK4xOB6RExGpiex83MrI7q0Ux0HDA5D08Gjq9DGczMrKDsZBDA7ZJmSZqQp42IiEUA+Xl4WytKmiBppqSZzc3NJRfTzGzLVvY3kA+NiGckDQemSXqk2hUjYhIwCaCpqSnKKqCZmZVcM4iIZ/LzEuAm4EBgsaSRAPl5SZllMDOzzpWWDCRtI2m7yjDwbuAh4GZgfF5sPDC1rDKYmVl1ymwmGgHcJKmynWsj4lZJ9wJTJJ0JPAmMK7EMZmZWhdKSQUTMBw5oY/pzwFFlbdfMzDaev4FsZmZOBmZm5mRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmYmRlOBmZmhpOBmZnhZGBmZjgZmJkZTgZmZkYVyUDSOZKG1qIwZmZWH9XUDHYC7pU0RdJY5T8oMDOzzUenySAivgiMBq4ATgfmSvqmpD1LLpuZmdVIVX0GERHAs/mxFhgK3CDp2yWWzczMaqTTfzqT9CnSfxUvBS4HPhcRr0jqA8wFPl9uEc3MrGzV/O3lMODfIuKJ4sSIWC/pfeUUy8zMaqmaZqLfAc9XRiRtJ+kggIiYU1bBzMysdqpJBpcBKwrjK/O0qkjqK+nvkm7J46+TdI+kuZKuk9R/44psZmbdrZpmIuUOZODV5qFq1qs4D5gDDM7j3wK+HxG/lPQj4Ew2IrmYWfe484fPbHKMw87e+TXTbnzsxU2Oe8Le229yDNs41dQM5kv6lKSt8uM8YH41wSXtAryX1PFM/o7CkcANeZHJwPEbX2wzM+tO1SSDTwBvA54GFgIHAROqjH8x6W6j9Xl8R2BZRKzN4wuBUW2tKGmCpJmSZjY3N1e5OTMz64pOm3siYglw8sYGzncaLYmIWZLeWZnc1iba2e4kYBJAU1NTm8uYmVn3qOZ7BgNJ7fr7AwMr0yPio52seijwAUnH5vUGk2oKQyT1y7WDXYBNb7g0M7NNUk0z0c9Jv0/0HuBPpBP48s5WiogLI2KXiGgk1Sz+GBGnAncAJ+bFxgNTu1BuMzPrRtUkg70i4kvAyoiYTOoQfuMmbPMC4LOSHif1IVyxCbHMzKwbVHOL6Cv5eZmkN5B+n6hxYzYSETOAGXl4PnDgxqxvZmblqiYZTMr/Z/BF4GZgW+BLpZbKzMxqqsNkkH+M7qWIeAH4M7BHTUplZmY11WGfQUSsB86pUVnMzKxOqulAnibpPyTtKmmHyqP0kpmZWc1U02dQ+T7BJwvTAjcZmZltNqr5BvLralEQMzOrn2q+gXxaW9Mj4qruL46ZmdVDNc1Eby0MDwSOAu4DnAzMzDYT1TQTnVscl7Q96ScqzMxsM1HN3UStrQJGd3dBzMysfqrpM/gNLT8z3QfYD5hSZqHMzKy2qukz+E5heC3wREQsLKk8ZmZWB9UkgyeBRRHxTwBJgyQ1RsSCUktmZmY1U02fwfW0/G0lwLo8zczMNhPVJIN+EbGmMpKH+5dXJDMzq7VqkkGzpA9URiQdBywtr0hmZlZr1fQZfAK4RtIleXwh0Oa3ks3MrHeq5ktn84CDJW0LKCI6/f9jMzPrXTptJpL0TUlDImJFRCyXNFTS12tRODMzq41q+gyOiYhllZH8r2fHllckMzOrtWqSQV9JAyojkgYBAzpY3szMeplqOpCvBqZLujKPnwFM7mwlSQNJ/5s8IG/nhoj4sqTXAb8EdiD9+ulHireumplZ7XVaM4iIbwNfB/Yl/S7RrcDuVcReDRwZEQcAY4Cxkg4GvgV8PyJGAy8AZ3ax7GZm1k2q/dXSZ0nfQj6B9H8GczpbIZIVeXSr/AjgSOCGPH0ycPzGFNjMzLpfu81EkvYGTgZOAZ4DriPdWnpEtcEl9QVmAXsBlwLzgGURsTYvshAY1c66E4AJALvttlu1mzQzsy7oqGbwCKkW8P6IOCwi/pv0u0RVi4h1ETEG2AU4kNTU9JrF2ll3UkQ0RURTQ0PDxmzWzMw2UkfJ4ARS89Adkn4i6ShAXdlIvjV1BnAwMERSpUayC/BMV2KamVn3aTcZRMRNEXES8HrSifwzwAhJl0l6d2eBJTVIGpKHBwHvIvU13AGcmBcbD0zdpD0wM7NNVs3dRCsj4pqIeB/pSv5+YGIVsUeSahUPAvcC0yLiFuAC4LOSHgd2BK7ocunNzKxbVPM9g1dFxPPAj/Ojs2UfBN7cxvT5pP4DMzPrIaq9tdTMzDZjTgZmZuZkYGZmTgZmZoaTgZmZ4WRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmYmRlOBmZmhpOBmZnhZGBmZjgZmJkZTgZmZoaTgZmZ4WRgZmY4GZiZGU4GZmZGiclA0q6S7pA0R9LDks7L03eQNE3S3Pw8tKwymJlZdcqsGawFzo+IfYGDgU9K2g+YCEyPiNHA9DxuZmZ1VFoyiIhFEXFfHl4OzAFGAccBk/Nik4HjyyqDmZlVpyZ9BpIagTcD9wAjImIRpIQBDK9FGczMrH2lJwNJ2wI3Ap+OiJc2Yr0JkmZKmtnc3FxeAc3MrNxkIGkrUiK4JiJ+lScvljQyzx8JLGlr3YiYFBFNEdHU0NBQZjHNzLZ4Zd5NJOAKYE5EfK8w62ZgfB4eD0wtqwxmZladfiXGPhT4CPAPSffnaV8ALgKmSDoTeBIYV2IZzMysCqUlg4i4E1A7s48qa7tmZrbx/A1kMzNzMjAzMycDMzPDycDMzHAyMDMznAzMzAwnAzMzw8nAzMxwMjAzM5wMzMwMJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzHAyMDMznAzMzAwnAzMzw8nAzMxwMjAzM0pMBpJ+KmmJpIcK03aQNE3S3Pw8tKztm5lZ9cqsGfwMGNtq2kRgekSMBqbncTMzq7PSkkFE/Bl4vtXk44DJeXgycHxZ2zczs+rVus9gREQsAsjPw9tbUNIESTMlzWxubq5ZAc3MtkQ9tgM5IiZFRFNENDU0NNS7OGZmm7VaJ4PFkkYC5OclNd6+mZm1odbJ4GZgfB4eD0yt8fbNzKwNZd5a+gvgbmAfSQslnQlcBBwtaS5wdB43M7M661dW4Ig4pZ1ZR5W1TTMz65oe24FsZma142RgZmZOBmZm5mRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmYmRlOBmZmhpOBmZnhZGBmZjgZmJkZTgZmZoaTgZmZ4WRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmYmRl1SgaSxkp6VNLjkibWowxmZtai5slAUl/gUuAYYD/gFEn71bocZmbWoh41gwOBxyNifkSsAX4JHFeHcpiZWaaIqO0GpROBsRFxVh7/CHBQRJzTarkJwIQ8ug/waInFGgYsdVzHdVzHrXHsMssMsHtENFSzYL8SC9EetTHtNRkpIiYBk8ovDkiaGRFNjuu4juu4tYxdZpk3Vj2aiRYCuxbGdwGeqUM5zMwsq0cyuBcYLel1kvoDJwM316EcZmaW1byZKCLWSjoHuA3oC/w0Ih6udTlaKas5ynEd13F7f9wyY9ekKbwaNe9ANjOznsffQDYzMycDMzMDIqLHPYCfAkuAhwrTdgCmAXPz89DCvK2AWXl4LOk7CY8DEwvLXAE8D6wFXgS2LcT9E7ASuBv4G/Aw8CDwoULc8cDLwBrgAaB/nv5Ajhn5+deFuAuA9Tn2nPx4GPhMIe6Jef6avG9D8/SZwCs57rpWcR8txJ3Xhbgrc9z1wEuFY/R/8rbWAKvydh4Gzqsc47z9+wrLfDWv+3rgpUJ5XymU+UJgRd7ePOB/etCxcNz03q7EfRno10PiXp1jrqbwmQcOytMiPw/t4XHPI31WAri41bnuvHxsKvEeyc8XFZYZAFxHOqfdAzTm6TsCd5A+W5e0insS6Rz2MPDtqs679T7xt5MMDgfewobJ4NvkkzswEfhWYd4RwH+TOqTnAXsA/Ukn6v3yMoMLcZcWYn0buB44H/gOMClP3xl4DvhxjrsC+FSOu5SWk2AjcDpwS37BT8vTbyDdRntJ3sbSXIbtgKfyG6cv8EKe3x9YBFye1x9O+tLdFcA/C3F/SkpqlwHfJN2WO3oj474VuAp4DFiYp78BWAb8lXS775Okzq3t8nKn52P8n/m47EFKHktIPytSjPt9UsI9LcdtzuW+BPhf4Ma8zZ5wLLbouKTWgaeBL+a4K4Az6x03L3ct8F/AQxQ+86T30a9y3OWF6T017qHACcATFJIB6bPxKPDbHGtWjtufdMF0TF7ubOBHefhk4Lo8vA1wGPAJCsmAlCSeBBry+GTgqF6ZDAon2WIyeBQYmYdHAo8W5n0LOBY4BLitMP1C4MI24j4HXFCIO5V0QmsddwlwVo67hpYrm8uAx1rFvYl0tTa4sO5fc7lGkk60H8zz5pJOpIeQrjQq+/VNoLlV3EtJVySVuE+Tsn0l7lLg812I+03SFcnsPD6OdII5tXBMl+ThqaQrk2PzG/qOwuvQXDzGwFdICWMdKfmNy/t+SV7/1bg96FhssXGBBtLPwwBcTPps/K7ecQufzbeSTq6vfjbz9JE57pLC9B4Zt/C6LWDDZDAOuAs4O49/qRD3B8DH8vBtwCF5uF/evgpxTmfDZPBW4A+F8Y8AP+zsnNub+gxGRMQigPw8vDDvCGAGMIqUqSsW5mkASLqS9D2HAaSrXIARpK9szy7GlXQgKVv/gtQE8nJErM3rPELKvkWjgNUR8VIe3xrYn1SteyXH2lVSI7Ab6eppFCnBLMrrzMnLFe0FrCnE3Y50Vf4g6ep7SBfj7ktLMxakN/AA4E+StibVogbnuG8G9iQd42Gk2lflddiawjHO9gFW5DI/lLf9tnwsDiclCXrQsdiS4y4FtpJU+RbsQFq+FFrPuJA+m83wms/8iML21hWm99S47XkIOAD4a/7MHZvjDgHeD0zPy716XsvnoBd57fmn6HHg9ZIaJfUDjmfDL/q2qR4/R9GtJO0MPB8RqyR1+FMXEXGGpD1ICeEk4EpSNfeeVjFHkqp890fESkltJc1oNb4r6QqpYh2wmPSCNpP6E/qQvmA3OyIWt1Pe1vYmXcEXt/t4jrMib0ddiPsGUjPa6/P4i6TEUIn7ACkJ3Ah8FRiXj3FbsVofi/1zPCJijqQ1pGRyUyWupG27UOayjsUWGzciQtLJpKa90bT0fdU1bhXHoM3N9rK480hJ83JaPnPrSBeg/xUR8/NyVf2Ez6szIl6Q9O+k2vx64C+kZNah3lQzWJxP0pWT9ZI8/RhSNQqq+6mL9aQT1Ql5fDU5GeS4S0lteNOAKXmZ2cCgnGUhnUCfrwSUtCOpk+mfhe28TGpLHENqO19Nau+bTWpXr5R3bWW/SFfry1vF3SmvW4x7fUS8hZTQVgBHdiHuKPIVfnYMqSr/3og4nFSbEXBNnl85xktJJ/bK8VrFhsd4EKmKvLww7UXguznu6jx+Yw86Flt03Ii4OyLeTvoF4UoncL3jQrqYasj7XfzMLy5sr29hek+N2563A1Mj4i35s/E86Ved50bExYXlXj2v5XPQ9hTOP22JiN9ExEERcQip+WpuR8tD70oGN5PuTiA/T83DY4Hf5+E2f+pCyV6FWINJTT2QPlSVBPJRUufNVaTmkGLcNcC/57jjSFm3Yhyp46yYrV8kVRsBLshx7yS9yYpxXwE+neOewYY/zTGO1OHaOu5OefhTwLaku6E2Nu5jtFypQTqO04DxknYDPka66vkeGx7j64E3SXpdPl7RKvb+pKuoYplXANvnuKeQ3shzetCx2KLjSqo0Z/QhdUr+qAfEJcc5MQ8XP/PFc8GgwvSeGrc9Y0lX7eTPxgTSxdanWy1XLNeJwB8jdwa0p3LsJQ0ldUBf3tHyQM/sQCZVkxaR3mQLgTNJbWTTSRluOulKvC+pKae47rGkE908Wjp2+pA6apblmOtJnULnAX8uxH04z7+fdDVwPzAmx/honvYK8A9gQKGzZh3pwxl5/lmk9sCVpGSzIs97sBD32Lz+SaSr6zW5zDvk6TvluOtpqWKfk7e9LMdd1cW4q3O8SuynSXcf3E3LLYKRj0flWFTi7pinVW4t/Xoh7sIcsxj3rLz/lWPxWA87Fo7b0iEdPSzujbTc6vlP4FN5+r5s+Jlb18PjnkHL7efr87zBpER6F6mGMTvPn5Nj3g+cldcfSLoIe5x06/sehfPdAtLF1QrS569y9+QvCnFPruq8W+8T/yYmjcPIt1x1cf0PU/guQjfGbXP93ha3N5bZcR23l8TdBfh9V9cv4+HfJjIzs17VZ2BmZiVxMjAzMycDMzNzMjAzM5wMzMwMJwMzMwP+P8o8fiSoO8tOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "when = plotdata.iloc[:,0]\n",
    "performance = plotdata.iloc[:,3]\n",
    "\n",
    "y_pos = np.arange(len(when))\n",
    "plt.bar(y_pos, performance, align='center', alpha=0.5, color=colors)\n",
    "\n",
    "\n",
    "plt.xticks(y_pos, when)\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title('Accuracy After Each Improvement Cycle')\n",
    "\n",
    "\n",
    "\n",
    "plt.savefig('AccuracyExamplesOct02.png')\n",
    "plt.savefig('AccuracyExamplesOct02.pdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import word_tokenize          \n",
    "from nltk.stem import WordNetLemmatizer\n",
    "porter = PorterStemmer()\n",
    "lancaster=LancasterStemmer()\n",
    "#proide a word to be stemmed\n",
    "print(\"Porter Stemmer\")\n",
    "print(porter.stem(\"cats\"))\n",
    "\n",
    "class LemmaTokenizer(object):\n",
    "    def __init__(self):\n",
    "            self.wnl = WordNetLemmatizer()\n",
    "    def __call__(self, doc):\n",
    "            return [self.wnl.lemmatize(t) for t in word_tokenize(doc)]\n",
    "\n",
    "#vect = CountVectorizer(tokenizer=LemmaTokenizer())  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Go Live"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What bad smells are there in live chat logs? This is a whole other talk but roughly\n",
    "\n",
    "- Low confidence answers\n",
    "- Second confidence high\n",
    "- Any agent escalate conversation\n",
    "- Any chatflow with loops\n",
    "- Any chatflow started but not finished\n",
    "- Any use of angry words or sentiment\n",
    "- But also do a random selection and look at them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}