cnn_for_nlp_in_keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cnn-for-nlp-in-keras.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cri5Castro/CNN_for_News_Classification/blob/master/cnn_for_nlp_in_keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "id": "c_YWoxDjmkiP",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "# Convolutional Neural Networks for text classification\n",
        "---\n",
        "<br>\n",
        "<center><h3> Abstract </h3></center>\n",
        " Text classification is one of the most common NLP tasks, there are plenty approaches that can be taken  but sometimes is difficult to think that famous approaches in other ML areas could be useful to perform text analysis, this time we'll try to apply CNN originally designed and widely applied to image analysis to perform text classification.\n",
        "\n",
        "***Motivation:***\n",
        "- CNNs are faster to train than LSTM models.\n",
        "- CNNs are translation invariant, that means they could recognize patterns in the text no matter where they are.\n",
        "- CNNs are also efficient in terms of representation with a large vocabulary.\n",
        "- Convolutional Filters learn good representations automatically, without needing to represent the whole vocabulary.\n",
        "\n",
        "***When to use it?:***\n",
        "- When there is no a strong dependance between a sequence and it long past words.\n",
        "\n",
        "***Note***: In this notebook we put and a special focus on computational performance,  we tried to avoid extra computational complexity repeating tasks, so feel free to contact us if there is any doubt.\n",
        "\n",
        "****Important****: this is a port from the original kernel hosted in kaggle take care of the dependecies and the data set.\n",
        "- [Dataset](https://www.kaggle.com/mgocen/20newsgroups)\n",
        "- [Pretrained Embeddings](https://www.kaggle.com/facebook/fasttext-english-word-vectors-including-subwords)\n",
        "- [Original Kernel](https://www.kaggle.com/criscastromaya/cnn-for-nlp-in-keras)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "67f060164c93f453826c5f70782367eb93525492",
        "id": "JmooLILLmkiQ",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "# Index\n",
        "---\n",
        "___1. Introduction___\n",
        "> - <a href='#1.1'>1.1 Data set Description</a>\n",
        "\n",
        "___2. Preprocessing___\n",
        "> - <a href='#2.1'>2.1 Data cleaning</a>\n",
        "> - <a href='#2.2'>2.2 Data Preparation and Analysis</a>\n",
        "\n",
        "___3.Feature extraction___\n",
        "> - <a href='#3.1'>BOW representation</a>\n",
        "> - <a href='#3.2'>FasText representation</a>\n",
        "\n",
        " ___<a href='#4.'>4.Model Desing</a>___\n",
        " \n",
        " ___5.Training and Testing___\n",
        " > - <a href='#5.1'>5.1 CNN+BOW representation</a>\n",
        "> - <a href='#5.2'>5.2 CNN+FastText representation</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "fd0ba1663acbf76af27af06db359fd6e95dedbfe",
        "id": "Xe62k62EmkiR",
        "colab_type": "text"
      },
      "source": [
        "<a id='1.1'></a>\n",
        "## Data set Description\n",
        "We worked in the 20 Newsgroups that can be found at \n",
        "> [20 news groups dataset](http://qwone.com/~jason/20Newsgroups/)\n",
        "\n",
        "Also this data set is available as a kaggle dataset.\n",
        "\n",
        "The data is divided in two folders one for the train set and the other one for the test set, then each subdirectory in the bundle represents a newsgroup.\n",
        "\n",
        "In order to simplify the data manipulation we constructed a pandas dataframe with the following structure\n",
        " \n",
        " **ID | Document | label | **. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": true,
        "id": "Z1qrpEmFmkiS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd #database manipulation\n",
        "import numpy as np #math library"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "9b18f9632b3fa29c2228c42fccfc87dc0fe9d0eb",
        "id": "W6c47YFJmkiW",
        "colab_type": "text"
      },
      "source": [
        "<a id='2.1'></a>\n",
        "## Preprocessing\n",
        "In the original 20 news group dataset we should remove headers,footers and quotes but this preprocessing have been made in the 20 news groups v3 by the data set uploader.\n",
        "At this time we  only worried about some preprocessing on the text such as:\n",
        ">\n",
        "- **Remove weird characters** (if they exist).\n",
        "- **Separate contractions**: Firtsly we thought about expand the contractions but actually as we will remove the most common characters, we just separate the contractions from other words  for example  **doesn't will be does n't**.\n",
        "- **Also we have  to remove the footer lines **.\n",
        "- **Remove long words**: we established a maximum length o 13 letters for each word which is the average maximum length for a word in English after this limit is exceeded probably this represents a spelling error.\n",
        "- **Remove emails and links**: we removed emails and links cause they doesn't apport important information for the problem.\n",
        " >"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "a63da9c005f2641d7cf1e2854f6ccb6e696e04ae",
        "id": "XivAR0nXmkiX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "#this are our cleaning rules\n",
        "cleaningOptions = {\n",
        "    '[A-Za-z0-9_-]{10,}':'',#long words nor\n",
        "    #expand contractions\n",
        "    \"\\'m\":\" am\",\n",
        "    \"\\'s\":\" is\",\n",
        "    \"\\'ve\":\" have\",\n",
        "    \"n\\'t\":\" not\",\n",
        "    \"\\'re\":\" are\",\n",
        "    \"\\'d\":\" had\",\n",
        "    \"\\'ll\":\" will\",\n",
        "    #delete double space, and sequences of \"-,*,^,.\"\n",
        "    '\\s{2,}|\\?{2,}|\\!{2,}|#{2,}|={2,}|-{2,}|_{2,}|\\.{2,}|\\*{2,}|\\^{2,}':'',\n",
        "    #Separate simbols from words\n",
        "    '(':' ( ',\n",
        "    '/':' / ',\n",
        "    ')':' ) ',\n",
        "    '?':' ? ',\n",
        "    '¿':' ¿ ',\n",
        "    ']':' ] ',\n",
        "    '[':' [ ',\n",
        "    '}':' } ',\n",
        "    '{':' { ',\n",
        "    '<':' < ',\n",
        "    '\"':' \" ',\n",
        "    '>':' > ',\n",
        "    ',':' , ',\n",
        "    '!':' ! ',\n",
        "    '.':' . ',\n",
        "    ':':' : ',\n",
        "    '-':' - ',\n",
        "    #delete emails\n",
        "    \"[A-Za-z0-9_-]*@[A-Za-z0-9._-]*\\s?\":\"\",\n",
        "    #delete links\n",
        "    \"https?://[A-Za-z0-9./-]+\":\"\",\n",
        "}\n",
        "\n",
        "def escapePattern(pattern):\n",
        "    \"\"\"Helper function to build our regex\"\"\"\n",
        "    if len(pattern)==1:\n",
        "        pattern=re.escape(pattern)\n",
        "    return pattern\n",
        "\n",
        "def compileCleanerRegex(cleaningOptions=None):\n",
        "    \"\"\"Given a dictionary of rules this contruct the regular expresion to detect the patterns \"\"\"\n",
        "    return re.compile(\"(%s)\" % \"|\".join(map(escapePattern,cleaningOptions.keys())))\n",
        "\n",
        "replacementDictRegex = compileCleanerRegex(cleaningOptions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "0459219f13c722b56afc30d9a0a83597e5c75880",
        "id": "sz-gb_bsmkia",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cleaning_text(text,cleaningOptions=None,replacementDictRegex=None,encode_format=\"utf-8-sig\",decode_format=\"ascii\",option=\"ignore\"):\n",
        "    \"\"\"Cleaning function for text\n",
        "       Given a text this function applies the cleaning rules defined\n",
        "       in a dictionary using a regex to detect the patterns and remove non-ascii characters.\n",
        "   Args:\n",
        "       text (str): The text we want to clean.\n",
        "       cleaning options (dict): The rules to be applied for the cleaning.\n",
        "       replacementDictRegex(regex): The regular expression for detecting\n",
        "                                    the patterns defined in the cleaning options\n",
        "                                    this has been compiled using the compileCleanerRegex(cleaningOptions) function.\n",
        "        encode_format(str):the format from the incomming text by default is utf-8 that fix for most of the cases\n",
        "        decode_format(str):the format of the cleaning results\n",
        "                            (the function use the encode/decode trick to remove unwanted characters)\n",
        "    Returns:\n",
        "        The cleaned text applying the  cleaning options.\n",
        "    \n",
        "    \"\"\"\n",
        "    \"\"\" REMOVING PUNCTUATIONS ALREADY PERFORMED by KERAS TOKENIZER\n",
        "    ##remove extra characters (TODO)\n",
        "    #s = re.sub(r'(.)\\1+', r'\\1\\1', \"asssigned\")\n",
        "    remove_punctuation=str.maketrans('','',string.punctuation)\n",
        "    text=text.translate(remove_punctuation)\n",
        "    also with this we can skip the \" #Separate simbols from words part\"\n",
        "    #\"\"\"\n",
        "    #optionals\n",
        "    #Removing weird characters\n",
        "    text = text.encode(encode_format).decode(decode_format,option)\n",
        "    #dict.get(key, default = None) default is the value to be returned if the key doesn't exist\n",
        "    return replacementDictRegex.sub(lambda mo:cleaningOptions.get(mo.group(1),), text)    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "73eeb4718aa019062a74e2f841b4071e739d53ad",
        "id": "4Jbcz1Krmkic",
        "colab_type": "text"
      },
      "source": [
        "__Let's made a test  for the cleaning function:__"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "d0cde6978f384cdbad8a17819d25d5f2aa992a55",
        "id": "NjKqGbDCmkid",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "oidDescriptionStr=\"\"\"I'm a  nicewhirrrclickwhirrr\"Clam\" test: ({}(. hi jij ... ,,)\\1+) https://www.kaggle.com/criscastromaya/cnn-for-text-classification((it's)((((djcriz5@gemail.com )))()((isn't)(---____--)) Control\"\"\"\n",
        "print(cleaning_text(oidDescriptionStr,cleaningOptions,replacementDictRegex))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "a16a4335cd2380ec0b91e618e10362324029126f",
        "id": "tnwZFLxrmkig",
        "colab_type": "text"
      },
      "source": [
        "Then  we  got the path of all the files "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "9aab3d26d715f23ceb1242a43d71f8578dc80a27",
        "id": "wByr58EMmkih",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import glob,string\n",
        "path = '../input/20newsgroups/20news-bydate-v3/*/*/*.txt'\n",
        "#list files\n",
        "files=glob.glob(path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2c7e84c807aa1514d8261c1035e204d7a6445a53",
        "id": "1EpfZ4camkim",
        "colab_type": "text"
      },
      "source": [
        "Afterwards we constructed a pandas dataframe for the test and the train set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "1e77c318822462b82cd895db9745aeb33f03c05a",
        "id": "AixIJPbJmkio",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import codecs\n",
        "from tqdm import tqdm\n",
        "def contructDataframe(file_list,cleaningOptions=cleaningOptions,replacementDictRegex=replacementDictRegex):\n",
        "    \"\"\"\n",
        "    This function contructs a pandas for the test and training  dataframe with the format **ID | Document | label | **. \n",
        "    and also will perfom the preprocessing for the data using the cleaning function  \n",
        "    Args:\n",
        "        file_list(list[str]): the path of the  files tobe cleaned and storein the dataframes\n",
        "        cleaning options (dict): The rules to be applied for the cleaning.\n",
        "        replacementDictRegex(regex): The regular expression for detecting\n",
        "                                    the patterns defined in the cleaning options\n",
        "                                    this has been compiled using the compileCleanerRegex(cleaningOptions) function.\n",
        "    returns:\n",
        "        training_df,testing-df(pandas.dataframe): the treaning and testing set as pandas dataframes in the format |ID|Text|Label.\n",
        "    \"\"\"\n",
        "    train=[]\n",
        "    test=[]\n",
        "    mode=\"r\"\n",
        "    encoding=\"utf-8\"\n",
        "    e_option=\"ignore\"\n",
        "    for file in tqdm(file_list):\n",
        "        text = codecs.open(file, mode,encoding, e_option).read() \n",
        "        if(\"20news-bydate-test\" in file):\n",
        "            test.append((cleaning_text(text,cleaningOptions,replacementDictRegex),file.split(\"/\")[-2]))\n",
        "        else:\n",
        "            train.append((cleaning_text(text,cleaningOptions,replacementDictRegex),file.split(\"/\")[-2]))\n",
        "    return pd.DataFrame(train,columns=['text','label']),pd.DataFrame(test,columns=['text','label'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "7900759dcc6b748aac4a83f781d8fe612fc8d5ae",
        "id": "nW1dOBLImkis",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train,df_test=contructDataframe(files)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "98c9818694923525cffcc14aaeb21e0717c0edf0",
        "id": "6E88PMNfmkiv",
        "colab_type": "text"
      },
      "source": [
        "<a id='2.2'></a>\n",
        "### Data Preparation and Analysis\n",
        "*** As a sanity check lets see if there is no missing data or evident errors***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "78ebeea24f70b65ad3d4471d13d0f83e1be1f92e",
        "id": "muI_LtA4mkix",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Train: \",df_train.isnull().values.any(),\" Test: \",df_test.isnull().values.any())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "9f049b50e9b55cd53fd78b2c59b86bb5fc3a0352",
        "id": "3XsrJbWsmki2",
        "colab_type": "text"
      },
      "source": [
        "**** Also we'll see the distribution of the classes****"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "4faaef929f715cf4af8bb9fb842609953fe06244",
        "id": "mcWkV7C8mki3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train.groupby(df_train.label).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Training set)\",color='red')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "86542b2e62195d1bf5ce1277391244e89fb32367",
        "id": "YYH-oxyJmki5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_test.groupby(df_test.label).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Test set)\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "41de9287b6973d6fd39fba4f18a11c78578f2153",
        "id": "OwxKjyoVmki9",
        "colab_type": "text"
      },
      "source": [
        "Luckily the train and the test set are pretty well balanced.But we still have to check about the state of the data.\n",
        "So we perfom a text and now at this case the size of the text counting tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "2844245ef91c4232a72f003838dad849a7b163bf",
        "scrolled": true,
        "id": "QM6NNwVbmki-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#df_train[df_train.text.str.split(\" \").apply(len)==df_train.text.str.split(\" \").apply(len).max()]\n",
        "max_l=df_train.text.str.split(\" \").apply(len).max()\n",
        "min_l=df_train.text.str.split(\" \").apply(len).min()\n",
        "print(f\"As we can see there is something not to good whit the dataset cause the bigger document contains {max_l} tokens  and the smaller document contains  {min_l} tokens\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "d3360df39d54c4b144b4dd922dfd110a503faba0",
        "id": "tUrH4Ns6mkjA",
        "colab_type": "text"
      },
      "source": [
        "The gap between the biggest and the smaller document is huge, In consequence, we should visualize the distribution for the length of the documents and also check the extreme values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "fefc2f5906042a0f1d63d8e3f8c0182316d2a9a7",
        "id": "E4vqMvJAmkjB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "df_train['doc_len'] = df_train.text.apply(lambda words: len(words.split()))\n",
        "df_test['doc_len'] = df_test.text.apply(lambda words: len(words.split()))\n",
        "\n",
        "def plot_doc_lengths(dataframe):\n",
        "    max_seq_len = np.round(dataframe.doc_len.mean() + dataframe.doc_len.std()).astype(int)\n",
        "    sns.distplot(tuple(dataframe.doc_len), hist=True, kde=True, label='Document lengths')\n",
        "    plt.axvline(x=max_seq_len, color='k', linestyle='--', label=f'Sequence length mean:{max_seq_len}')\n",
        "    plt.title('Document lengths')\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "    print(f\" the bigger document contain {df_train['doc_len'].max()} words  and the smaller {df_train['doc_len'].min()} words\")\n",
        "plot_doc_lengths(df_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "375fc4e0596cb07a099b74e709f0d12e34d2f6a3",
        "id": "9_acXE_MmkjE",
        "colab_type": "text"
      },
      "source": [
        "Then we looked at the smaller and the biggest document, just to see what's wrong."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "a8437aab147eba8d3a666b3befc38aaa4c8dcbd7",
        "id": "rkGsdgxjmkjF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train[df_train.doc_len==df_train['doc_len'].max()]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "cec8accba83ad8ca2e1b3e1f5986b41cae7c5572",
        "id": "8kfz4sZpmkjH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train[df_train.doc_len==df_train['doc_len'].min()].tail(2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2f670860db73223ff926b9ab78bd7216d9fe93d3",
        "id": "WTumaiSPmkjK",
        "colab_type": "text"
      },
      "source": [
        "As we can see there are many empty texts. \n",
        "After an examination of the data set, we decided to delete the entries smaller than 10 tokens and bigger than 3250 tokens cause outside of this range, base 64  strings and other unwanted noise starts to appear."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "0dd86ae8f12b29b907999b1d8eea298bd89d9da3",
        "id": "YlIbqcummkjL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train=df_train[(10<df_train.doc_len)&(3250>df_train.doc_len)]\n",
        "##also I'll do the same for the test set\n",
        "df_test=df_test[(10<df_test.doc_len)&(3250>df_test.doc_len)]\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "7eff78533669ce7dd2ed107566beca6d3c2e0832",
        "id": "agzV6UEJmkjN",
        "colab_type": "text"
      },
      "source": [
        "Let's see how this filtration altered the data distribution."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "f5ed58b96a7f3ad4775b373ff798aa5a96b513aa",
        "id": "2lRvWdt_mkjP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#df_train.sort_values(by=['doc_len'])\n",
        "df_train.groupby(df_train.label).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Train set)\",color='red')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "86edffd8fba40e96ed39b2cd39eccf6602e29850",
        "id": "Ocg2czoZmkjU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_test.groupby(df_test.label).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Test set)\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "849f76d69fec7e2566e7e083e80f8044e8cc2124",
        "id": "ocBlz2rxmkjZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_doc_lengths(df_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "4801bbf571bf1901b14ecb396b74aa027e26cfc1",
        "id": "tEVuyC9Zmkjc",
        "colab_type": "text"
      },
      "source": [
        "<a id='3.1'></a>\n",
        "# Feature extraction\n",
        "In this section we will transform our text data in a numerical representation, We will use for this experiment the a Bag of words  representation implemented  in keras tokenizer (sparse) and a Skip-gram based pre-trained model using the Facebook fasttext representation.  I highly recommend the following article written by Dipanjan (DJ) Sarkar to going deeper in this subject: [ A hands-on intuitive approach to Deep Learning Methods for Text Data ](https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "trusted": true,
        "_uuid": "88bf20a234a91d586da8bc6fc6ef8b73bf740af6",
        "id": "9i-FcYQmmkjd",
        "colab_type": "text"
      },
      "source": [
        "Before starting with the feature extraction we  splited our ***Training data*** in two parts:  ***training*** and ***validation***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "19cfdd598e9154118bdc2be76b339528e4b0c018",
        "id": "SJrg8UL1mkje",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "SEED = 200\n",
        "X_train, X_validation, y_train, y_validation = train_test_split(df_train.text, df_train.label, test_size=0.2, random_state=3,stratify= df_train.label)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "16c8d28d7ca6eb5e7f44a682b7c1062f64dc9406",
        "id": "mknJbbp-mkjg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_train.groupby(y_train).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Train set)\",color='red')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "936164ae6409305f3a281c05be9e92886a02d1ea",
        "id": "JsZIP6zimkjj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_validation.groupby(y_validation).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Validation set)\",color='green')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "37d130eec7b5a0084944ae6324f446a966265b5e",
        "id": "cQDj0uJwmkjn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_test.text.groupby(df_test.label).size().reset_index(name=\"counts\").plot.bar(x='label',title=\"Samples per each class (Test set)\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "aad5953469c88f5251f792c0ce82222e3c9321da",
        "id": "BeR5gx1xmkjp",
        "colab_type": "text"
      },
      "source": [
        "Then we transformed our data into a Bag of words based model representation.\n",
        "Basically the Keras tokenizer make a dictionary of the whole dataset and it will present each document as a sequence of words asigning a number to each word according to its frequence in the texts."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "764e771b589c4ff22af37ba8d902e477d633da18",
        "id": "CpJ6_6ydmkjq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.preprocessing.text import Tokenizer\n",
        "tokenizer = Tokenizer(num_words=500000)\n",
        "tokenizer.fit_on_texts(X_train)\n",
        "sequences_train = tokenizer.texts_to_sequences(X_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "45f3bc2b5e3b2b7f57d61e37fc8a29fdab523a1a",
        "id": "yg4TPQUDmkjs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f\"Original document: {X_train.values[0]} \\nNumerical representation: {sequences_train[0]}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "04fd71f70d36be7a8dfbc5ecd57659fbcf9e72ae",
        "id": "iJ-0xazSmkju",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sequences_validation = tokenizer.texts_to_sequences(X_validation)\n",
        "sequences_test = tokenizer.texts_to_sequences(df_test.text.values)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "c6d2d19fe63e3ecd26047389776730b394bef46d",
        "id": "kPePdZ9Omkjx",
        "colab_type": "text"
      },
      "source": [
        "Then we visualize what the tokenizer has learned and also delete the most 10 frequent  and unfrequent words "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "5f9dd1e1d47d10795c619a623960290f96970071",
        "id": "rqNQwY65mkjx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        " \"\"\"\n",
        "    Citiation\n",
        "    ---------\n",
        "    DL4NLP lab by Oier Lopez de Lacalle\n",
        "    https://www.researchgate.net/profile/Oier_Lopez_de_Lacalle2\n",
        "\"\"\"\n",
        "# Recorver the word index that was created with the tokenizer\n",
        "word_index = tokenizer.word_index\n",
        "print('Found {} unique tokens.\\n'.format(len(word_index)))\n",
        "word_count = tokenizer.word_counts\n",
        "print(\"Show the most frequent word index:\")\n",
        "for i, word in enumerate(sorted(word_count, key=word_count.get, reverse=True)):\n",
        "    print('   {} ({}) --> {}'.format(word, word_count[word], word_index[word]))\n",
        "    del tokenizer.index_word[tokenizer.word_index[word]]\n",
        "    del tokenizer.index_docs[tokenizer.word_index[word]]\n",
        "    del tokenizer.word_index[word]\n",
        "    del tokenizer.word_docs[word]\n",
        "    del tokenizer.word_counts[word]\n",
        "    if i == 9: \n",
        "        print('')\n",
        "        break\n",
        "print(\"Show the least frequent word index:\")\n",
        "for i, word in enumerate(sorted(word_count, key=word_count.get, reverse=False)):\n",
        "    print('   {} ({}) --> {}'.format(word, word_count[word], word_index[word]))\n",
        "    del tokenizer.index_word[tokenizer.word_index[word]]\n",
        "    del tokenizer.index_docs[tokenizer.word_index[word]]\n",
        "    del tokenizer.word_index[word]\n",
        "    del tokenizer.word_docs[word]\n",
        "    del tokenizer.word_counts[word]\n",
        "    if i == 9: \n",
        "        print('')\n",
        "        break"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "8923750f7dc216e2c01b6f5a9431a0e892200463",
        "id": "nwBQrS77mkjz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Recorver the word index that was created with the tokenizer\n",
        "word_index = tokenizer.word_index\n",
        "print('Found {} unique tokens.\\n'.format(len(word_index)))\n",
        "word_count = tokenizer.word_counts\n",
        "print(\"Show the most frequent word index:\")\n",
        "for i, word in enumerate(sorted(word_count, key=word_count.get, reverse=True)):\n",
        "    print('   {} ({}) --> {}'.format(word, word_count[word], word_index[word]))\n",
        "    if i == 9: \n",
        "        print('')\n",
        "        break\n",
        "print(\"Show the least frequent word index:\")\n",
        "for i, word in enumerate(sorted(word_count, key=word_count.get, reverse=False)):\n",
        "    print('   {} ({}) --> {}'.format(word, word_count[word], word_index[word]))\n",
        "    if i == 9: \n",
        "        print('')\n",
        "        break"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "3e13f167cd22b273296de5c2fc6a0a21312ef481",
        "id": "cQE5CEpGmkj4",
        "colab_type": "text"
      },
      "source": [
        "This vectors has different lenght for this reason we need to pad the sequences in order to fit them in our model.\n",
        "We know from our previous data analysis that the most useful documents has a mean of 500 words so we will delimit the length of documents to 600."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "31410ee88a051904ba24418e98935ca372180fec",
        "id": "ig2PO9N8mkj5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "max_length=600"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "804f51bf90cc67d184f10bc2fee38bdbbfc07c53",
        "id": "w5d2LAn3mkj-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.preprocessing import sequence\n",
        "x_train=sequence.pad_sequences(sequences_train,maxlen=max_length)\n",
        "x_validation=sequence.pad_sequences(sequences_validation,maxlen=max_length)\n",
        "x_test=sequence.pad_sequences(sequences_test,maxlen=max_length)\n",
        "print(f\"Train set shape: {x_train.shape}\\nValidation set shape: {x_validation.shape}\\nTest set shape: {x_test.shape}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2f6ac18bd84755ca3f89a2dd8ce8cbad9d711939",
        "id": "pITJ0c5WmkkC",
        "colab_type": "text"
      },
      "source": [
        "Also we need to transform our labels into something recognizable for the model so we  used the scikit-learn label Binarizer to perfom this task using the one hot encode representation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "119aebff04e689a4b70f218197ae6e27ee1415ae",
        "id": "6eBy-ZP_mkkD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.preprocessing import LabelBinarizer\n",
        "encoder = LabelBinarizer()\n",
        "y_train_categorical=encoder.fit_transform(y_train.values.reshape(-1, 1))\n",
        "y_validation_categorical=encoder.transform(y_validation.values.reshape(-1, 1))\n",
        "y_test_categorical=encoder.transform(df_test.label.values.reshape(-1, 1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "558c09d1cfa2018e7c5973f9db70e7b6440a7d3b",
        "id": "BjIsrgMGmkkH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f\"Train set labels: {y_train_categorical.__len__()}\\nValidation set labels: {y_validation_categorical.__len__()}\\nTest set labels: {y_test_categorical.__len__()}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "trusted": true,
        "_uuid": "5f6846896398dcfa82cb01792d104a9e7f8ec9b5",
        "id": "jVWXX21dmkkK",
        "colab_type": "text"
      },
      "source": [
        "<a id='4.'></a>\n",
        "## Model Desing\n",
        "\n",
        "This architecture is composed by the following layers:\n",
        ">\n",
        "- **Embedding Layer**: This layer learn provide a dense representation of words and their relative meanings, this is used to find relationships between words and their context, we decided to establish the dimension as 100 and used and input lenght of 500 which is also the mean of document lenghts in our data set, also this has a vocabulary of the same size of the training set vocabulary,in the case of the fastText integration we will use a dimension of 300 according to our pre-trained embedding , the weights of this layer will be defined by a embedding matrix.\n",
        "- **Convolutional Layer**: This layer tries to find patterns in the sentences applying filters and then will generate feature maps, this first layer is composed by 64 filters with a size of 7 and uses the relu function as activation function, this layer pads the input in such a way that the output feature maps has the same dimension.\n",
        "- **Max pooling layer**: This layer will select the  most important features from the conv layer  generated feature maps this uses a pool size of 2 and stride equal to 1.\n",
        "- **Convolutional Layer**: This layer will  find patterns in the feature maps  and then will generate new feature maps also this one has 64 filters with a size of 7 and uses the relu function as activation function, this layer pads the input in such a way that the output feature maps has the same dimension.\n",
        "- **Global Max pooling layer**: This layer will select the most important features from  generated feature maps of the conv layer.\n",
        "- **Dropout**: This layer is used to improve the generalization of the model in this case this drops 50%  of the neurons from the previous layer to force the weights to be equitative distributed.\n",
        "- **Dense layer**: In order to learn some additional information and actually didn't apply dropout to the output layer we added a dense layer of 64 neurons also I used the l2  regularization method  also known as weight decay it forces the weights to decay towards zero (but not exactly zero). I highly recommend this [article about generalization in DL models](https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques/).\n",
        "- **Output Layer(dense)**: The final layer has 20 neurons that corresponds to each class, in this case we used the softmax function to map a probability for each class.\n",
        "\n",
        "Also  we used binary cross-entropy loss function widely used for multi-classification problems and a custom adam optimizer to learn the parameters and decrease the loss function.\n",
        ">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "eac61eb459c1863ef6d25d3e5eac7ef4bfa8d1d9",
        "id": "s6ryqpPzmkkL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import *\n",
        "from keras import Sequential,optimizers\n",
        "from keras_sequential_ascii import keras2ascii\n",
        "\n",
        "class CNNtext(Sequential):\n",
        "    \"\"\"\n",
        "    This class extends  keras.sequencial in order to build our \n",
        "    model according to the designed architecture\n",
        "    \"\"\"\n",
        "    #params for the convolutional layers\n",
        "    __num_filters = 64\n",
        "    __weight_decay = 1e-4\n",
        "    #optimizers\n",
        "    __adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n",
        "    def __init__(self,max_length,number_of_classes,embedding_matrix=None,vocab_size=None,tokenizer=None):\n",
        "        #creating the model heritance from Keras.sequencial\n",
        "        super().__init__()\n",
        "        #params for the embedding layer\n",
        "        self.__embedding_dim=100 if embedding_matrix is None else embedding_matrix.shape[1]\n",
        "        #self.__vocab_size=vocab_size if tokenizer is None else tokenizer.word_index.__len__()+1\n",
        "        self.__vocab_size=vocab_size if tokenizer is None else max(tokenizer.index_word.keys())+1\n",
        "        try:\n",
        "            self.__max_length=max_length\n",
        "            self.__number_of_classes=number_of_classes \n",
        "        except NameError as error:\n",
        "            print(\"Error \",error,\" must be defined.\")\n",
        "            \n",
        "        #defining layers\n",
        "        #This layer will learn an embedding the vocab_size is the vocabulary learn from our tokenizer\n",
        "        #the embedding dimension is defined by our selfs in this case we choose a dimension of 100\n",
        "        #the input length is the maximum length of the documents we will use\n",
        "        if embedding_matrix is None:\n",
        "            self.add(Embedding(self.__vocab_size,\n",
        "                               self.__embedding_dim,\n",
        "                               input_length=self.__max_length,trainable=True))\n",
        "        else:\n",
        "            self.add(Embedding(embedding_matrix.shape[0],\n",
        "                               embedding_matrix.shape[1],\n",
        "                               weights=[embedding_matrix],\n",
        "                               input_length=self.__max_length,\n",
        "                               trainable=False))\n",
        "        #then we apply a 1D conv layer that should apply filters to the sequence and generate features maps.\n",
        "        self.add(Conv1D(self.__num_filters, 7, activation='relu', padding='same'))\n",
        "        #then we will get the most important features using a max pooling layer\n",
        "        self.add(MaxPooling1D(2))\n",
        "        #afterwards we apply a conv 1D layer to learn new features form the previous results\n",
        "        self.add(Conv1D(self.__num_filters, 7, activation='relu', padding='same'))\n",
        "        #we select again the most important features\n",
        "        self.add(GlobalMaxPooling1D())\n",
        "        #then we apply dropout to improve the generalization\n",
        "        self.add(Dropout(0.5))\n",
        "        #then we will pass the results into a dense layer that will also learn some internal representation and we also use the l2 regularization\n",
        "        self.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(self.__weight_decay)))\n",
        "        #for the final layer we will use softmax to obtain the probabilities of each class.\n",
        "        self.add(Dense(self.__number_of_classes, activation='softmax'))  \n",
        "        #to compute the loss function we use binary_crossentropy\n",
        "        #which is widely used for multi-classification problems\n",
        "        #we also use the adam optimazer to learn the parameters(weights)\n",
        "        #and minimize the loss function.\n",
        "        self.compile(loss='binary_crossentropy', optimizer=self.__adam, metrics=['accuracy'])     "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "2d2eaaad74840a17d5d7db753c5542c13a51be0c",
        "id": "_ognQLRGmkkN",
        "colab_type": "text"
      },
      "source": [
        "<a id='5.1'></a>\n",
        "## Training and testing (CNN+BOW)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "b7852ed11ab15700f16f487b530efd6afc207dd1",
        "id": "JoIJa4IImkkO",
        "colab_type": "text"
      },
      "source": [
        "We will use the early stopping technique which monitors the status of  the validation loss  to stop the training when the loss stops its improving."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "78a7c24e925afda3d1e3225aea8510b8c7acd19f",
        "id": "vSX-1Q7nmkkQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "early_stopping = EarlyStopping(monitor='val_loss', min_delta=0.01, patience=4, verbose=1)\n",
        "callbacks_list = [early_stopping]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "c27c4c60a18b8dc7e7f5647d78581edfa4a8ff1a",
        "id": "r8-jSpPxmkkS",
        "colab_type": "text"
      },
      "source": [
        "We define the batch size and the number of epochs,we trained to fit the whole train set but the actual number of epochs would be decided by the condition established in the callback."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "eb542d20577909fcea005ad6a0c7fcaa147462f0",
        "id": "k-a2KFaqmkkT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#training params\n",
        "batch_size = 150\n",
        "num_epochs = 20"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "12e479cd94e48cb351f1849fbae2889bebea8ae4",
        "id": "AyP7D4Y5mkkW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tokenizer.num_words"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "3d620915a8fdb6480a1c0086cabb2dc1116457a8",
        "id": "jwk0KTAzmkkY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "CNN_BOW=CNNtext(max_length,\n",
        "              encoder.classes_.__len__(),\n",
        "              tokenizer=tokenizer)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "3120d042ea0ee9e87a0dc92cdf7f7ee0eb7a4f0e",
        "id": "BsHy7X8fmkka",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "keras2ascii(CNN_BOW)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "22b58f6b34354847ac4453bbe6e2da93ea532328",
        "id": "Su3LjvNpmkkg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "hist = CNN_BOW.fit(x_train, y_train_categorical,\n",
        "                 batch_size=batch_size, epochs=num_epochs, callbacks=callbacks_list,\n",
        "                 validation_data=(x_validation,y_validation_categorical),\n",
        "                 shuffle=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "8cb9d99e2a4bae6259069cfc3fa5deab8fa27189",
        "id": "20SbizyZmkki",
        "colab_type": "text"
      },
      "source": [
        "We  checked the perfomance using the test set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "88c5b02f4947bd366827f9db4376917caec2382a",
        "id": "xzFKiE-xmkkj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "loss, accuracy = CNN_BOW.evaluate(x_test,encoder.transform(df_test.label.values), verbose=1)\n",
        "print('Accuracy: %f' % (accuracy*100),'loss: %f' % (loss*100))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "d1d438c0b9ffc97727be85e95b74cb49b714ad3a",
        "id": "qtwd0B5fmkkm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def plot_model_perfomance(hist,name):\n",
        "    plt.style.use('fivethirtyeight')\n",
        "    plt.figure(1)\n",
        "    plt.plot(hist.history['loss'], lw=2.0, color='b', label='train')\n",
        "    plt.plot(hist.history['val_loss'], lw=2.0, color='r', label='val')\n",
        "    plt.title(name)\n",
        "    plt.xlabel('Epochs')\n",
        "    plt.ylabel('Cross-Entropy Loss')\n",
        "    plt.legend(loc='upper right')\n",
        "    plt.figure(2)\n",
        "    plt.plot(hist.history['acc'], lw=2.0, color='b', label='train')\n",
        "    plt.plot(hist.history['val_acc'], lw=2.0, color='r', label='val')\n",
        "    plt.title(name)\n",
        "    plt.xlabel('Epochs')\n",
        "    plt.ylabel('Accuracy')\n",
        "    plt.legend(loc='upper left')\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "3fadfe1eebadbcda19f4102f5dcfff092ed0f516",
        "scrolled": true,
        "id": "-PzLlV9Pmkkr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_model_perfomance(hist,'CNN BOW')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "fa7fad55eb1a373d92b791eebb1496a4e39a40f6",
        "id": "QHv1Jzmsmkkw",
        "colab_type": "text"
      },
      "source": [
        "Know we will construct the confusion matrix making the predictions for the test ,validation and train sets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "d890710c3750315e78e9fa14da5177c84553b9c7",
        "id": "vW1vcIRamkkx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bow_predict_y_test = CNN_BOW.predict(x_test,verbose=1)\n",
        "bow_predict_y_train = CNN_BOW.predict(x_train,verbose=1)\n",
        "bow_predict_y_validation = CNN_BOW.predict(x_validation,verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "873e9ed1e64030f6b5ea44cf1b3c44eb595088f3",
        "id": "sE4O-rSlmkkz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bow_predict_y_test= encoder.inverse_transform(bow_predict_y_test)\n",
        "bow_predict_y_train= encoder.inverse_transform(bow_predict_y_train)\n",
        "bow_predict_y_validation= encoder.inverse_transform(bow_predict_y_validation)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "54c44d921dd260429b111da7e24874bf58a3034c",
        "id": "LVNz9BMlmkk1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "def plot_confusion_matrix(y=None,y_predict=None,classes=None,name=None):\n",
        "    plt.figure(figsize=(30, 30))\n",
        "    sns.heatmap(confusion_matrix(y,y_predict), \n",
        "                xticklabels=classes,\n",
        "                yticklabels=classes)\n",
        "    plt.title(name)\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "7e9e5e827276931fe8b0754a6aab22606c77d56d",
        "id": "P1lQHjzKmkk3",
        "colab_type": "text"
      },
      "source": [
        "This is the confusion matrix for the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "6ef47e75b1c21bf9045b42c3c4e4f4474c967323",
        "id": "XoSty-xAmkk4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_confusion_matrix(df_test.label.values,bow_predict_y_test,encoder.classes_,'Test accuracy CNN BOW')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "30858fc264477da990f55ceebec16a635a9655d8",
        "id": "w5olk7Nimkk7",
        "colab_type": "text"
      },
      "source": [
        "This is the confusion matrix for the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "73a97e14270464e3652a5925338ced04e20f6796",
        "id": "Zll8jiWjmkk8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_confusion_matrix(y_validation,bow_predict_y_validation,encoder.classes_,'Validation accuracy CNN BOW')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "94e7152d36b1c33dff44f94866df5877b25df981",
        "id": "bKVo0CN-mkk-",
        "colab_type": "text"
      },
      "source": [
        "This is the confusion matrix for the train set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "5cbdaada4a270af6519fd07cee80db28765fdd47",
        "id": "gH4Yd3zPmkk_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_confusion_matrix(y_train,bow_predict_y_train,encoder.classes_,'Train accuracy CNN BOW')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "trusted": true,
        "_uuid": "bb273cbe710702a8bae051492e02ba8746f20af1",
        "id": "_o0phtfGmklB",
        "colab_type": "text"
      },
      "source": [
        "<a id='3.2'></a>\n",
        "## FastText integration \n",
        "In this section we will use the fastext embeddings and see how our results could be affected \n",
        "we are using a pretrained [Fasttext embedding](https://www.kaggle.com/facebook/fasttext-english-word-vectors-including-subwords#wiki-news-300d-1M-subword.vec) from kaggle."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "c239be933058223d15384e882fda77322659259a",
        "id": "jI2MTrfZmklC",
        "colab_type": "text"
      },
      "source": [
        "First we build the embedding matrix for our vocabulary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "66603928ef5f4e875e9a505cc0593604d64bd0fa",
        "id": "yIKBmzefmklD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def read(file=None,embed_dim=300,threshold=None, vocabulary=None):\n",
        "    embedding_matrix= np.zeros((max(vocabulary.index_word.keys())+1, embed_dim)) if threshold is None else np.zeros((threshold, embed_dim))\n",
        "    #embedding_matrix= np.zeros((vocabulary.word_index.__len__()+1, embed_dim)) if threshold is None else np.zeros((threshold, embed_dim))\n",
        "    words_not_found=[]\n",
        "    matching=[]\n",
        "    f = codecs.open(file, encoding='utf-8')\n",
        "    for line in tqdm(f):\n",
        "        vec = line.rstrip().rsplit(' ')\n",
        "        word=vec[0].lower()\n",
        "        if word in vocabulary.word_index:\n",
        "            matching.append(word)\n",
        "            embedding_matrix[vocabulary.word_index[word]]= np.asarray(vec[1:], dtype='float32')\n",
        "        else:\n",
        "            words_not_found.append(word)      \n",
        "    f.close()\n",
        "    return embedding_matrix,words_not_found,matching"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "ebdb8e98c47d7c2907a2d67a3a21d3640205509e",
        "id": "n2ee2KiQmklK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "embedding_matrix,words_not_found,match= read(\"../input/fasttext-english-word-vectors-including-subwords/wiki-news-300d-1M-subword.vec\",vocabulary=tokenizer)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "e51682686aaca018f0e2839881de2397b018cfb6",
        "id": "h8ex2rD5mklO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f\"{len(words_not_found)} words not found\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "71fee43b4a0ca6f79894bd38be00a0bc848d670d",
        "id": "s_Unsm2CmklR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "embedding_matrix.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_uuid": "dc8d3dd1c5040df015dc817784986b037c82c671",
        "id": "gx_1b-oUmklW",
        "colab_type": "text"
      },
      "source": [
        "<a id='5.2'></a>\n",
        "Know is time to build the model, this time we will set weights of the embedddings layer using the embedding matrix from the fastetext vectors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "3acb8fe16b5b336dfba3f2d30b2067929ad5af69",
        "id": "iQEd2sT9mklX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "CNN_fastText=CNNtext(max_length,\n",
        "                     encoder.classes_.__len__(),\n",
        "                     embedding_matrix=embedding_matrix,\n",
        "                     tokenizer=tokenizer)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "898a90390758c46b1b187a1bf602c143b788522e",
        "id": "HYw7tD5omkla",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "keras2ascii(CNN_fastText)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "deca072d1fd3fdf35a1c578c30719566589e85a3",
        "id": "O9WchdO0mkle",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "hist = CNN_fastText.fit(x_train, y_train_categorical,\n",
        "                 batch_size=batch_size, epochs=num_epochs, callbacks=callbacks_list,\n",
        "                 validation_data=(x_validation,y_validation_categorical),\n",
        "                 shuffle=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "6cff73bfffab0bd181b803c2d4e0a181cd66da4d",
        "id": "_bsZDy1Zmklg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_model_perfomance(hist,'CNN FastText')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "b89749e9c6d80a742409f7781ba53e1f3f0831ff",
        "id": "g5s4giU3mkli",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ft_predict_y_test = CNN_fastText.predict(x_test,verbose=1)\n",
        "ft_predict_y_train = CNN_fastText.predict(x_train,verbose=1)\n",
        "ft_predict_y_validation = CNN_fastText.predict(x_validation,verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "8eb791dea5b11655759e001d512e7bed741e6d33",
        "id": "TWjM2442mklk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ft_predict_y_test= encoder.inverse_transform(ft_predict_y_test)\n",
        "ft_predict_y_train= encoder.inverse_transform(ft_predict_y_train)\n",
        "ft_predict_y_validation= encoder.inverse_transform(ft_predict_y_validation)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "b4880fbfa612f88245b860cc19e20048b722a612",
        "id": "I6Hbh7EQmklo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "loss, accuracy = CNN_fastText.evaluate(x_test,encoder.transform(df_test.label.values), verbose=1)\n",
        "print('Accuracy: %f' % (accuracy*100),'loss: %f' % (loss*100))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "4e1616f554df0eca1183e13de40e6d33b3a1b2cb",
        "id": "vknSBk_smklv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_confusion_matrix(df_test.label.values,ft_predict_y_test,encoder.classes_,'Test accuracy CNN FastText')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "7ca4970315fc96f9111ff5dc3c27420c2a5700d5",
        "id": "mZ0oVMdumklx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_confusion_matrix(y_validation,ft_predict_y_validation,encoder.classes_,'Validation accuracy CNN FastText')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "_uuid": "5870e35cd9511c52682318a04e4e11d3542c3474",
        "id": "JDwKXbH7mklz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_confusion_matrix(y_train,ft_predict_y_train,encoder.classes_,'Train accuracy CNN FastText')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "trusted": true,
        "_uuid": "fc0b9560775895a68d40b5b4155d715381825f7d",
        "id": "e4hm8zqamkl4",
        "colab_type": "text"
      },
      "source": [
        "### Conclusions\n",
        "\n",
        "CNNs are very useful for recognize  text patterns and its properties allows us to design very strong models for NLP tasks, making a comparison between the FastText embeddings and the embeddings learn  from the embedding layer in the first approach we see that the results for the confusion matrix are better for the first approach but seeing the behaviour of the loss and accuracy we can notice that results are better using FastText embeddings cause the gap between validation and train are smaller."
      ]
    }
  ]
}