frequency_analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "frequency_analysis.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yjAcfuQ-IMkz"
      },
      "source": [
        "#Project Summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AgHk01AJIOO0"
      },
      "source": [
        "This is the source code for the IU Antisemitism Datathon and Hackathon 2020.\n",
        "The final project classifier is an LSTM Keras network using word embeddings and several hidden layers.\n",
        "We experimented using several different classification methods before concluding that the LSTM network gave superior performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nIT5LjEDJwtA"
      },
      "source": [
        "##Methods Explored:\n",
        "**1) Ngrams Model with Tf-Idf vectorization** <br>\n",
        "Used NLTK to preprocess text of each tweet along with the user's profile description. The tweet text concatenated with the user's profile description was stored under the feature column \"total_text\". After setting text to lower case, stemming, lemmatizing, and removing stop words, there were ~550 unique words. Used NLTK's frequency distribution to find unigrams, bigrams, and trigrams that appeared more frequently(10 times more) in antisemitic tweets than clean tweets(or vice-versa). Those most significant ngrams were used for the next phase of the model. We used sklearn to extract the Term Frequency-Inverse Document Frequency(Tf-Idf) for each ngram. This method gives higher scores to ngrams that appear less frequently in the corpus as a whole but more frequently in an individual tweet, thus giving more weight to terms significant to any given tweet. The vectors of the Tfidf frequency of these ngrams served as inputs for a Naive Bayes Classifier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_hO1_0oLng_"
      },
      "source": [
        "**2) Spacy Text Classifiers** <br>\n",
        "Vectorized the tweet data using the same ngram vocabularly determined in the first NLTK model. The data was vectorized using only a CountVectorizer instead of the tf-idf method of determining ngram frequency. Used sklearn's Pipeline to easily test this method of text preparation on a variety of different models. Tested models including LogisticRegression, Naive Bayes(MultinomialNB), Support Vector Classifier(SVC), RandomForestClassifier, AdaBoostClassifier, and RandomForest Classifier. Finally, these algorithms were combined into a VoterClassifier where each subclass was given a single vote. The AdaBoostClassifier and the VotingClassifer had the highest accuracy and f1score of all of the models tested."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ZkPzqafOpSv"
      },
      "source": [
        "**3) LSTM classifier** <br>\n",
        "Used Keras Tokenizer, which converted all text to lowercase and stripped punctuation but otherwise skipped text preprocessing used in earlier models. Vectorized data is fed into deep network with several hidden layers, including an LSTM layer, a dropout layer, and several dense sigmoid layers. Despite the decreased amount of text preprocessing, this network outperformed all other models and was ultimately chosen to classify the data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kErHutcUPVn6"
      },
      "source": [
        "**4) Classifier Combination** <br>\n",
        "We hypothesized that the LSTM networks already strong predictions could be augmented through the incorporation of prediction by the other classifiers. We engineered feature columns representing the predictions of the TfidfClassifier, VoterClassifier, AdaBoostClassifier, and LSTM network. These predictions were used as input for a Support Vector Classifier(SVC). However, this ensemble classifier performed worse when measured by the accuracy and f1scores than the LSTM network alone. This suggests that the LSTM network already encompasses all of the information gleaned by the previous text classifiers, and thus was unaided by their input.\n",
        "Consequently, we chose to use solely the LSTM network.<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d0kVOanBo66L"
      },
      "source": [
        "**Final Result**<br>\n",
        "The LSTM network outputs a probability between 0 and 1 that the tweet is antisemitic. We found that feeding this input into the SVC, which outputted a binary prediction, obtained greater accuracy than setting an arbitrary threshold such as 0.5 and splitting the LSTM probability into a binary output in that manner. The SVC determines the optimal threshold. Thus, the final classifier is an SVC that outputs a binary probability given single input of the probability determined by the LSTM network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jabFJ0Pvq3Mf"
      },
      "source": [
        "The \"Parent Classifier\" outputs the binary prediction given the probability that the tweet is antisemitic according to the LSTM Classifier. <br>\n",
        "The \"Parent Classifier\" is an SVM that determines the best probability threshold for classification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r0xkH07wSjTc"
      },
      "source": [
        "##To Run The File"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gMvXOw_woBUB"
      },
      "source": [
        "1) Upload training data and testing data.<br>Click the folder icon to the left of the screen and upload the two files, then change the two variable names below to the names of the two data files. <br>\n",
        "2) Run All Cells <br>\n",
        "3) View the output of the last cell in the notebook to see the F1 Score of the classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K0yz4oOMSgca"
      },
      "source": [
        "#Set this variable equal to the name of your training data file(json)\n",
        "JSON_FILE_NAME = \"hackathon2.json\"\n",
        "\n",
        "#Set this variable equal to the name of your test data file(json)\n",
        "JSON_TEST_DATA_FILE_NAME = \"to_test.json\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dq2VGqxlMKpE"
      },
      "source": [
        "#Import Necessary Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9PKLo0lhs3JO"
      },
      "source": [
        "#To get reproducible results, fix random number generators\n",
        "import os\n",
        "import random\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import os\n",
        "def reset_random_seeds():\n",
        "   os.environ['PYTHONHASHSEED']=str(1)\n",
        "   tf.random.set_seed(1)\n",
        "   np.random.seed(1)\n",
        "   random.seed(1)\n",
        "reset_random_seeds()\n",
        "session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "tf.compat.v1.keras.backend.set_session(sess)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sXGgzU2jW4eu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "41e8095f-d8b2-465a-bebe-5e3bbc8a61e7"
      },
      "source": [
        "!pip install flair flask\n",
        "\n",
        "!pip install spacy\n",
        "\n",
        "!python -m spacy download en\n",
        "from spacy.lang.en import English\n",
        "import spacy\n",
        "\n",
        "import nltk\n",
        "from nltk.stem import WordNetLemmatizer \n",
        "nltk.download('punkt')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('stopwords')\n",
        "\n",
        "from nltk.corpus import stopwords\n",
        "stop_words=set(stopwords.words(\"english\"))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: flair in /usr/local/lib/python3.6/dist-packages (0.5)\n",
            "Requirement already satisfied: flask in /usr/local/lib/python3.6/dist-packages (1.1.2)\n",
            "Requirement already satisfied: gensim>=3.4.0 in /usr/local/lib/python3.6/dist-packages (from flair) (3.6.0)\n",
            "Requirement already satisfied: segtok>=1.5.7 in /usr/local/lib/python3.6/dist-packages (from flair) (1.5.10)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from flair) (2019.12.20)\n",
            "Requirement already satisfied: langdetect in /usr/local/lib/python3.6/dist-packages (from flair) (1.0.8)\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.6/dist-packages (from flair) (3.2.1)\n",
            "Requirement already satisfied: sqlitedict>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from flair) (1.6.0)\n",
            "Requirement already satisfied: pytest>=5.3.2 in /usr/local/lib/python3.6/dist-packages (from flair) (5.4.3)\n",
            "Requirement already satisfied: transformers>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from flair) (2.11.0)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from flair) (0.8.7)\n",
            "Requirement already satisfied: hyperopt>=0.1.1 in /usr/local/lib/python3.6/dist-packages (from flair) (0.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from flair) (2.8.1)\n",
            "Requirement already satisfied: mpld3==0.3 in /usr/local/lib/python3.6/dist-packages (from flair) (0.3)\n",
            "Requirement already satisfied: torch>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from flair) (1.5.0+cu101)\n",
            "Requirement already satisfied: deprecated>=1.2.4 in /usr/local/lib/python3.6/dist-packages (from flair) (1.2.10)\n",
            "Requirement already satisfied: tqdm>=4.26.0 in /usr/local/lib/python3.6/dist-packages (from flair) (4.41.1)\n",
            "Requirement already satisfied: bpemb>=0.2.9 in /usr/local/lib/python3.6/dist-packages (from flair) (0.3.0)\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.6/dist-packages (from flair) (0.22.2.post1)\n",
            "Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask) (7.1.2)\n",
            "Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask) (2.11.2)\n",
            "Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask) (1.0.1)\n",
            "Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask) (1.1.0)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (2.0.0)\n",
            "Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.18.5)\n",
            "Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.12.0)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.4.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (1.2.0)\n",
            "Requirement already satisfied: pluggy<1.0,>=0.12 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (0.13.1)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (1.8.1)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (19.3.0)\n",
            "Requirement already satisfied: importlib-metadata>=0.12; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (1.6.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (20.4)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (0.2.3)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=5.3.2->flair) (8.3.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (3.0.12)\n",
            "Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (0.0.43)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (2.23.0)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (0.7)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (0.1.91)\n",
            "Requirement already satisfied: tokenizers==0.7.0 in /usr/local/lib/python3.6/dist-packages (from transformers>=2.10.0->flair) (0.7.0)\n",
            "Requirement already satisfied: pymongo in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (3.10.1)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (0.16.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (2.4)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.6/dist-packages (from deprecated>=1.2.4->flair) (1.12.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.21.3->flair) (0.15.1)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask) (1.1.1)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim>=3.4.0->flair) (1.13.23)\n",
            "Requirement already satisfied: boto in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim>=3.4.0->flair) (2.49.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.12; python_version < \"3.8\"->pytest>=5.3.2->flair) (3.1.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers>=2.10.0->flair) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers>=2.10.0->flair) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers>=2.10.0->flair) (2.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers>=2.10.0->flair) (2020.4.5.1)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->hyperopt>=0.1.1->flair) (4.4.2)\n",
            "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (0.3.3)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (0.10.0)\n",
            "Requirement already satisfied: botocore<1.17.0,>=1.16.23 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (1.16.23)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.23->boto3->smart-open>=1.2.1->gensim>=3.4.0->flair) (0.15.2)\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.6/dist-packages (2.2.4)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (2.23.0)\n",
            "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (0.4.1)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (0.6.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy) (3.0.2)\n",
            "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.1.3)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (4.41.1)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy) (2.0.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from spacy) (47.1.1)\n",
            "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.0.2)\n",
            "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.0.0)\n",
            "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (7.4.0)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.0.2)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.18.5)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2020.4.5.1)\n",
            "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy) (1.6.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy) (3.1.0)\n",
            "Requirement already satisfied: en_core_web_sm==2.2.5 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz#egg=en_core_web_sm==2.2.5 in /usr/local/lib/python3.6/dist-packages (2.2.5)\n",
            "Requirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.6/dist-packages (from en_core_web_sm==2.2.5) (2.2.4)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (2.23.0)\n",
            "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (0.4.1)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (0.6.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (3.0.2)\n",
            "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.1.3)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (4.41.1)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (2.0.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (47.1.1)\n",
            "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.2)\n",
            "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.0)\n",
            "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (7.4.0)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.0.2)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_sm==2.2.5) (1.18.5)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (2.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_sm==2.2.5) (2020.4.5.1)\n",
            "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (1.6.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_sm==2.2.5) (3.1.0)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the model via spacy.load('en_core_web_sm')\n",
            "\u001b[38;5;2m✔ Linking successful\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/en_core_web_sm -->\n",
            "/usr/local/lib/python3.6/dist-packages/spacy/data/en\n",
            "You can now load the model via spacy.load('en')\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VSScz66wSEch",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "978c5ce7-8b98-40dd-b0a2-9c1a479a194f"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\n",
        "from sklearn.base import TransformerMixin\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.tokenize import sent_tokenize, word_tokenize\n",
        "\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.ensemble import AdaBoostClassifier\n",
        "from sklearn.linear_model import LinearRegression\n",
        "\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from nltk.corpus import wordnet\n",
        "\n",
        "from flask import abort, Flask, request\n",
        "from flair.models import TextClassifier\n",
        "from flair.data import Sentence\n",
        "from flask import render_template\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "from sklearn.feature_selection import chi2\n",
        "from sklearn import metrics\n",
        "from sklearn.metrics import accuracy_score\n",
        "import numpy as np\n",
        "import spacy\n",
        "from spacy.lang.en import English\n",
        "spacy_nlp = spacy.load('en')\n",
        "import string\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from keras.models import Model\n",
        "from keras.layers import LSTM, Activation, Dense, Dropout, Input, Embedding, Conv1D, Embedding, MaxPooling1D\n",
        "from keras.optimizers import RMSprop\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing import sequence\n",
        "from keras.utils import to_categorical\n",
        "import keras.backend as K\n",
        "from keras.callbacks import EarlyStopping"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n",
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LcStCIKAMR6n"
      },
      "source": [
        "#Preprocess Data into Vector Formats"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BtNzndHaYNO5"
      },
      "source": [
        "def preprocess(text, remove_uncommon=True):\n",
        "  #move everything to lowercase\n",
        "  text = text.lower()\n",
        "  tokens = nltk.word_tokenize(text)\n",
        "  #remove stop words\n",
        "  go_tokens = []\n",
        "  for w in tokens:\n",
        "      if w not in stop_words:\n",
        "          go_tokens.append(w)\n",
        "  ps = PorterStemmer()\n",
        "  #stem text\n",
        "  stem_tokens=[]\n",
        "  for w in go_tokens:\n",
        "      stem_tokens.append(ps.stem(w))\n",
        "  #lemmatize text\n",
        "  lemmatizer = WordNetLemmatizer()\n",
        "  final_tokens = [lemmatizer.lemmatize(word) for word in stem_tokens]\n",
        "  if(remove_uncommon):\n",
        "    #only keep words that occur more than 2 times\n",
        "    myTokenFD = nltk.FreqDist(final_tokens)\n",
        "    final_tokens = [w for w in list(final_tokens) if myTokenFD[w] > 5]\n",
        "  return final_tokens\n",
        "def preprocess_ngrams(text, n):\n",
        "  final_tokens = preprocess(text, remove_uncommon = False)\n",
        "  final_tokens = nltk.ngrams(final_tokens, n)\n",
        "  return list(final_tokens)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Vzi7fVfdDR8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "b9e8499d-2570-4e7c-805c-6a1d7009beae"
      },
      "source": [
        "print(preprocess_ngrams(\"This! is a bagging! of words bags bag mice bagging mice. bag mice\", 3))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('!', 'bag', '!'), ('bag', '!', 'word'), ('!', 'word', 'bag'), ('word', 'bag', 'bag'), ('bag', 'bag', 'mouse'), ('bag', 'mouse', 'bag'), ('mouse', 'bag', 'mouse'), ('bag', 'mouse', '.'), ('mouse', '.', 'bag'), ('.', 'bag', 'mouse')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V-I1casCMX0F"
      },
      "source": [
        "#Create DataFrame"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "08AV6GDDlFbT"
      },
      "source": [
        "df = pd.read_json(JSON_FILE_NAME)\n",
        "df_test = pd.read_json(JSON_TEST_DATA_FILE_NAME)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zz27FzbwGXey",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "dcc0e30b-11ca-4a8e-d5e5-4c158f2c6e81"
      },
      "source": [
        "type(list(df[\"user\"].iloc[4].keys()))#.values[0][\"description\"])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "list"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "88K314mvj5vf"
      },
      "source": [
        "#Add text column \"total_text\" combining the text of the tweet with the tweet author's profile descritpion\n",
        "#Add binary column \"antisem_binary\" that represents tweets as either antisemitic(1) or clean(0)\n",
        "def create_new_features(df):\n",
        "  for i in range(0,len(df)):\n",
        "    if \"description\" not in list(df[\"user\"].iloc[i].keys()):\n",
        "      df[\"user\"].iloc[i][\"description\"] = \"\"\n",
        "  df[\"total_text\"] = df[\"text\"] + \" \" + df[\"user\"].values[0][\"description\"]\n",
        "  #add the binary classification of antisemitism if the dataframe is training data\n",
        "  if(\"antisemitism_rating\" in df.columns):\n",
        "    df[\"antisem_binary\"] = np.where(df[\"antisemitism_rating\"] > 3, 1, 0)\n",
        "  return df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9vpeTeNbn7Sw"
      },
      "source": [
        "def get_all_text_from_column(df, column):\n",
        "  text = df[column]\n",
        "  text = ' '.join(np.asarray(text))\n",
        "  return text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3fMMozNGlH7y"
      },
      "source": [
        "df = create_new_features(df)\n",
        "df_test = create_new_features(df_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SnDV1Fc-pLMm"
      },
      "source": [
        "total_text = get_all_text_from_column(df, \"total_text\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XmMQtoYgpa2Q"
      },
      "source": [
        "def get_semitic_df(df):\n",
        "  return df[df[\"antisemitism_rating\"] > 3]\n",
        "def get_clean_df(df):\n",
        "  return df[df[\"antisemitism_rating\"] <= 3]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uauruS75qk5V",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        },
        "outputId": "16bc68bd-f81d-45c6-9350-8ddfdbf38df5"
      },
      "source": [
        "sem_df = get_semitic_df(df)\n",
        "clean_df = get_clean_df(df)\n",
        "print(sem_df.shape)\n",
        "print(clean_df.shape)\n",
        "\n",
        "total_text = get_all_text_from_column(df, \"total_text\")\n",
        "sem_text = get_all_text_from_column(sem_df, \"total_text\")\n",
        "clean_text = get_all_text_from_column(clean_df, \"total_text\")\n",
        "\n",
        "total_tokens = preprocess(total_text)\n",
        "sem_tokens = preprocess(sem_text)\n",
        "clean_tokens = preprocess(clean_text)\n",
        "\n",
        "total_bigrams = preprocess_ngrams(total_text, 2)\n",
        "sem_bigrams = preprocess_ngrams(sem_text, 2)\n",
        "clean_bigrams = preprocess_ngrams(clean_text, 2)\n",
        "total_trigrams = preprocess_ngrams(total_text, 3)\n",
        "sem_trigrams = preprocess_ngrams(sem_text, 3)\n",
        "clean_trigrams = preprocess_ngrams(clean_text, 3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(436, 31)\n",
            "(569, 31)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LckDRScwZgPb"
      },
      "source": [
        "#Get Significant Ngrams"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FRO-WH4Aq2fS"
      },
      "source": [
        "def get_freq_distr(tokens):\n",
        "  return nltk.FreqDist(tokens)\n",
        "def get_percent_frequency_distr(tokens):\n",
        "  fd = nltk.FreqDist(tokens)\n",
        "  num_words = float(sum(fd.values()))\n",
        "  relfrq = [x/num_words for x in fd.values() ]\n",
        "  return relfrq\n",
        "total_fd = get_freq_distr(total_tokens)\n",
        "sem_fd = get_freq_distr(sem_tokens)\n",
        "clean_fd = get_freq_distr(clean_tokens)\n",
        "total_pfd = get_percent_frequency_distr(total_tokens)\n",
        "sem_pfd = get_percent_frequency_distr(sem_tokens)\n",
        "clean_pfd = get_percent_frequency_distr(clean_tokens)\n",
        "\n",
        "total_bi_fd = get_freq_distr(total_bigrams)\n",
        "sem_bi_fd = get_freq_distr(sem_bigrams)\n",
        "clean_bi_fd = get_freq_distr(clean_bigrams)\n",
        "\n",
        "total_tri_fd = get_freq_distr(total_trigrams)\n",
        "sem_tri_fd = get_freq_distr(sem_trigrams)\n",
        "clean_tri_fd = get_freq_distr(clean_trigrams)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yWpLJsB3sw5S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "235c26d5-42ec-462b-f806-d518b636863e"
      },
      "source": [
        "print(total_fd)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<FreqDist with 556 samples and 30771 outcomes>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Maiuc8LStf9j",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        },
        "outputId": "981dd8cc-550c-4a5a-98b3-febaecec523f"
      },
      "source": [
        "significant_words = []\n",
        "for x in total_fd:\n",
        "  dif = abs(sem_fd[x] - clean_fd[x])\n",
        "  if(dif > 10):\n",
        "    #print(x, p_dif, sem_fd[x], clean_fd[x])\n",
        "    significant_words.append(x)\n",
        "print(\"There are \", len(significant_words), \"significant words\")\n",
        "\n",
        "significant_bigrams = []\n",
        "for x in total_bi_fd:\n",
        "  dif = abs(sem_bi_fd[x] - clean_bi_fd[x])\n",
        "  if(dif > 10):\n",
        "    #print(x, p_dif, sem_fd[x], clean_fd[x])\n",
        "    significant_bigrams.append(x)\n",
        "print(\"There are \", len(significant_bigrams), \"significant bigrams\")\n",
        "\n",
        "significant_trigrams = []\n",
        "for x in total_tri_fd:\n",
        "  dif = abs(sem_tri_fd[x] - clean_tri_fd[x])\n",
        "  if(dif > 10):\n",
        "    #print(x, p_dif, sem_fd[x], clean_fd[x])\n",
        "    significant_trigrams.append(x)\n",
        "print(\"There are \", len(significant_trigrams), \"significant trigrams\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "There are  140 significant words\n",
            "There are  92 significant bigrams\n",
            "There are  52 significant trigrams\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2uObT78Ay0HY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "1e0a201a-4769-44aa-84bd-7e8bc19b88db"
      },
      "source": [
        "print(significant_words)\n",
        "print(significant_bigrams)\n",
        "print(significant_trigrams)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['medium', 'jew', 'http', ':', 'block', 'ann', 'appelbaum', '&', 'radek', 'sikorski', 'svenska', 'dagbladet😎', 'anti-semit', 'alt-right', '.', 'antysemitów', 'blokuję', 'automatyczni', '#', 'lol', 'dream', 'kike', 'evid', 'parliament', ')', 'fad', 'til', '(', 'sad', '?', 'rt', '@', 'make', 'holocaust', '’', ',', 'un', 'white', '“', 'real', '”', 'de', 'jewish', 'zionazi', \"'s\", 'hate', '``', '!', 'syria', 'rawr', 'xd', 'christian', 'vanguardiacom', 'huevo', 'donará', 'millón', 'en', 'cauca', 'santand', '//t.co/ku32x06m10', '//t.co/mwbqqk40ol', 'hitler', 'result', 'u', 'think', 'lo', 'la', 'el', 'like', 'million', 'woman', 'call', \"''\", 'zionist', \"'\", 'attack', 'muslim', 'remembr', 'day', 'live', '6', 'murder', 'due', 'fake', 'need', 'use', 'maga', 'thing', 'saddest', 'peopl', 'say', 'fuck', 'feel', 'wo', 'nazi', 'rememb', 'auschwitz', 'bet', 'fakenew', 'black', 'bad', 'could', 'apartheid', 'zionazist', 'ethnic', 'clean', 'land', 'militari', 'bomb', 'deport', 'antisemit', 'que', 'le', 'auschwitzmuseum', '|', 'synagogu', 'fear', 'stop', 'provid', 'conclus', 'german', 'best', 'randum', 'sale', 'alternativefact', '🐚', 'stori', 'without', '🐯', 'del', '…', 'one-sid', 'even', 'tri', 'angri', 'necesitada', 'round', 'gent', 'má', 'fault']\n",
            "[('http', ':'), ('block', 'ann'), ('ann', 'appelbaum'), ('appelbaum', '&'), ('&', 'radek'), ('radek', 'sikorski'), ('sikorski', 'svenska'), ('svenska', 'dagbladet😎'), ('dagbladet😎', 'anti-semit'), ('anti-semit', 'block'), ('block', '&'), ('&', 'alt-right'), ('alt-right', '.'), ('.', 'antysemitów'), ('antysemitów', 'blokuję'), ('blokuję', 'automatyczni'), ('automatyczni', '.'), (':', '#'), ('#', 'lol'), ('dream', 'kike'), (':', ')'), ('#', 'til'), ('(', '('), (')', ')'), ('.', 'rt'), ('rt', '@'), ('.', '@'), ('.', 'block'), ('jew', '.'), (':', '``'), ('!', '!'), ('!', '#'), ('#', 'syria'), ('medium', \"'s\"), ('view', 'kike'), ('rawr', 'xd'), ('xd', '#'), ('@', 'vanguardiacom'), ('vanguardiacom', ':'), (':', 'huevo'), ('huevo', 'kike'), ('kike', 'donará'), ('donará', 'un'), ('un', 'millón'), ('millón', 'de'), ('de', 'huevo'), ('huevo', 'en'), ('en', 'cauca'), ('cauca', 'santand'), ('santand', 'http'), (':', '//t.co/ku32x06m10'), ('//t.co/ku32x06m10', 'http'), (':', '//t.co/mwbqqk40ol'), ('//t.co/mwbqqk40ol', 'block'), ('.', 'http'), ('.', 'jew'), ('million', 'jew'), ('``', 'zionazi'), ('zionazi', \"''\"), ('6', 'million'), ('#', 'maga'), ('thing', 'kike'), ('saddest', 'thing'), ('wo', \"n't\"), (':', '('), (',', 'jew'), ('jew', ':'), (\"'\", ','), ('bet', '#'), ('?', '#'), ('#', 'fakenew'), ('kike', '('), ('could', 'due'), ('jew', ','), ('@', 'auschwitzmuseum'), ('auschwitzmuseum', ':'), ('provid', 'conclus'), ('conclus', 'evid'), ('.', '’'), ('#', 'randum'), ('#', 'alternativefact'), (',', '#'), (':', '“'), ('.', 'zionazi'), ('…', 'block'), (\"'s\", 'one-sid'), ('one-sid', 'view'), ('zionazi', 'block'), (',', \"''\"), ('la', 'gent'), (\"''\", '``'), (';', '#')]\n",
            "[('block', 'ann', 'appelbaum'), ('ann', 'appelbaum', '&'), ('appelbaum', '&', 'radek'), ('&', 'radek', 'sikorski'), ('radek', 'sikorski', 'svenska'), ('sikorski', 'svenska', 'dagbladet😎'), ('svenska', 'dagbladet😎', 'anti-semit'), ('dagbladet😎', 'anti-semit', 'block'), ('anti-semit', 'block', '&'), ('block', '&', 'alt-right'), ('&', 'alt-right', '.'), ('alt-right', '.', 'antysemitów'), ('.', 'antysemitów', 'blokuję'), ('antysemitów', 'blokuję', 'automatyczni'), ('blokuję', 'automatyczni', '.'), ('automatyczni', '.', 'rt'), ('.', 'rt', '@'), ('automatyczni', '.', '@'), ('!', '!', '#'), ('rt', '@', 'vanguardiacom'), ('@', 'vanguardiacom', ':'), ('vanguardiacom', ':', 'huevo'), (':', 'huevo', 'kike'), ('huevo', 'kike', 'donará'), ('kike', 'donará', 'un'), ('donará', 'un', 'millón'), ('un', 'millón', 'de'), ('millón', 'de', 'huevo'), ('de', 'huevo', 'en'), ('huevo', 'en', 'cauca'), ('en', 'cauca', 'santand'), ('cauca', 'santand', 'http'), ('santand', 'http', ':'), ('http', ':', '//t.co/ku32x06m10'), (':', '//t.co/ku32x06m10', 'http'), ('//t.co/ku32x06m10', 'http', ':'), ('http', ':', '//t.co/mwbqqk40ol'), (':', '//t.co/mwbqqk40ol', 'block'), ('//t.co/mwbqqk40ol', 'block', 'ann'), ('.', 'http', ':'), ('``', 'zionazi', \"''\"), ('6', 'million', 'jew'), ('saddest', 'thing', 'kike'), ('rt', '@', 'auschwitzmuseum'), ('@', 'auschwitzmuseum', ':'), ('provid', 'conclus', 'evid'), ('…', 'block', 'ann'), ('medium', \"'s\", 'one-sid'), (\"'s\", 'one-sid', 'view'), ('one-sid', 'view', 'kike'), ('automatyczni', '.', 'zionazi'), ('zionazi', 'block', 'ann')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f41_2MOKOQse"
      },
      "source": [
        "#Train and Test Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A8LFcRZX-2tA"
      },
      "source": [
        "y_train = df[\"antisem_binary\"]\n",
        "X_train = df.drop(\"antisem_binary\", axis=1)\n",
        "#y_test = df_test[\"antisem_binary\"]\n",
        "X_test = df_test\n",
        "#X_test = df_test.drop(\"antisem_binary\", axis=1)\n",
        "#X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zczpOD89E42_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        },
        "outputId": "45f657d7-4d7f-4d1d-d854-5b980991affa"
      },
      "source": [
        "print(X_train.shape)\n",
        "print(y_train.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1005, 30)\n",
            "(1005,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0lfd_zIxYDa-"
      },
      "source": [
        "#NGrams Classifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j3eTwWL6Zhoo"
      },
      "source": [
        "def get_vocabulary(significant_words, significant_bigrams, significant_trigrams):\n",
        "  significant_bigrams = [(bigram[0] + \" \" + bigram[1]) for bigram in significant_bigrams]\n",
        "  significant_trigrams = [(trigram[0] + \" \" + trigram[1]) for trigram in significant_trigrams]\n",
        "  vocab = significant_words + significant_bigrams + significant_trigrams\n",
        "  return list(set(vocab))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3_QizpCO9bNf"
      },
      "source": [
        "tfidf_clf = MultinomialNB()\n",
        "tfidf_transformer = TfidfTransformer()\n",
        "vocabulary = get_vocabulary(significant_words, significant_bigrams, significant_trigrams)\n",
        "count_vect = CountVectorizer(ngram_range=(1, 3), vocabulary=vocabulary)\n",
        "#X_train is the array of tweets, y_train is the label of each tweet\n",
        "#only retain vocab_length many ngrams that are most relevant to classification(to decrease complexity)\n",
        "def train_tfidf_clf(X_train, y_train):\n",
        "  X_train_counts = count_vect.fit_transform(np.asarray(X_train[\"total_text\"]))\n",
        "  X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)\n",
        "  tfidf_clf.fit(X_train_tfidf, np.asarray(y_train))\n",
        "#input: unprocessed text\n",
        "#return classification of tweet\n",
        "def predict_tfidf_clf(X_test):\n",
        "  return tfidf_clf.predict(count_vect.transform(np.asarray(X_test[\"total_text\"])))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4LvqCnZrjER-"
      },
      "source": [
        "def predict_tfidf_clf(X_test):\n",
        "  X_test = count_vect.transform(np.asarray(X_test[\"total_text\"]))\n",
        "  return tfidf_clf.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "349YWQfVDxLg"
      },
      "source": [
        "#X is the formatted output of each of the classifiers\n",
        "#y is the label for each tweet-0 for clean, y for antisemitic\n",
        "def score_tfidf_clf(X_test,y_test):\n",
        "  X_test = count_vect.transform(np.asarray(X_test[\"total_text\"]))\n",
        "  y_test = np.asarray(y_test)\n",
        "  score = tfidf_clf.score(X_test,y_test)\n",
        "  return score"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6CMWO9T2BnEo"
      },
      "source": [
        "train_tfidf_clf(X_train,y_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wkHyP3p5jvN6"
      },
      "source": [
        "def add_tfidf_pred(df):\n",
        "  pred = pd.Series(predict_tfidf_clf(df), name=\"tfidf_pred\")\n",
        "  return pd.concat([df.reset_index(drop=True), pred.reset_index(drop=True)], axis=1)\n",
        "df = add_tfidf_pred(df)\n",
        "X_train = add_tfidf_pred(X_train)\n",
        "X_test = add_tfidf_pred(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RlrJVrT8DrB_"
      },
      "source": [
        "#accuracy = score_tfidf_clf(X_test, y_test)\n",
        "#f1_score = metrics.f1_score(predict_tfidf_clf(X_test), y_test)\n",
        "#print(\"Accuracy of this model: \", accuracy)\n",
        "#print(\"F1 Score: \", f1_score)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IzmL6KxhYLGD"
      },
      "source": [
        "#Spacy Text Classifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r5TBiwodjQwL"
      },
      "source": [
        "# Custom transformer using spaCy\n",
        "class predictors(TransformerMixin):\n",
        "    def transform(self, X, **transform_params):\n",
        "        return [text.strip().lower() for text in X[\"total_text\"]]\n",
        "\n",
        "    def fit(self, X, y=None, **fit_params):\n",
        "        return self\n",
        "\n",
        "    def get_params(self, deep=True):\n",
        "        return {}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DTb-xIhnhd8I",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 420
        },
        "outputId": "b49e1a02-9074-4627-859d-c8a14aab35ff"
      },
      "source": [
        "# Logistic Regression Classifier\n",
        "classifier = LogisticRegression()\n",
        "\n",
        "# Create pipeline using Bag of Words\n",
        "pipe = Pipeline([(\"cleaner\", predictors()),\n",
        "                 ('vectorizer', count_vect),\n",
        "                 ('classifier', classifier)])\n",
        "\n",
        "# model generation\n",
        "pipe.fit(X_train,pd.Series(y_train))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(memory=None,\n",
              "         steps=[('cleaner', <__main__.predictors object at 0x7fdce5907710>),\n",
              "                ('vectorizer',\n",
              "                 CountVectorizer(analyzer='word', binary=False,\n",
              "                                 decode_error='strict',\n",
              "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
              "                                 input='content', lowercase=True, max_df=1.0,\n",
              "                                 max_features=None, min_df=1,\n",
              "                                 ngram_range=(1, 3), preprocessor=None,\n",
              "                                 stop_words=None, strip_accents=None,\n",
              "                                 t...\n",
              "                                             'militari', 'cauca', '; #',\n",
              "                                             'zionazi block', 'fakenew', 'nazi',\n",
              "                                             '# randum', 'ann', 'fad', 'http :',\n",
              "                                             'woman', ...])),\n",
              "                ('classifier',\n",
              "                 LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
              "                                    fit_intercept=True, intercept_scaling=1,\n",
              "                                    l1_ratio=None, max_iter=100,\n",
              "                                    multi_class='auto', n_jobs=None,\n",
              "                                    penalty='l2', random_state=None,\n",
              "                                    solver='lbfgs', tol=0.0001, verbose=0,\n",
              "                                    warm_start=False))],\n",
              "         verbose=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A1CK7uKrhMNL"
      },
      "source": [
        "def get_clf(classifier):\n",
        "  return Pipeline([(\"cleaner\", predictors()),\n",
        "                 ('vectorizer', count_vect),\n",
        "                 ('classifier', classifier)])\n",
        "def fit_clf(classifier):\n",
        "  pipe = get_clf(classifier)\n",
        "  pipe.fit(X_train, y_train)\n",
        "def get_clf_f1score(classifier):\n",
        "  pipe = get_clf(classifier)\n",
        "  pipe.fit(X_train,y_train)\n",
        "  #print(\"Recall: \", metrics.recall_score(y_test, pipe.predict(X_test)))\n",
        "  #print(\"Precision: \", metrics.precision_score(y_test, pipe.predict(X_test)))\n",
        "  return metrics.f1_score(y_test, pipe.predict(X_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ol-i724oi7zc"
      },
      "source": [
        "from sklearn import metrics\n",
        "# Predicting with a test dataset\n",
        "predicted = pipe.predict(X_test)\n",
        "\n",
        "# Model Accuracy\n",
        "#print(\"Logistic Regression Accuracy:\",metrics.accuracy_score(y_test, predicted))\n",
        "#print(\"Logistic Regression Precision:\",metrics.precision_score(y_test, predicted))\n",
        "#print(\"Logistic Regression Recall:\",metrics.recall_score(y_test, predicted))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GzaBUSKif-7V"
      },
      "source": [
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.ensemble import VotingClassifier\n",
        "svm = SGDClassifier(loss='hinge', penalty='l2',\n",
        "                             alpha=1e-3, random_state=42,\n",
        "                             max_iter=5, tol=None)\n",
        "svc = SVC(kernel=\"rbf\", C=1.0,gamma=\"scale\")\n",
        "ada = AdaBoostClassifier(base_estimator=LogisticRegression(), n_estimators=50)\n",
        "\n",
        "classifiers = [LogisticRegression(), MultinomialNB(), svc, RandomForestClassifier(n_estimators=10), ada]\n",
        "voter = VotingClassifier(estimators=[(\"lr\", classifiers[0]), (\"mnb\", classifiers[1]), (\"svc\", classifiers[2]), (\"rf\", RandomForestClassifier(n_estimators=10))], voting=\"hard\")\n",
        "#print(\"Classifier F1 Scores\")\n",
        "ada_clf = get_clf(ada)\n",
        "voter_clf = get_clf(voter)\n",
        "fit_clf(voter)\n",
        "#print(\"Voter: \", get_clf_f1score(voter))\n",
        "for classifier in classifiers.copy():\n",
        "  fit_clf(classifier)\n",
        "  #print(classifier.__class__.__name__, \": \", get_clf_f1score(classifier))\n",
        "  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fzGLkEAbFch-"
      },
      "source": [
        "def add_ada_pred(df):\n",
        "  pred = pd.Series(ada_clf.predict(df), name=\"ada_pred\")\n",
        "  return pd.concat([df.reset_index(drop=True), pred.reset_index(drop=True)], axis=1)\n",
        "df = add_ada_pred(df)\n",
        "X_train = add_ada_pred(X_train)\n",
        "X_test = add_ada_pred(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XRN02PssT7GN"
      },
      "source": [
        "def add_voter_pred(df):\n",
        "  pred = pd.Series(voter_clf.predict(df), name=\"voter_pred\")\n",
        "  return pd.concat([df.reset_index(drop=True), pred.reset_index(drop=True)], axis=1)\n",
        "df = add_voter_pred(df)\n",
        "X_train = add_voter_pred(X_train)\n",
        "X_test = add_voter_pred(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iGol1rtAAEUa"
      },
      "source": [
        "# **LSTM Network**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CwJ9hgbvAOvT"
      },
      "source": [
        "X_LSTM_train = X_train[\"text\"]\n",
        "y_LSTM_train = y_train\n",
        "#X_LSTM_train,X_LSTM_test,y_LSTM_train,y_LSTM_test = train_test_split(X_LSTM,y_LSTM,test_size=0.1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yPNVgwC4Ge_1"
      },
      "source": [
        "max_words = 1000\n",
        "max_len = 150\n",
        "tok = Tokenizer(num_words=max_words)\n",
        "tok.fit_on_texts(X_LSTM_train)\n",
        "sequences = tok.texts_to_sequences(X_LSTM_train)\n",
        "sequences_matrix = sequence.pad_sequences(sequences,maxlen=max_len)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2ojAh-zbIaU4"
      },
      "source": [
        "def get_f1(y_true, y_pred): #taken from old keras source code\n",
        "    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
        "    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
        "    precision = true_positives / (predicted_positives + K.epsilon())\n",
        "    recall = true_positives / (possible_positives + K.epsilon())\n",
        "    f1_val = 2*(precision*recall)/(precision+recall+K.epsilon())\n",
        "    return f1_val"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xmarHejcGwmy"
      },
      "source": [
        "def RNN():\n",
        "    inputs = Input(name='inputs',shape=[max_len])\n",
        "    layer = Embedding(max_words,50,input_length=max_len)(inputs)\n",
        "    layer = LSTM(100)(layer)\n",
        "    layer = Dense(256,name='FC1')(layer)\n",
        "    layer = Activation('relu')(layer)\n",
        "    layer = Dropout(0.2)(layer)\n",
        "    layer = Dense(1,name='out_layer')(layer)\n",
        "    layer = Activation('sigmoid')(layer)\n",
        "    model = Model(inputs=inputs,outputs=layer)\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HNlwV1yWGxIg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 776
        },
        "outputId": "4eba5aeb-c93e-4dcc-9b58-dae0f8aa7eb9"
      },
      "source": [
        "reset_random_seeds()\n",
        "model = RNN()\n",
        "model.compile(loss='binary_crossentropy',optimizer=\"adam\",metrics=['accuracy',get_f1])\n",
        "callbacks = []#[EarlyStopping(monitor='val_loss',min_delta=0.0001,patience=3)]\n",
        "model.fit(sequences_matrix,y_LSTM_train,batch_size=128,epochs=20,\n",
        "          validation_split=0.2,callbacks=callbacks)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/indexed_slices.py:434: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
            "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 804 samples, validate on 201 samples\n",
            "Epoch 1/20\n",
            "804/804 [==============================] - 3s 4ms/step - loss: 0.6901 - accuracy: 0.5672 - get_f1: 0.0866 - val_loss: 0.6801 - val_accuracy: 0.5821 - val_get_f1: 0.0000e+00\n",
            "Epoch 2/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.6722 - accuracy: 0.5622 - get_f1: 0.0000e+00 - val_loss: 0.6414 - val_accuracy: 0.5821 - val_get_f1: 0.0000e+00\n",
            "Epoch 3/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.6082 - accuracy: 0.6629 - get_f1: 0.3914 - val_loss: 0.5463 - val_accuracy: 0.7711 - val_get_f1: 0.6562\n",
            "Epoch 4/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.4930 - accuracy: 0.7985 - get_f1: 0.7419 - val_loss: 0.4382 - val_accuracy: 0.8259 - val_get_f1: 0.7905\n",
            "Epoch 5/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.3686 - accuracy: 0.8532 - get_f1: 0.8119 - val_loss: 0.5100 - val_accuracy: 0.7662 - val_get_f1: 0.7714\n",
            "Epoch 6/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.3266 - accuracy: 0.8532 - get_f1: 0.8412 - val_loss: 0.3772 - val_accuracy: 0.8408 - val_get_f1: 0.7975\n",
            "Epoch 7/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.2723 - accuracy: 0.8905 - get_f1: 0.8707 - val_loss: 0.3462 - val_accuracy: 0.8806 - val_get_f1: 0.8586\n",
            "Epoch 8/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.2015 - accuracy: 0.9254 - get_f1: 0.9130 - val_loss: 0.3939 - val_accuracy: 0.8756 - val_get_f1: 0.8659\n",
            "Epoch 9/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.1623 - accuracy: 0.9428 - get_f1: 0.9369 - val_loss: 0.4569 - val_accuracy: 0.8806 - val_get_f1: 0.8696\n",
            "Epoch 10/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.1342 - accuracy: 0.9490 - get_f1: 0.9428 - val_loss: 0.4593 - val_accuracy: 0.8657 - val_get_f1: 0.8522\n",
            "Epoch 11/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.1037 - accuracy: 0.9614 - get_f1: 0.9501 - val_loss: 0.5463 - val_accuracy: 0.8756 - val_get_f1: 0.8626\n",
            "Epoch 12/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0863 - accuracy: 0.9701 - get_f1: 0.9643 - val_loss: 0.5869 - val_accuracy: 0.8607 - val_get_f1: 0.8468\n",
            "Epoch 13/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0731 - accuracy: 0.9726 - get_f1: 0.9689 - val_loss: 0.7372 - val_accuracy: 0.8806 - val_get_f1: 0.8571\n",
            "Epoch 14/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0766 - accuracy: 0.9664 - get_f1: 0.9643 - val_loss: 0.6777 - val_accuracy: 0.8060 - val_get_f1: 0.7993\n",
            "Epoch 15/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0716 - accuracy: 0.9764 - get_f1: 0.9759 - val_loss: 0.6527 - val_accuracy: 0.8706 - val_get_f1: 0.8557\n",
            "Epoch 16/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0597 - accuracy: 0.9789 - get_f1: 0.9783 - val_loss: 0.6476 - val_accuracy: 0.8657 - val_get_f1: 0.8582\n",
            "Epoch 17/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0485 - accuracy: 0.9826 - get_f1: 0.9824 - val_loss: 0.7268 - val_accuracy: 0.8358 - val_get_f1: 0.8332\n",
            "Epoch 18/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0442 - accuracy: 0.9826 - get_f1: 0.9820 - val_loss: 0.7399 - val_accuracy: 0.8657 - val_get_f1: 0.8503\n",
            "Epoch 19/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0364 - accuracy: 0.9851 - get_f1: 0.9846 - val_loss: 0.7998 - val_accuracy: 0.8607 - val_get_f1: 0.8577\n",
            "Epoch 20/20\n",
            "804/804 [==============================] - 2s 3ms/step - loss: 0.0292 - accuracy: 0.9876 - get_f1: 0.9828 - val_loss: 0.8378 - val_accuracy: 0.8557 - val_get_f1: 0.8399\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.callbacks.History at 0x7fdce3f69128>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mTsepalaM8gH"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ueXtxFFzM8cE"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBrpSi13G6mg"
      },
      "source": [
        "#test_sequences = tok.texts_to_sequences(X_LSTM_test)\n",
        "#test_sequences_matrix = sequence.pad_sequences(test_sequences,maxlen=max_len)\n",
        "#accr = model.evaluate(test_sequences_matrix,y_LSTM_test)\n",
        "#print('Test set\\n  Loss: {:0.3f}\\n  Accuracy: {:0.3f}\\n  F1Score: {:0.3f}'.format(accr[0],accr[1],accr[2]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F7p6irtDJYjm"
      },
      "source": [
        "def add_lstm_pred(df):\n",
        "  if(\"lstm_pred\" in df.columns):\n",
        "    df = df.drop(\"lstm_pred\", axis=1)\n",
        "  test_sequences = tok.texts_to_sequences(df[\"text\"])\n",
        "  test_sequences_matrix = sequence.pad_sequences(test_sequences,maxlen=max_len)\n",
        "  #print(pd.Series(model.predict(test_sequences_matrix)[:,0]))\n",
        "  pred = pd.Series(model.predict(test_sequences_matrix)[:,0],name=\"lstm_pred\")\n",
        "  #print(pred)\n",
        "  return pd.concat([df.reset_index(drop=True), pred.reset_index(drop=True)], axis=1)\n",
        "  #return df\n",
        "def add_lstm_binary_pred(df):\n",
        "  df[\"lstm_binary_pred\"] = np.where(df[\"lstm_pred\"] > 0.5, 1, 0)\n",
        "  return df\n",
        "df = add_lstm_pred(df)\n",
        "X_train = add_lstm_pred(X_train)\n",
        "X_test = add_lstm_pred(X_test)\n",
        "\n",
        "df = add_lstm_binary_pred(df)\n",
        "X_train = add_lstm_binary_pred(X_train)\n",
        "X_test = add_lstm_binary_pred(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kHCI2kjDUPf-"
      },
      "source": [
        "#print(\"Accuracy Score: \", metrics.accuracy_score(X_test[\"lstm_binary_pred\"], y_test))\n",
        "#print(\"Recall Score: \", metrics.recall_score(X_test[\"lstm_binary_pred\"], y_test))\n",
        "#print(\"Precision Score: \", metrics.precision_score(X_test[\"lstm_binary_pred\"], y_test))\n",
        "#print(\"F1 Score: \", metrics.f1_score(X_test[\"lstm_binary_pred\"], y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kMEFZAvDBza"
      },
      "source": [
        "#Parent Classifier\n",
        "Support Vector Classifier uses predictions of subclassifiers to create ensemble classifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RPOShU3UGHag"
      },
      "source": [
        "parent = SVC(kernel=\"rbf\", C=1.0,gamma=\"scale\")\n",
        "\n",
        "\n",
        "#X is the formatted output of each of the classifiers\n",
        "#y is the label for each tweet-0 for clean, y for antisemitic\n",
        "#We originally tested using the outputs of the AdaBoost, Voter, Tfidf, and LSTM classifiers as inputs\n",
        "#columns = [\"ada_pred\", \"voter_pred\", \"tfidf_pred\", \"lstm_pred\"]\n",
        "\n",
        "#However, the additional inputs weakened the SVC's predictive power, so we decided to only use the LSTM classifier for prediction.\n",
        "columns = [\"lstm_pred\"]\n",
        "def train_parent(X_train,y_train):\n",
        "  X_train = X_train[columns]\n",
        "  parent.fit(X_train,y_train)\n",
        "def predict_parent(X):\n",
        "  preds = parent.predict(X[columns])\n",
        "  return preds\n",
        "def score_parent(X,y):\n",
        "  score = parent.score(X[columns],y)\n",
        "  return score\n",
        "def f1_parent(X,y):\n",
        "  preds = predict_parent(X)\n",
        "  return metrics.f1_score(y,preds)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UNXw36-HH0Q0"
      },
      "source": [
        "#This is the f1 score of the classifier taking into account the antisemitic probability given by the LSTM classifier\n",
        "train_parent(X_train, y_train)\n",
        "#print(\"Accuracy: \", accuracy_score(y_test,predict_parent(X_test)))\n",
        "#print(\"F1 Score: \", f1_parent(X_test, y_test))\n",
        "test_predictions = predict_parent(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NempaFXuTkLc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e7dd14b-a1b1-43ba-ddcc-7b6aa9a24303"
      },
      "source": [
        "print(test_predictions)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0 0 1 1 1 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 1 1 1 0 1 1 0 0 0\n",
            " 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            " 1 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 1 1 1 1 1 1 1 0 1 1\n",
            " 1 0 1 0 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8qCppSmbeeSg"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}