market_basket_analysis_(Ukraine_Conflict).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "market-basket-analysis (Ukraine Conflict).ipynb",
      "provenance": [],
      "collapsed_sections": [
        "N_pw0vh00ZNQ",
        "08-if852aIHA",
        "KhMq5TvFaO0L",
        "hLXw_eVR7YMs",
        "8XqLj57YmYNv",
        "P_bADBvC-H-C",
        "dKU0RFgVlhuF",
        "oj0iQErAllbD",
        "QBOJ9s5nln_3",
        "eEUu5KOpg_KC",
        "sKEr7-875oTm",
        "2wPUoZF65sP3",
        "wXk05CGo5y3t"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bigliolimatteo/AMD/blob/main/market_basket_analysis_(Ukraine_Conflict).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Market Basket Analysis (Ukraine Conflict)\n"
      ],
      "metadata": {
        "id": "0OFLgB6HbIWp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "DISCLAIMER: In this script there will be different passages which could be improved both performance-wise and interpretability-wise. Our goal was to propose the most academic and hadoop-like approach as possible in order to mimic the algorithms explained during the course. Most of these passages are highlighted and a more compact/fast approach is provided in the comments."
      ],
      "metadata": {
        "id": "6GGE3tokfZjF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# IMPORTANT: Put here your access token to kaggle\n",
        "\n",
        "import os\n",
        "os.environ[\"KAGGLE_USERNAME\"] = \"\"\n",
        "os.environ[\"KAGGLE_KEY\"] = \"\""
      ],
      "metadata": {
        "id": "LB1VxkJIYhfS"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download dataset from kaggle"
      ],
      "metadata": {
        "id": "N_pw0vh00ZNQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!kaggle datasets download -d bwandowando/ukraine-russian-crisis-twitter-dataset-1-2-m-rows\n",
        "!unzip ukraine-russian-crisis-twitter-dataset-1-2-m-rows.zip -d data"
      ],
      "metadata": {
        "id": "dF60Swj3aawH"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Move all *.gzip files to *.gz in order to have spark read directly the compressed file\n",
        "sh = \"\"\"\n",
        "for file in data/*.gzip; do\n",
        "    mv \"$file\" \"data/$(basename \"$file\" .gzip).gz\"\n",
        "done\n",
        "\"\"\"\n",
        "with open('script.sh', 'w') as file:\n",
        "  file.write(sh)\n",
        "\n",
        "!bash script.sh"
      ],
      "metadata": {
        "id": "90vEvzgdelx0"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Ingestion and Preprocessing"
      ],
      "metadata": {
        "id": "XBUknlxa0g-5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Import libraries and build spark context"
      ],
      "metadata": {
        "id": "08-if852aIHA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!pip install pyspark\n",
        "!pip install findspark\n",
        "!wget http://setup.johnsnowlabs.com/colab.sh -O - | bash"
      ],
      "metadata": {
        "id": "62YXF_hH0kh_"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "wFqNA6HvW1_m"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import csv\n",
        "import os\n",
        "from pyspark.sql import SparkSession, Row\n",
        "import random\n",
        "import sparknlp\n",
        "import math\n",
        "\n",
        "from sparknlp.base import *\n",
        "from sparknlp.annotator import *\n",
        "from pyspark.ml import Pipeline"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Build spark context\n",
        "spark = sparknlp.start()\n",
        "sc = spark.sparkContext"
      ],
      "metadata": {
        "id": "AQFBq7791SN9"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Ingest dataset"
      ],
      "metadata": {
        "id": "KhMq5TvFaO0L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Here we read one of the partitions of our dataset directly with spark (distributed read)\n",
        "FILENAME = r\"data/UkraineCombinedTweetsDeduped_FEB28_part1.csv.gz\"\n",
        "raw_df = spark.read.csv(FILENAME, header=True, escape=\"\\\"\", quote=\"\\\"\", multiLine=True)\n",
        "\n",
        "# A possible next step could be to work w/ multiple languages\n",
        "filtered_df = raw_df.where(raw_df.language == \"en\").select(\"text\")"
      ],
      "metadata": {
        "id": "fFVaRt0pau5U"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Text preprocessing "
      ],
      "metadata": {
        "id": "hLXw_eVR7YMs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# In this step we define a sparkNLP pipeline which will preprocess our data\n",
        "# by tokenizing it and removing unwanted tokens\n",
        "\n",
        "documentAssembler = DocumentAssembler() \\\n",
        "    .setInputCol(\"text\") \\\n",
        "    .setOutputCol(\"document\")\n",
        "\n",
        "tokenizer = Tokenizer() \\\n",
        "    .setInputCols([\"document\"]) \\\n",
        "    .setOutputCol(\"token\")\n",
        "\n",
        "linkRemover = Normalizer() \\\n",
        "    .setInputCols([\"token\"]) \\\n",
        "    .setOutputCol(\"tokensWoutLinks\") \\\n",
        "    .setCleanupPatterns([\"http\\S+|www\\S+|https\\S+\"]) \\\n",
        "    .setLowercase(True)\n",
        "\n",
        "punctuationRemover = Normalizer() \\\n",
        "    .setInputCols([\"tokensWoutLinks\"]) \\\n",
        "    .setOutputCol(\"tokensWoutLinksAndPuct\") \\\n",
        "    .setCleanupPatterns([\"(?U)[^\\w -]|_|-(?!\\w)|(?<!\\w)-\"])\n",
        "\n",
        "stopWordsCleaner = StopWordsCleaner.pretrained() \\\n",
        "      .setInputCols(\"tokensWoutLinksAndPuct\")\\\n",
        "      .setOutputCol(\"cleanedTokens\")\\\n",
        "      .setCaseSensitive(False)\n",
        "\n",
        "stemmer = Stemmer() \\\n",
        "    .setInputCols([\"cleanedTokens\"]) \\\n",
        "    .setOutputCol(\"cleanedStemmedTokens\")\n",
        "\n",
        "pipeline = Pipeline().setStages([\n",
        "    documentAssembler,\n",
        "    tokenizer,\n",
        "    linkRemover,\n",
        "    punctuationRemover,\n",
        "    stopWordsCleaner,\n",
        "    stemmer\n",
        "])\n",
        "\n",
        "result = pipeline.fit(filtered_df).transform(filtered_df)\n",
        "\n",
        "preprocessed_df = result.selectExpr(\"cleanedStemmedTokens.result\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DWhoi1XR6Qt9",
        "outputId": "1ca28072-c79b-4e7b-e6ae-2fd71666cd04"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "stopwords_en download started this may take some time.\n",
            "Approximate size to download 2.9 KB\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Market Basket Analysis Algorithm"
      ],
      "metadata": {
        "id": "aqrh2dE970NA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Introduction code"
      ],
      "metadata": {
        "id": "8XqLj57YmYNv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# We reduce the dimension of the dataframe for the naive approach\n",
        "preprocessed_df = preprocessed_df.limit(2000)\n",
        "\n",
        "# Because from now on we will define the algorithm using the Map and Reduce functions,\n",
        "# which are only defined over rdd in pyspark, we will convert our preprocessed dataframe into an rdd\n",
        "# with the same structure that we saw in the hadoop framework: (key, value)\n",
        "# + we will map each tweet in a list(set(tweet)) in order to remove duplicate words which are not useful in our analysis\n",
        "input_rdd = preprocessed_df.rdd.map(lambda x: (1, list(set(x[0]))))\n",
        "\n",
        "# This is our input rdd\n",
        "input_rdd.take(1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GspqK1Vv8PDl",
        "outputId": "2dba0b0b-b5f1-457a-96f0-36b80016b2b1"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(1,\n",
              "  ['construct',\n",
              "   'nation',\n",
              "   'visit',\n",
              "   'centr',\n",
              "   'vladimir',\n",
              "   'putin',\n",
              "   'moscow',\n",
              "   'space',\n",
              "   'site'])]"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# We start by defining a proper threshold in order to understand if an itemset is frequent.\n",
        "# To do that we count the number of baskets (tweets) in our whole dataset and we define \n",
        "# an itemset \"frequent\" if it appears in over x% of the baskets\n",
        "\n",
        "# Note that the countByKey function is basically equivalent to the map-reduce structure\n",
        "# .map(lambda x: (1, 1)).reduceByKey(lambda x,y: x + y)\n",
        "# But in our test it performed at twice the speed!\n",
        "\n",
        "THRESHOLD_PERCENTAGE = 0.1\n",
        "n_of_baskets = input_rdd.countByKey()[1]\n",
        "\n",
        "THRESHOLD = math.ceil(n_of_baskets * THRESHOLD_PERCENTAGE)"
      ],
      "metadata": {
        "id": "nVHGV7l7C3mU"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Naive approach\n",
        "\n",
        "First of all we try and code the algorithm without worrying too much about possible generalizations and without a proper \"software engineering\" approach, which will be discussed later"
      ],
      "metadata": {
        "id": "P_bADBvC-H-C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Frequent singletons"
      ],
      "metadata": {
        "id": "dKU0RFgVlhuF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# First of all we compute the frequent singleton itemsets\n",
        "\n",
        "# Extract all singletons\n",
        "singleton_itemsets_rdd = input_rdd \\\n",
        "                          .flatMap(lambda x: x[1]) \\\n",
        "                          .map(lambda x: (x,1))\n",
        "\n",
        "# Compute frequencies using a reduce operation\n",
        "singleton_itemsets_w_frequencies_rdd = singleton_itemsets_rdd \\\n",
        "                                        .reduceByKey(lambda x,y: x + y)\n",
        "\n",
        "# Filter singletons with a frequency higher than THRESHOLD\n",
        "frequent_singleton_itemsets_rdd = singleton_itemsets_w_frequencies_rdd \\\n",
        "                                    .filter(lambda x: x[1] > THRESHOLD)\n",
        "\n",
        "# These are examples of frequent singleton itemsets with the relative frequency\n",
        "frequent_singleton_itemsets_rdd.take(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "my1XVTIx-OIU",
        "outputId": "81b94e73-07e1-4b2c-b27b-44704e096bff"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('nation', 213),\n",
              " ('putin', 575),\n",
              " ('ukrain', 1413),\n",
              " ('russia', 1098),\n",
              " ('govern', 241),\n",
              " ('countri', 268),\n",
              " ('amp', 291),\n",
              " ('ukrainian', 387),\n",
              " ('kyiv', 215),\n",
              " ('russian', 477)]"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Frequent pairs"
      ],
      "metadata": {
        "id": "oj0iQErAllbD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# We can then compute frequent pairs by accounting only for pairs of frequent singletons.\n",
        "# Thanks to the monotonicity property that ensures that in this way we won't generate false negatives\n",
        "\n",
        "# Extract all frequent singleton without frequency\n",
        "frequent_singleton_itemsets_wout_freq_rdd = frequent_singleton_itemsets_rdd \\\n",
        "                                              .map(lambda x: (1, x[0]))\n",
        "\n",
        "# Generate all candidate pairs (note that we need to remove from the join all possible duplicates)\n",
        "# This is not the fastest approach as we could have leveraged specific functions (like .distinct())\n",
        "# but it is the most academic and hadoop like approach\n",
        "candidate_pairs_rdd = frequent_singleton_itemsets_wout_freq_rdd \\\n",
        "                      .join(frequent_singleton_itemsets_wout_freq_rdd) \\\n",
        "                      .filter(lambda x: len(set(x[1])) == len(x[1])) \\\n",
        "                      .map(lambda x: (tuple(sorted(x[1])), 1)) \\\n",
        "                      .reduceByKey(lambda x,y: x)\n",
        "\n",
        "# We can then generate a list of candidate pairs which can be broadcasted to all executors \n",
        "# due to the fact that we expect this to be small (it could be even part of our final output)\n",
        "candidate_pairs_list = candidate_pairs_rdd.map(lambda x: x[0]).collect()\n",
        "broadcasted_candidate_pairs_list = sc.broadcast(candidate_pairs_list)\n",
        "\n",
        "# Compute all possible pairs on our whole dataset\n",
        "input_w_unique_key_rdd = input_rdd \\\n",
        "                        .map(lambda x: x[1]) \\\n",
        "                        .zipWithUniqueId() \\\n",
        "                        .flatMap(lambda x: [(x[1], word) for word in x[0]])\n",
        "\n",
        "pair_itemsets_rdd = input_w_unique_key_rdd \\\n",
        "                  .join(input_w_unique_key_rdd)\n",
        "\n",
        "# Filter only pairs that are in the candidate frequent pairs list and compute their freq\n",
        "frequent_pairs_itemsets_rdd = pair_itemsets_rdd \\\n",
        "                                .filter(lambda x: x[1] in broadcasted_candidate_pairs_list.value) \\\n",
        "                                .map(lambda x: (x[1],1)) \\\n",
        "                                .reduceByKey(lambda x,y: x + y) \\\n",
        "                                .filter(lambda x: x[1] > THRESHOLD)\n",
        "\n",
        "# These are examples of frequent pairs itemsets with the relative frequency\n",
        "frequent_pairs_itemsets_rdd.take(5)\n",
        "\n",
        "\n",
        "\n",
        "# BONUS: We know that a more appropriate approach with respect to zipWithUniqueId\n",
        "# would have been to generate a unique index using an hash function like the one reported below\n",
        "# but it seems that (maybe due to the lazy not linear computational approach of spark)\n",
        "# this approach did not manage to generate a proper joined rdd (even if we used caching)\n",
        "\n",
        "#def unique_key(basket):\n",
        "#  unique_value = f\"{random()}-{basket}\"\n",
        "#  return hashlib.sha256(unique_value).encode(\"utf-8\")).hexdigest()\n",
        "\n",
        "#input_w_unique_key_rdd = input_rdd \\\n",
        "#                        .map(lambda x: (unique_key(x[1]), x[1])) \\\n",
        "#                        .flatMap(lambda x: [(x[0], word) for word in x[1]])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7CQkzPffF9Kf",
        "outputId": "e0f930f9-5a16-4b70-f8a9-0e345e84ed22"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(('russia', 'ukrain'), 889),\n",
              " (('countri', 'ukrain'), 226),\n",
              " (('countri', 'russia'), 209),\n",
              " (('russia', 'russian'), 273),\n",
              " (('russian', 'ukrain'), 383)]"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Frequent triplets"
      ],
      "metadata": {
        "id": "QBOJ9s5nln_3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# BE AWARE: because this is a naive approach, this piece of code takes ~ 30sec to compute\n",
        "# (performance will increase in the generalized approach)\n",
        "\n",
        "# Before generalizing the approach we will compute frequent triplets by accounting only for \n",
        "# those generated as join of a frequent pair and a frequent singleton.\n",
        "# Thanks to the monotonicity property that ensures that in this way we won't generate false negatives\n",
        "\n",
        "# Extract all frequent pairs without frequency\n",
        "frequent_pairs_itemsets_wout_freq_rdd = frequent_pairs_itemsets_rdd.map(lambda x: (1, x[0]))\n",
        "\n",
        "# Generate all candidate triplets (note that we need to remove from the join all possible duplicates)\n",
        "# This is not the fastest approach as we could have leveraged specific functions (like .distinct())\n",
        "# but it is the most academic and hadoop like approach\n",
        "# NOTE that we generate candidate triplets by joining frequent pairs, in this way we generate the lowest \n",
        "# possible number of candidates by leveraging the monotonicity property\n",
        "candidate_triplets_rdd = frequent_pairs_itemsets_wout_freq_rdd \\\n",
        "                      .join(frequent_pairs_itemsets_wout_freq_rdd) \\\n",
        "                      .map(lambda x: (1, sum(x[1], ()))) \\\n",
        "                      .map(lambda x: (1, tuple(set(x[1]))) ) \\\n",
        "                      .filter(lambda x: len(x[1]) == 3) \\\n",
        "                      .map(lambda x: (tuple(sorted(x[1])), 1)) \\\n",
        "                      .reduceByKey(lambda x,y: x)\n",
        "\n",
        "# We can then generate a list of candidate triplets which can be broadcasted to all executors \n",
        "# due to the fact that we expect this to be small (it could be even part of our final output)\n",
        "candidate_triplets_list = candidate_triplets_rdd.map(lambda x: x[0]).collect()\n",
        "broadcasted_candidate_triplets_list= sc.broadcast(candidate_triplets_list)\n",
        "\n",
        "# Compute all triplets on our whole dataset\n",
        "# Note that this is definetly an extremely expensive approach, we could have just saved\n",
        "# all the frequent pairs and join only those or (better) avoid having to compute all triplets\n",
        "# and directly check, for each tweet, the presence of a candidate triplet\n",
        "# (this approach will be used in the generalization phase, for now we stick with the naive approach)\n",
        "triplets_itemsets_rdd = input_w_unique_key_rdd \\\n",
        "                        .map(lambda x: (x[0], (x[1], ))) \\\n",
        "                        .join(pair_itemsets_rdd)  \\\n",
        "                        .map(lambda x: (1, sum(x[1], ())))\n",
        "\n",
        "# Filter only triplets that are in the candidate frequent triples list and compute their freq\n",
        "frequent_triplets_itemsets_rdd = triplets_itemsets_rdd \\\n",
        "                                .filter(lambda x: x[1] in broadcasted_candidate_triplets_list.value) \\\n",
        "                                .map(lambda x: (x[1],1)) \\\n",
        "                                .reduceByKey(lambda x,y: x + y) \\\n",
        "                                .filter(lambda x: x[1] > THRESHOLD)\n",
        "\n",
        "# These are examples of frequent pairs itemsets with the relative frequency\n",
        "frequent_triplets_itemsets_rdd.take(5)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Eo2pdV0fjk5",
        "outputId": "8879df49-963e-41b8-959d-3da99224ef50"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(('russia', 'russian', 'ukrain'), 243)]"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generalized Approach"
      ],
      "metadata": {
        "id": "eEUu5KOpg_KC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# We define some functions that could help make the code more readable \n",
        "\n",
        "def contains_duble(t): return len(set(t)) != len(t)\n",
        "\n",
        "def normalize_candidate_itemsets(candidate_itemsets_rdd):\n",
        "  return candidate_itemsets_rdd .filter(lambda x: not(contains_duble(x[1])) ) \\\n",
        "                                .map(lambda x: (tuple(sorted(x[1])), 1)) \\\n",
        "                                .reduceByKey(lambda x,y: x)"
      ],
      "metadata": {
        "id": "v7hA2LEdbeji"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Here we define the actual function which will perform the generalized approach\n",
        "\n",
        "def a_priori_algorithm(input_rdd, THRESHOLD, approach = \"single_machine-wise\"):\n",
        "\n",
        "  # Compute singleton itemsets with a frequency higher than THRESHOLD\n",
        "  frequent_singleton_itemsets_rdd = input_rdd \\\n",
        "                                    .flatMap(lambda x: x[1]) \\\n",
        "                                    .map(lambda x: (x,1)) \\\n",
        "                                    .reduceByKey(lambda x,y: x + y) \\\n",
        "                                    .filter(lambda x: x[1] > THRESHOLD) \\\n",
        "                                    .map(lambda x: (1, x[0]))\n",
        "\n",
        "  # Create a list which will contain all frequent itemsets\n",
        "  # this list will be updated during each step of the while cycle\n",
        "  frequent_itemsets_list = frequent_singleton_itemsets_rdd.map(lambda x: x[1]).collect()\n",
        "\n",
        "  # Compute candidate pairs of words by joining frequent singletons (leveraging monotonicity)\n",
        "  candidate_itemsets_rdd = normalize_candidate_itemsets( \n",
        "                            frequent_singleton_itemsets_rdd \\\n",
        "                            .join(frequent_singleton_itemsets_rdd))\n",
        "                          \n",
        "  # We start by looking for candidate pairs, we already compute frequent singletons\n",
        "  candidate_itemsets_length = 2\n",
        "\n",
        "\n",
        "  # If the approach is hadoop-wise we need to compute this rdd.\n",
        "  # note that we can put it here without worrying for a drop in performance\n",
        "  # because due to the lazy computation of spark if will be executed the first time \n",
        "  # (amd cached) during the first action\n",
        "  if approach == \"hadoop-wise\":\n",
        "    input_w_unique_key_rdd = input_rdd \\\n",
        "                          .map(lambda x: x[1]) \\\n",
        "                          .zipWithUniqueId() \\\n",
        "                          .flatMap(lambda x: [(x[1], word) for word in x[0]]) \\\n",
        "                          .map(lambda x: (x[0], (x[1], )))\n",
        "\n",
        "    itemsets_rdd = input_w_unique_key_rdd\n",
        "\n",
        "    # We cache the RDD as we will use it different times during our computation\n",
        "    input_w_unique_key_rdd.cache()\n",
        "\n",
        "\n",
        "  # Loop until we have candidates for frequent itemsets\n",
        "  while candidate_itemsets_rdd.count() != 0:\n",
        "\n",
        "    # Collect and broadcast the list of candidate itemsets \n",
        "    # (we expect its size to be manageable, due to the fact that it could be part of our output)\n",
        "    candidate_itemsets_list = candidate_itemsets_rdd.map(lambda x: x[0]).collect()\n",
        "    broadcasted_candidate_itemsets_list = sc.broadcast(candidate_itemsets_list)\n",
        "\n",
        "    # Generate all itemsets of length \"candidate_itemsets_length\" and filter them based on THRESHOLD\n",
        "    if approach == \"single_machine-wise\":\n",
        "      tmp_frequent_itemsets_rdd = input_rdd \\\n",
        "                                  .map(lambda x : [(candidate_itemset, 1) \n",
        "                                                    for candidate_itemset in broadcasted_candidate_itemsets_list.value \n",
        "                                                    if set(candidate_itemset).issubset(x[1])]) \\\n",
        "                                  .flatMap(lambda x : x) \\\n",
        "                                  .reduceByKey(lambda x,y: x + y) \\\n",
        "                                  .filter(lambda x: x[1] > THRESHOLD) \\\n",
        "                                  .map(lambda x: (1, x[0]))\n",
        "\n",
        "    elif approach == \"hadoop-wise\":\n",
        "      itemsets_rdd = input_w_unique_key_rdd \\\n",
        "                      .join(itemsets_rdd) \\\n",
        "                      .map(lambda x: (x[0], sum(x[1], ())))\n",
        "\n",
        "      tmp_frequent_itemsets_rdd = itemsets_rdd \\\n",
        "                                  .filter(lambda x: x[1] in broadcasted_candidate_itemsets_list.value) \\\n",
        "                                  .map(lambda x: (x[1],1)) \\\n",
        "                                  .reduceByKey(lambda x,y: x + y) \\\n",
        "                                  .filter(lambda x: x[1] > THRESHOLD) \\\n",
        "                                  .map(lambda x: (1, x[0]))\n",
        "                         \n",
        "    # Add the current frequent itemsets to the list of all frequent itemsets\n",
        "    frequent_itemsets_list += tmp_frequent_itemsets_rdd.map(lambda x: x[1]).collect()\n",
        "\n",
        "    # Increment the length of the candidate itemsets we are going to work with\n",
        "    candidate_itemsets_length += 1 \n",
        "\n",
        "    # Compute the candidate itemsets for the next iteration\n",
        "    candidate_itemsets_rdd = normalize_candidate_itemsets( \n",
        "                            tmp_frequent_itemsets_rdd \\\n",
        "                            .join(tmp_frequent_itemsets_rdd) \\\n",
        "                            .map(lambda x: (1, sum(x[1], ()))) \\\n",
        "                            .map(lambda x: (1, tuple(set(x[1]))) ) \\\n",
        "                            .filter(lambda x: len(x[1]) == candidate_itemsets_length))\n",
        "\n",
        "  # Unpersist previously cached RDD\n",
        "  if approach == \"hadoop-wise\":\n",
        "    input_w_unique_key_rdd.unpersist()\n",
        "\n",
        "  return frequent_itemsets_list"
      ],
      "metadata": {
        "id": "pSN9E7wRBHyZ"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "a_priori_algorithm(input_rdd, THRESHOLD, \"hadoop-wise\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wIQhpRq9M48m",
        "outputId": "82ee1539-9b26-40ae-ce22-9a6d8d4950d1"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['nation',\n",
              " 'putin',\n",
              " 'ukrain',\n",
              " 'russia',\n",
              " 'govern',\n",
              " 'countri',\n",
              " 'amp',\n",
              " 'ukrainian',\n",
              " 'kyiv',\n",
              " 'russian',\n",
              " 'peopl',\n",
              " 'russiaukrainewar',\n",
              " 'ukrainerussiawar',\n",
              " 'war',\n",
              " ('russia', 'ukrain'),\n",
              " ('countri', 'ukrain'),\n",
              " ('countri', 'russia'),\n",
              " ('russia', 'russian'),\n",
              " ('russian', 'ukrain'),\n",
              " ('amp', 'ukrain'),\n",
              " ('ukrain', 'ukrainian'),\n",
              " ('putin', 'ukrain'),\n",
              " ('putin', 'russia'),\n",
              " ('govern', 'ukrain'),\n",
              " ('russia', 'russian', 'ukrain')]"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "a_priori_algorithm(input_rdd, THRESHOLD, \"single_machine-wise\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SSV6CSnK1AIX",
        "outputId": "4395c677-89fb-4c0a-99bc-faac9eebfa70"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['nation',\n",
              " 'putin',\n",
              " 'ukrain',\n",
              " 'russia',\n",
              " 'govern',\n",
              " 'countri',\n",
              " 'amp',\n",
              " 'ukrainian',\n",
              " 'kyiv',\n",
              " 'russian',\n",
              " 'peopl',\n",
              " 'russiaukrainewar',\n",
              " 'ukrainerussiawar',\n",
              " 'war',\n",
              " ('russia', 'ukrain'),\n",
              " ('countri', 'ukrain'),\n",
              " ('amp', 'ukrain'),\n",
              " ('countri', 'russia'),\n",
              " ('govern', 'ukrain'),\n",
              " ('putin', 'ukrain'),\n",
              " ('ukrain', 'ukrainian'),\n",
              " ('putin', 'russia'),\n",
              " ('russian', 'ukrain'),\n",
              " ('russia', 'russian'),\n",
              " ('russia', 'russian', 'ukrain')]"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generate Results\n",
        "\n",
        "In here we develop a small script on top in order to easily generate results for different days"
      ],
      "metadata": {
        "id": "gIhfiQBambWd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Useful functions"
      ],
      "metadata": {
        "id": "sKEr7-875oTm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# We define some utility functions that could help make the code more readable \n",
        "\n",
        "def generate_preprocessing_pipeline():\n",
        "  documentAssembler = DocumentAssembler() \\\n",
        "    .setInputCol(\"text\") \\\n",
        "    .setOutputCol(\"document\")\n",
        "\n",
        "  tokenizer = Tokenizer() \\\n",
        "      .setInputCols([\"document\"]) \\\n",
        "      .setOutputCol(\"token\")\n",
        "\n",
        "  linkRemover = Normalizer() \\\n",
        "      .setInputCols([\"token\"]) \\\n",
        "      .setOutputCol(\"tokensWoutLinks\") \\\n",
        "      .setCleanupPatterns([\"http\\S+|www\\S+|https\\S+\"]) \\\n",
        "      .setLowercase(True)\n",
        "\n",
        "  punctuationRemover = Normalizer() \\\n",
        "      .setInputCols([\"tokensWoutLinks\"]) \\\n",
        "      .setOutputCol(\"tokensWoutLinksAndPuct\") \\\n",
        "      .setCleanupPatterns([\"(?U)[^\\w -]|_|-(?!\\w)|(?<!\\w)-\"])\n",
        "\n",
        "  stopWordsCleaner = StopWordsCleaner.pretrained() \\\n",
        "        .setInputCols(\"tokensWoutLinksAndPuct\")\\\n",
        "        .setOutputCol(\"cleanedTokens\")\\\n",
        "        .setCaseSensitive(False)\n",
        "\n",
        "  stemmer = Stemmer() \\\n",
        "      .setInputCols([\"cleanedTokens\"]) \\\n",
        "      .setOutputCol(\"cleanedStemmedTokens\")\n",
        "\n",
        "  return Pipeline().setStages([\n",
        "      documentAssembler,\n",
        "      tokenizer,\n",
        "      linkRemover,\n",
        "      punctuationRemover,\n",
        "      stopWordsCleaner,\n",
        "      stemmer\n",
        "  ])\n",
        "\n",
        "def apply_preprocessing_pipeline(pipeline, dataframe):\n",
        "  return pipeline.fit(dataframe).transform(dataframe)\\\n",
        "          .selectExpr(\"cleanedStemmedTokens.result\")\n",
        "\n",
        "def generate_numeric_threshold_from_percentage(threshold_percentage, rdd):\n",
        "  n_of_baskets = rdd.countByKey()[1]\n",
        "  return math.ceil(n_of_baskets * threshold_percentage)\n"
      ],
      "metadata": {
        "id": "vnrxgsXqnNnq"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Generalize to different days"
      ],
      "metadata": {
        "id": "2wPUoZF65sP3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Set limit = -1 to avoid limit\n",
        "# for FEB28 we specify only part1\n",
        "\n",
        "# The idea of this function is to apply the a_priori_algorithm just by specifing the date which we want to process \n",
        "# (and a few other input params)\n",
        "def generate_results(date, preprocessing_pipeline = generate_preprocessing_pipeline(), threshold_percentage = 0.2, limit = 20000):\n",
        "\n",
        "  # Extracting correct filename \n",
        "  if date[5:7] == \"02\":\n",
        "    month = \"FEB\"\n",
        "  elif date[5:7] == \"03\":\n",
        "    month = \"MAR\"\n",
        "  else:\n",
        "    raise Exception(\"Wrong month!\")\n",
        "\n",
        "  if date[5:7] == \"02\" and date[8:10] == \"28\":\n",
        "    suffix = \"_part1\"\n",
        "  else:\n",
        "    suffix = \"\"\n",
        "\n",
        "  FILENAME = f\"data/UkraineCombinedTweetsDeduped_{month}{date[8:10]}{suffix}.csv.gz\"\n",
        "\n",
        "  # Reading and filtering the dataset\n",
        "  raw_df = spark.read.csv(FILENAME, header=True, escape=\"\\\"\", quote=\"\\\"\", multiLine=True)\n",
        "  filtered_df = raw_df.where(raw_df.language == \"en\").select(\"text\")\n",
        "\n",
        "  # Preprocessing the dataset using a predefined pipeline (This avoid having to re-download the stopwords set each time)\n",
        "  preprocessed_df = apply_preprocessing_pipeline(preprocessing_pipeline, filtered_df)\n",
        "\n",
        "  # Subsetting our dataset if needed\n",
        "  if limit > 0:\n",
        "    preprocessed_df = preprocessed_df.limit(limit)\n",
        "\n",
        "  # Preparing the RDD for the a_priori_algorithm\n",
        "  input_rdd = preprocessed_df.rdd.map(lambda x: (1, list(set(x[0]))))\n",
        "\n",
        "  # Defining the threshold\n",
        "  THRESHOLD = generate_numeric_threshold_from_percentage(threshold_percentage, input_rdd)\n",
        "\n",
        "  # Returning the actual result\n",
        "  return a_priori_algorithm(input_rdd, THRESHOLD)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z981TrRPmxmp",
        "outputId": "76a5913f-375c-48ce-a325-7fed7bb03cea"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "stopwords_en download started this may take some time.\n",
            "Approximate size to download 2.9 KB\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Defining the pipeline once (donwload stopwords only once)\n",
        "preprocessing_pipeline = generate_preprocessing_pipeline()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lQxMG1Hc5FkE",
        "outputId": "7ccbc232-f50a-4fc3-fdb6-bef341f07938"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "stopwords_en download started this may take some time.\n",
            "Approximate size to download 2.9 KB\n",
            "[OK!]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Perform a computation on the dataset of 2022-02-28\n",
        "generate_results(\"2022-02-28\", preprocessing_pipeline)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y_rSDG52mZxb",
        "outputId": "53e11183-d127-4144-c4d8-1a149c521d7f"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['putin', 'ukrain', 'russia', 'russian', ('russia', 'ukrain')]"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Perform a computation on the dataset of 2022-03-04 (the day in which a power plant was attacked)\n",
        "generate_results(\"2022-03-04\", preprocessing_pipeline)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zch2u8bbpbng",
        "outputId": "d287d281-92f5-4dca-bb78-656a4ab8fc5b"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['ukrain',\n",
              " 'russian',\n",
              " 'plant',\n",
              " 'power',\n",
              " 'russia',\n",
              " 'nuclear',\n",
              " 'zaporizhzhia',\n",
              " ('russian', 'ukrain'),\n",
              " ('plant', 'power'),\n",
              " ('plant', 'ukrain'),\n",
              " ('power', 'ukrain'),\n",
              " ('russia', 'ukrain'),\n",
              " ('nuclear', 'plant'),\n",
              " ('nuclear', 'power'),\n",
              " ('nuclear', 'ukrain'),\n",
              " ('plant', 'power', 'ukrain'),\n",
              " ('nuclear', 'plant', 'ukrain'),\n",
              " ('nuclear', 'power', 'ukrain'),\n",
              " ('nuclear', 'plant', 'power'),\n",
              " ('nuclear', 'plant', 'power', 'ukrain')]"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Compute results for different days"
      ],
      "metadata": {
        "id": "wXk05CGo5y3t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# The small piece of code below just computes the results for all the days in our dataset and\n",
        "# writes them down to a file so that they could be easier to compare\n",
        "\n",
        "from datetime import date, timedelta\n",
        "import json\n",
        "\n",
        "# Define daterange\n",
        "def daterange(start_date, end_date):\n",
        "    for n in range(int((end_date - start_date).days)):\n",
        "        yield start_date + timedelta(n)\n",
        "start_date = date(2022, 2, 27)\n",
        "end_date = date(2022, 3, 27)\n",
        "\n",
        "# Prepare result dict\n",
        "result_dict = dict()\n",
        "\n",
        "# Define once the preprocessing pipeline\n",
        "preprocessing_pipeline = generate_preprocessing_pipeline()\n",
        "\n",
        "# Generate and store results for each day\n",
        "for single_date in daterange(start_date, end_date):\n",
        "  date_str = single_date.strftime(\"%Y-%m-%d\")\n",
        "  print(\"processing \" + date_str)\n",
        "  result_dict[date_str] = generate_results(date_str, preprocessing_pipeline, threshold_percentage=0.15, limit=1000)\n",
        "\n",
        "# Write results to a file\n",
        "with open('result.json', 'w') as fp:\n",
        "    json.dump(result_dict, fp)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YJ7yk-whpnPI",
        "outputId": "6164b629-72e8-4b54-8b24-866969cd9c98"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "stopwords_en download started this may take some time.\n",
            "Approximate size to download 2.9 KB\n",
            "[OK!]\n",
            "processing 2022-02-27\n",
            "processing 2022-02-28\n",
            "processing 2022-03-01\n",
            "processing 2022-03-02\n",
            "processing 2022-03-03\n",
            "processing 2022-03-04\n",
            "processing 2022-03-05\n",
            "processing 2022-03-06\n",
            "processing 2022-03-07\n",
            "processing 2022-03-08\n",
            "processing 2022-03-09\n",
            "processing 2022-03-10\n",
            "processing 2022-03-11\n",
            "processing 2022-03-12\n",
            "processing 2022-03-13\n",
            "processing 2022-03-14\n",
            "processing 2022-03-15\n",
            "processing 2022-03-16\n",
            "processing 2022-03-17\n",
            "processing 2022-03-18\n",
            "processing 2022-03-19\n",
            "processing 2022-03-20\n",
            "processing 2022-03-21\n",
            "processing 2022-03-22\n",
            "processing 2022-03-23\n",
            "processing 2022-03-24\n",
            "processing 2022-03-25\n",
            "processing 2022-03-26\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize the different results\n",
        "result_dict"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QSYn0cL76XUg",
        "outputId": "2fe232e0-5bc3-4a30-dd5b-124ccfdc2fef"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'2022-02-27': ['ukrain',\n",
              "  'russian',\n",
              "  'putin',\n",
              "  'russia',\n",
              "  'kyiv',\n",
              "  'war',\n",
              "  'anonym',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'ukrain'),\n",
              "  ('putin', 'ukrain'),\n",
              "  ('ukrain', 'war'),\n",
              "  ('anonym', 'russian'),\n",
              "  ('anonym', 'ukrain'),\n",
              "  ('anonym', 'russian', 'ukrain')],\n",
              " '2022-02-28': ['putin',\n",
              "  'ukrain',\n",
              "  'russia',\n",
              "  'countri',\n",
              "  'amp',\n",
              "  'ukrainian',\n",
              "  'russian',\n",
              "  'russiaukrainewar',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('amp', 'ukrain'),\n",
              "  ('putin', 'ukrain'),\n",
              "  ('ukrain', 'ukrainian'),\n",
              "  ('putin', 'russia'),\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'russian'),\n",
              "  ('russia', 'russian', 'ukrain')],\n",
              " '2022-03-01': ['ukrain',\n",
              "  'russia',\n",
              "  'ukrainerussiawar',\n",
              "  'russian',\n",
              "  'kyiv',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-02': ['ukrain',\n",
              "  'russia',\n",
              "  'war',\n",
              "  'russian',\n",
              "  'peopl',\n",
              "  'ukrainerussiawar',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-03': ['ukrain', 'russian', 'russia', 'kyiv', ('russia', 'ukrain')],\n",
              " '2022-03-04': ['ukrain',\n",
              "  'russian',\n",
              "  'plant',\n",
              "  'power',\n",
              "  'russia',\n",
              "  'nuclear',\n",
              "  'putin',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('plant', 'power'),\n",
              "  ('power', 'ukrain'),\n",
              "  ('russia', 'ukrain'),\n",
              "  ('nuclear', 'plant'),\n",
              "  ('nuclear', 'power'),\n",
              "  ('nuclear', 'ukrain')],\n",
              " '2022-03-05': ['russian', 'ukrain', 'russia', ('russia', 'ukrain')],\n",
              " '2022-03-06': ['russia',\n",
              "  'russian',\n",
              "  'ukrain',\n",
              "  'putin',\n",
              "  'ukrainian',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'ukrain'),\n",
              "  ('ukrain', 'ukrainian')],\n",
              " '2022-03-07': ['ukrain',\n",
              "  'russian',\n",
              "  'russia',\n",
              "  'ukrainerussianwar',\n",
              "  'video',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'ukrain'),\n",
              "  ('ukrain', 'video')],\n",
              " '2022-03-08': ['ukrain',\n",
              "  'russian',\n",
              "  'russia',\n",
              "  'ukrainerussianwar',\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-09': ['ukrain', 'russian', 'russia', ('russia', 'ukrain')],\n",
              " '2022-03-10': ['ukrain', 'russian', 'russia'],\n",
              " '2022-03-11': ['ukrain',\n",
              "  'russia',\n",
              "  'ukrainian',\n",
              "  'russian',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('ukrain', 'ukrainian'),\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-12': ['ukrain',\n",
              "  'putin',\n",
              "  'russia',\n",
              "  'ukrainian',\n",
              "  'russian',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('ukrain', 'ukrainian')],\n",
              " '2022-03-13': ['russian', 'ukrain', 'russia', 'putin', ('russia', 'ukrain')],\n",
              " '2022-03-14': ['ukrain', 'putin', 'russia', 'war', 'russian'],\n",
              " '2022-03-15': ['russian',\n",
              "  'ukrain',\n",
              "  'russia',\n",
              "  'putin',\n",
              "  'standwithukrain',\n",
              "  ('russia', 'ukrain')],\n",
              " '2022-03-16': ['ukrain',\n",
              "  'russian',\n",
              "  'russia',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'ukrain')],\n",
              " '2022-03-17': ['ukrain', 'russia', 'russian', ('russia', 'ukrain')],\n",
              " '2022-03-18': ['ukrain',\n",
              "  'putin',\n",
              "  'ukrainian',\n",
              "  'russia',\n",
              "  'russian',\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-19': ['ukrain',\n",
              "  'russian',\n",
              "  'putin',\n",
              "  'russia',\n",
              "  'standwithukrain',\n",
              "  ('russia', 'ukrain')],\n",
              " '2022-03-20': ['amp',\n",
              "  'ukrain',\n",
              "  'russia',\n",
              "  'russian',\n",
              "  '500daysoftigraygenocid',\n",
              "  'tigrai',\n",
              "  'standwithukrain',\n",
              "  ('500daysoftigraygenocid', 'tigrai')],\n",
              " '2022-03-21': ['ukrain', 'russia', 'russian', 'standwithukrain'],\n",
              " '2022-03-22': ['war',\n",
              "  'ukrain',\n",
              "  'ukrainian',\n",
              "  'russia',\n",
              "  'standwithukrain',\n",
              "  'russian',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-23': ['ukrain',\n",
              "  'ukrainian',\n",
              "  'russian',\n",
              "  'russia',\n",
              "  'standwithukrain',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'ukrain')],\n",
              " '2022-03-24': ['ukrain',\n",
              "  'russia',\n",
              "  'russian',\n",
              "  'putin',\n",
              "  'ukrainian',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-25': ['ukrain',\n",
              "  'war',\n",
              "  'russia',\n",
              "  'ukrainian',\n",
              "  'standwithukrain',\n",
              "  'russian',\n",
              "  ('russia', 'ukrain'),\n",
              "  ('ukrain', 'ukrainian'),\n",
              "  ('russian', 'ukrain')],\n",
              " '2022-03-26': ['ukrain',\n",
              "  'russian',\n",
              "  'russia',\n",
              "  ('russian', 'ukrain'),\n",
              "  ('russia', 'ukrain')]}"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "YJynGSeKiGa4"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}