NLPF_06_P.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLPF_06_P.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOxZsolhp4k+RDrWKpxQJB3",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/satuelisa/NLPF/blob/main/NLPF_06_P.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As warm-up, let's take a look at the dynamic-programming algorithm to compute the *edit distance* ([Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance)) of two strings. Let's assume that the cost of inserting a character is a constant, the cost of inserting a character is another constant, and that the replacement cost depends on the pair being replaced, with a default value for when none is given."
      ],
      "metadata": {
        "id": "41-vyZpWAtDV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def editdist(p, q, elimination = 1, insertion = 1, defrep = 1, repcost = dict()):\n",
        "    d = dict()\n",
        "    np = len(p) + 1 # length of first string plus one \n",
        "    nq = len(q) + 1 # length of second string plus one\n",
        "    for i in range(np): # initialize each row\n",
        "        d[(i, 0)] = i * insertion\n",
        "    for j in range(nq): # initialize each column\n",
        "        d[(0, j)] = j * elimination\n",
        "    for i in range(1, np):\n",
        "        for j in range(1, nq):\n",
        "            lp = p[i - 1] # corresponding letter of the first string\n",
        "            lq = q[j - 1] # corresponding letter of the second string\n",
        "            eli = d[(i - 1, j)] + elimination\n",
        "            ins = d[(i, j - 1)] + insertion\n",
        "            ree = d[(i - 1, j - 1)] # no cost of replacement unless they differ\n",
        "            if lp != lq:\n",
        "              # include cost of that pair or default cost if undefined\n",
        "              ree += repcost.get((lp, lq), defrep) \n",
        "            d[(i, j)] = min(eli, ins, ree) # dynamic programming step: the cheapest option wins\n",
        "    return d[(np -1, nq - 1)] # final cost\n",
        " \n",
        "print(editdist(\"orthography\", \"ortografy\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aZJnl8r-Ay3-",
        "outputId": "ba028f89-f95b-40fd-e41e-99684f76dc43"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can specify special pairs for when replacement is easier than usual."
      ],
      "metadata": {
        "id": "n6bRW-UICqMz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(editdist(\"elisa\", \"élise\", repcost = {('e', 'é'): 0.1, ('é', 'e'): 0.2}))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TDpXUureCt8P",
        "outputId": "f1bd44a1-898b-4c55-fd5d-3d7adef9b0bf"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we have a different sense of \"word similarity\" that will be explored in the first question of this week's reflection quiz in the content of spell-checking. For the second question, we will start off with *n*-grams and `WordNet` to figure out what are possible ways a given sentence could continue. Let's use Twitter data (which is also one of the case studies in the R textbook)."
      ],
      "metadata": {
        "id": "BEVFjnrnC2z0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('twitter_samples') # once per computer"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dkOKylTGGlRr",
        "outputId": "e28e3def-95e0-47e7-f7dc-f96000968fae"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package twitter_samples to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/twitter_samples.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "What's in it?"
      ],
      "metadata": {
        "id": "we37nQPcHRBj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.corpus import twitter_samples\n",
        "print(twitter_samples.fileids())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8taGwQo1HR_o",
        "outputId": "f466a839-cab4-473e-fb04-e0666279a357"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['negative_tweets.json', 'positive_tweets.json', 'tweets.20150430-223406.json']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We do not need labels for this, just the text of the tweets themselves."
      ],
      "metadata": {
        "id": "zayHkDa5HhAR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "corpus = twitter_samples.strings()\n",
        "print(len(corpus))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q5Adhc-3HnBl",
        "outputId": "3dde1abd-b6ec-4c8f-fdd4-f881ff644ad4"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "30000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(corpus[42])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2ecLPscUH1q6",
        "outputId": "7f4a9e82-8c4b-4065-a499-db631b9b42ec"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Ouucchhh one of my wisdom teeth are coming through :(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this example, I will use bigrams only, meaning *n* = 2 (the second quiz question lets you play with this more)."
      ],
      "metadata": {
        "id": "gn37atzBIOCy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.tokenize import RegexpTokenizer\n",
        "tokenizer = RegexpTokenizer(r'\\w+') # remove punctuation\n",
        "\n",
        "from collections import defaultdict\n",
        "freq = defaultdict(int) # count bigraph occurences \n",
        "\n",
        "for tweet in corpus:\n",
        "  tokens = tokenizer.tokenize(tweet) \n",
        "  # not skipping stop words on purpose today\n",
        "  for bigram in nltk.ngrams(tokens, 2):\n",
        "      freq[bigram] += 1"
      ],
      "metadata": {
        "id": "gtgLMvqkI8ZH"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see how many combinations we got and how the frequencies are distributed (we should expect it to be heavy-tailed so we need a log scale on the vertical axis)."
      ],
      "metadata": {
        "id": "gzleQunNJ6Vd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(freq))\n",
        "from collections import Counter\n",
        "counts = Counter(freq.values())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WCZkOrGWKDML",
        "outputId": "afe944a1-7c60-479c-bbc1-0657be55f27e"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "146912\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "-AwBCSKyd-Z5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This can now be a barchart that effectively is a histogram."
      ],
      "metadata": {
        "id": "VcasSDifLegP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.bar(counts.keys(), counts.values())\n",
        "plt.xscale('log')\n",
        "plt.yscale('log')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "id": "W-vrZaU-Lhff",
        "outputId": "ca814387-2f23-4b2d-e86d-e3ca6cb71722"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAO+ElEQVR4nO3dQYjc53nH8e+vMkrBpT7EogdJqRRWiOgWWJRCLj4kZYUjK7glkZpLgpBwQDlHgVxLk0sPJmrdLRHbliJVmBCkWkE9ucpBB21ykiIEi0nw6qJ1XAQNBaHk6WHXZL3Z2f3Pzoxm9p3vB3yYd2b+8+hl+fH6+b/zTqoKSVJ7/mjcBUiSRsOAl6RGGfCS1CgDXpIaZcBLUqMMeElq1AvjLgDg5ZdfrkOHDo27DEnaVX72s599UFX7ej0/EQF/6NAhFhcXx12GJO0qSX611fO2aCSpUQa8JDXKgJekRhnwktQoA16SGjX0gE/ySpKfJnkrySvDvr4kqZtOAZ/kcpLHSe5tGJ9L8jDJUpKLa8MF/C/wx8DycMuVJHXVdR/8AvAD4F8/GkiyB7gEfJHVIL+b5Drw06r67yR/Bvw98LWhVrzBoYvvjPLyH/PL77363D5LkgbVaQVfVbeBDzcMHweWquq9qnoKXAVOVdXv1p7/H+ATva6Z5HySxSSLKysrOyhdkrSVQXrw+4H31z1eBvYneT3JPwH/xuqqf1NVNV9Vs1U1u29fz2/aSpJ2aOhHFVTVj4AfdXltkpPAyZmZmWGXIUlTb5AV/CPg4LrHB9bGOquqG1V1/qWXXhqgDEnSZgYJ+LvAkSSHk+wFTgPX+7lAkpNJ5p88eTJAGZKkzXTdJnkFuAMcTbKc5GxVPQMuALeAB8C1qrrfz4e7gpek0enUg6+qMz3GbwI3d/rh9uAlaXTGelSBK3hJGh3PopGkRo014L3JKkmjY4tGkhpli0aSGmWLRpIaZYtGkhpli0aSGmXAS1Kj7MFLUqPswUtSo2zRSFKjDHhJapQBL0mN8iarJDXKm6yS1ChbNJLUKANekhplwEtSowx4SWqUAS9JjXKbpCQ1ym2SktQoWzSS1CgDXpIaZcBLUqNeGHcBu92hi+/0/Z5ffu/VEVQiSR/nCl6SGmXAS1KjRhLwSV5MspjkS6O4viRpe50CPsnlJI+T3NswPpfkYZKlJBfXPfVt4NowC5Uk9afrCn4BmFs/kGQPcAk4ARwDziQ5luSLwC+Ax0OsU5LUp067aKrqdpJDG4aPA0tV9R5AkqvAKeBPgBdZDf3/S3Kzqn43tIolSZ0Msk1yP/D+usfLwOeq6gJAkq8DH/QK9yTngfMAn/rUpwYoQ5K0mZHtoqmqhar6zy2en6+q2aqa3bdv36jKkKSpNUjAPwIOrnt8YG2sM0+TlKTRGSTg7wJHkhxOshc4DVzv5wKeJilJo9N1m+QV4A5wNMlykrNV9Qy4ANwCHgDXqup+Px/uCl6SRqfrLpozPcZvAjd3+uFVdQO4MTs7e26n15AkbW6sh40lOQmcnJmZGWcZY9PloDIPJpO0U/6ikyQ1yt9klaRGuYKXpEZ5XLAkNcqAl6RG2YOXpEbZg5ekRtmikaRG+UWnXaLXl6L8IpSkXmzRSFKjbNFIUqMMeElqlAEvSY1yH7wkNcqbrJLUKFs0ktQoA16SGjXWLzppeDb7IpRfgpKmmyt4SWqUAS9JjfIsmsZtbN3YtpGmh9skJalRtmgkqVEGvCQ1yoCXpEYZ8JLUKL/oNIXcWSNNBwNewMdD38CX2mCLRpIaNfSAT/KZJG8leTvJN4d9fUlSN50CPsnlJI+T3NswPpfkYZKlJBcBqupBVb0BfAX4/PBLliR10XUFvwDMrR9Isge4BJwAjgFnkhxbe+414B3g5tAqlST1pVPAV9Vt4MMNw8eBpap6r6qeAleBU2uvv15VJ4Cv9bpmkvNJFpMsrqys7Kx6SVJPg+yi2Q+8v+7xMvC5JK8ArwOfYIsVfFXNA/MAs7OzNUAdkqRNDH2bZFW9C7zb5bWeJjm53DYp7X6D7KJ5BBxc9/jA2lhnniYpSaMzSMDfBY4kOZxkL3AauN7PBZKcTDL/5MmTAcqQJG2m6zbJK8Ad4GiS5SRnq+oZcAG4BTwArlXV/X4+3BW8JI1Opx58VZ3pMX6TAbZC2oPfHezHS7uTv+gkSY3yLBpJatRYA96brLvboYvv/MHRw5Imx1iPC66qG8CN2dnZc+OsQ90Z6NLuYYtGkhpli0aSGuUuGg2F/Xhp8tiikaRGGfAaKlfy0uSwBy9JjbIHL0mNskUjSY0y4CWpUQa8JDXKm6wamY9207irRhoPb7JKUqNs0UhSowx4SWqUAS9JjTLg9dx4s1V6vgx4PXfrg97Ql0ZnrL/olOQkcHJmZmacZWgMDHZp9NwmKUmNskWjieCKXho+A16SGmXAa2K4ipeGy4DXRDLspcEZ8JLUKANekho1kn3wSb4MvAr8KfDDqvqvUXyOJKm3ziv4JJeTPE5yb8P4XJKHSZaSXASoqh9X1TngDeCrwy1ZrdvqHHl781J3/bRoFoC59QNJ9gCXgBPAMeBMkmPrXvLdteelHTPUpZ3pHPBVdRv4cMPwcWCpqt6rqqfAVeBUVn0f+ElV/Xyz6yU5n2QxyeLKyspO65ck9TDoTdb9wPvrHi+vjX0L+ALw10ne2OyNVTVfVbNVNbtv374By1DrXMVL/RvJTdaqehN4c7vXediYJI3OoCv4R8DBdY8PrI114mFj2glX81I3gwb8XeBIksNJ9gKngetd35zkZJL5J0+eDFiGJGmjfrZJXgHuAEeTLCc5W1XPgAvALeABcK2q7ne9pit49WOrlbureukPde7BV9WZHuM3gZtDq0iSNBRjParAFo0kjY6/6KRdy7aMtDVX8NqVDHdpe67gJalRHhcsSY2yRaNdb2O7xlMopVW2aCSpUbZotKutX5l3WclL08SAl6RG2YNXU7bqv2+3onfFr9bYg9fUMcg1LWzRqGmGuaaZAS9JjTLgJalR3mSVpEZ5k1VTa6s99FILbNFIUqMMeElqlAEvSY0y4DXV7L2rZQa8JDXKbZLSFlzhazdzm6QkNcoWjbQJV+5qgQEvSY0y4CWpUQa8JDXKgJekRhnw0jYGveHqDVuNy9ADPsmnk/wwydvDvrYkqbtOAZ/kcpLHSe5tGJ9L8jDJUpKLAFX1XlWdHUWx0vPQ9Ue6pUnXdQW/AMytH0iyB7gEnACOAWeSHBtqdZKkHesU8FV1G/hww/BxYGltxf4UuAqcGnJ9kqQdGqQHvx94f93jZWB/kk8meQv4bJLv9HpzkvNJFpMsrqysDFCGNDy2ZdSSF4Z9war6NfBGh9fNA/MAs7OzNew6JGnaDbKCfwQcXPf4wNpYZ54mqd2kn5uv/p+AJsEgAX8XOJLkcJK9wGngej8X8DRJSRqdrtskrwB3gKNJlpOcrapnwAXgFvAAuFZV9/v5cFfw2m2GsTJ3da/npVMPvqrO9Bi/Cdzc6YdX1Q3gxuzs7LmdXkOStDl/0UkaAlflmkT+opMkNcoVvCQ1yhW8JDXK44IlqVEGvCQ1yh68tAM7PVLY3TZ6nuzBS1KjbNFIUqNs0UhSo2zRSFKjbNFIUqMMeElqlAEvSY3yJqu0Q/3sad/ute6n1yh4k1WSGmWLRpIaZcBLUqMMeElqlAEvSY0y4CWpUW6TlIakn62QvV67m7Y/7qZap5XbJCWpUbZoJKlRBrwkNcqAl6RGGfCS1CgDXpIaZcBLUqMMeElq1AvDvmCSF4F/AJ4C71bVvw/7MyRJ2+u0gk9yOcnjJPc2jM8leZhkKcnFteHXgber6hzw2pDrlSR11LVFswDMrR9Isge4BJwAjgFnkhwDDgDvr73st8MpU5LUr04BX1W3gQ83DB8Hlqrqvap6ClwFTgHLrIb8ltdPcj7JYpLFlZWV/iuXJtwwzmrZeI3Nrjmqnw70rJndb5CbrPv5/UodVoN9P/Aj4K+S/CNwo9ebq2q+qmaranbfvn0DlCFJ2szQb7JW1W+Ab3R5bZKTwMmZmZlhlyFJU2+QFfwj4OC6xwfWxjrzNElJGp1BAv4ucCTJ4SR7gdPA9X4u4HnwkjQ6XbdJXgHuAEeTLCc5W1XPgAvALeABcK2q7vfz4a7gJWl0OvXgq+pMj/GbwM2dfrg9eEkaHX/RSZIa5Vk0ktQof3RbkhqVqhp3DSRZAX4FvAT0Svtez202vnFsu8cvAx/0UXJXW/17BnmP8zS58zSqOepV2zDe09I8jWqOtnvduObpz6uq9zdFq2pi/gPm+31us/GNYx0eLz7vf88g73GeJneeRjVHztN452i3ztOk9eB7Hm2wxXObjW8c2+7xqOzkc7q8x3lynob5npbmaVRztN3rJnKeJqJFM25JFqtqdtx1TDrnaXvOUTfOUzeDztOkreDHZX7cBewSztP2nKNunKduBponV/CS1ChX8JLUKANekhplwEtSowz4DZK8mORfkvxzkq+Nu55JleTTSX6Y5O1x1zLJknx57W/pP5L85bjrmVRJPpPkrSRvJ/nmuOuZZGsZtZjkS9u9dioCPsnlJI+T3NswPpfkYZKlJBfXhl8H3q6qc8Brz73YMepnnmr1t3jPjqfS8epznn689rf0BvDVcdQ7Ln3O04OqegP4CvD5cdQ7Ln3mE8C3gWtdrj0VAQ8sAHPrB5LsAS4BJ4BjwJkkx1j9ZaqPfmv2t8+xxkmwQPd5mmYL9D9P3117fpos0Mc8JXkNeIcBjiDfpRboOE9Jvgj8Anjc5cJTEfBVdRv4cMPwcWBpbSX6FLgKnGL1x8MPrL1mKubnI33O09TqZ56y6vvAT6rq58+71nHq9++pqq5X1Qlgqlqjfc7TK8BfAH8DnEuyZUYN/Ue3d5H9/H6lDqvB/jngTeAHSV7l+X0FfZJtOk9JPgn8LfDZJN+pqr8bS3WTo9ff07eALwAvJZmpqrfGUdwE6fX39Aqr7dFPMH0r+M1sOk9VdQEgydeBD6rqd1tdZJoDflNV9RvgG+OuY9JV1a9Z7StrC1X1JquLBm2hqt4F3h1zGbtGVS10ed1UtSA2eAQcXPf4wNqYPs556sZ56sZ56mYo8zTNAX8XOJLkcJK9wGng+phrmkTOUzfOUzfOUzdDmaepCPgkV4A7wNEky0nOVtUz4AJwC3gAXKuq++Osc9ycp26cp26cp25GOU8eNiZJjZqKFbwkTSMDXpIaZcBLUqMMeElqlAEvSY0y4CWpUQa8JDXKgJekRhnwktSo/wd5391a+1a/lwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Yep, Zipf's Law is visible here as well. Now, not just *any* two words that happened to be following one another are necessarily a **meaningful** combination of words. We can use *pointwise mutual information* (PMI) as a way to quantify whether two words are together by chance or on purpose (the latter implying that they depend on each other in some sense). This can be automated with `ntlk`."
      ],
      "metadata": {
        "id": "Hpd055uKLk9G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# join the tweets into a single big string\n",
        "howmany = 10000 # faster with less text\n",
        "text = '\\n'.join(corpus[:howmany])\n",
        "length = 1000\n",
        "start = len(text) // 2 # midpoint\n",
        "end = start + length\n",
        "spacer = '*' * 20\n",
        "print(spacer, '\\nRaw\\n', text[start:end])\n",
        "\n",
        "# let's clean out all URLs (tweets contain a surprising amount of them) with a regular expression\n",
        "# see https://stackoverflow.com/questions/11331982/how-to-remove-any-url-within-a-string-in-python\n",
        "import re\n",
        "text = re.sub('http[s]?://\\S+', '', text)\n",
        "print(spacer, '\\nNo URLs\\n', text[start:end])\n",
        "\n",
        "# all HTMl tags need to go, too\n",
        "tags = re.compile(r'<[^>]+>') # like <em>\n",
        "symbols = re.compile(r'&[^;]+;') # like &amp;\n",
        "text = tags.sub('', text)\n",
        "text = symbols.sub(' ', text)\n",
        "print(spacer, '\\nNo HTML\\n', text[start:end])\n",
        "\n",
        "# also twitter username tags @something are best thrown out for our purposesr\n",
        "text = re.sub('@[A-Za-z0-9_]+', '', text)\n",
        "print(spacer, '\\nNo usernames\\n', text[start:end])\n",
        "\n",
        "# and yes, hashtags as well\n",
        "text = re.sub('#[A-Za-z0-9]+', '', text)\n",
        "print(spacer, '\\nNo hashtags\\n', text[start:end])\n",
        "\n",
        "# and all words containing numbers, we should not try to autocomplete numbers\n",
        "text = re.sub(r'[0-9]+', '', text)\n",
        "print(spacer, '\\nNo digits\\n', text[start:end])\n",
        "\n",
        "# no emojis, too\n",
        "# see https://thewebdev.info/2022/04/16/how-to-remove-emojis-from-a-string-in-python/\n",
        "\n",
        "emote = re.compile(pattern = \"[\" u\"\\U0001F600-\\U0001F64F\" u\"\\U0001F300-\\U0001F5FF\" u\"\\U0001F680-\\U0001F6FF\" u\"\\U0001F1E0-\\U0001F1FF\" \"]+\", flags = re.UNICODE)\n",
        "text = emote.sub(r'', text)\n",
        "print(spacer, '\\nNo emojis\\n', text[start:end])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8LofV6VyLtTa",
        "outputId": "cb10153f-eb04-4f4c-e2be-bd70d1b5c2b2"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "******************** \n",
            "Raw\n",
            " the #Cheltenham area :)\n",
            "Hi beautiful follow me please? :) @iggyazalea @lanadelrey @megannicole @madisonellebeer @tiffanyalvord $9\n",
            "@astro_lass They weren’t kale crisps :-) Why ruin a good thing?\n",
            "Hi BAM ! @BarsAndMelody \n",
            "Can you follow my bestfriend @969Horan696 ? \n",
            "She loves you a lot :) \n",
            "See you in Warsaw &lt;3 \n",
            "Love you &lt;3 x37\n",
            "@mykidsloveme2 Open Worldwide :)\n",
            "get the fuck outta here :D https://t.co/neJd4AaQWZ\n",
            "@StreetFighter Cool down guys. It's #SFVBeta, not release. They work on it. :)\n",
            "@CrazyLeoNet @Woodslenny @MSportLtd @FordPerformance It really looks Vantastic :) @VBGIE @FordOwnersClub @autocultureirl\n",
            "Today I finally have time to work on Xcylin again :D\n",
            "@UPCIreland Thank you. So if its a bundle shows the internet price no phone than realisticly only paying for Net?:)\n",
            "@PublicProtector :) True Education is the most powerful weapon which you can use to change the world,\"Nelson Mandela\n",
            "@agavrilychev2  follow @jnlazts &amp; http://t.co/RCvcYYO0Iq follow u back :)\n",
            "Most recent trip to \n",
            "******************** \n",
            "No URLs\n",
            " )\n",
            "@momoismail9 come kill it for me :)\n",
            "@ZaloraID hi, im so interested about your internship program. But sadly, I cant access the Career Page. Can u help me with this issue? :)\n",
            "@dischanmedia Its sad to hear about this, thank you so much for the overwhelmingly beautiful games, thank you for your hard work. :)\n",
            "@Murrayyyyyyyy aha yesss! Who are you most looking forward to seeing?:)\n",
            "Happy Friday you beauts! :) ♬ #FF @sophieintsticks @catecawley @Jon_Clifton78  @jasmoonbutterfl  @andreasson_ \n",
            "@halfwaytohemmo We do love to help!  :) /E\n",
            "@BionsenBeauty would love to win this deo is faaaaaabulous! :D #FreebieFriday #aluminiumfree #stayfresh xx\n",
            "@MrLythgoe @MeatBingo Yes, and John :)\n",
            "@ithloopwithcoco Thanks!   No worries, I'll navigate to your blog and check it out. :)\n",
            "@zara_arsalan thnks progrmr :-)\n",
            "@lewisssrg92 bet you do! Well I won't be getting any sleep it's a 9pm shift until 9am so it's quit hard work! But thank you :) xx\n",
            "@arabicaah figures :-)\n",
            "@andreaholzner hardly surprising that roses ins\n",
            "******************** \n",
            "No HTML\n",
            " tive poetry :)\n",
            "@R0CKMYDNA followed :)\n",
            "@iFLYflat I joined the @VirginAustralia #FrequentFlyer program :)\n",
            "I always end up breaking down and apologizing when I shouldn't have to. fuck me for being too nice :-))))))))\n",
            "@tonywhittaker Fabulous! Well we hope you all have a great day :-) ^KB\n",
            "@MusicTubenl Hi! Would you like an @imPastel concert? Let me know your city and country and I'll start working on it! Thanks :)\n",
            "@ignitionphoto You're very welcome. :)\n",
            "@HUNCOY Visit my blog  thanks :D\n",
            "@kunal8966 @Uber_Mumbai Did you enjoy the #LondonDairy #Icecream? Share with us the experience. :)\n",
            "@LeahCarla11  follow @jnlazts    follow u back :)\n",
            "@Playacreep I luv urs too :-)\n",
            "@craftbeercoll I have in the past. A hoodie covers a lot of sins. :)\n",
            "@AstonMerrygold bet your excited :) :')  xxxxxxx good luck hope it get to number one\n",
            "@MSLJim You're welcome Jim! Made me chuckle on the train this morning :-) Happy Friday to you too!\n",
            "@KageYashsa Shopping for a bit :p\n",
            "@JadeTheMong agreed haha :p\n",
            "@BakingBar Happy Birt\n",
            "******************** \n",
            "No usernames\n",
            " an many. :)\n",
            "  awww thanks guys!! Wishing you both a wicked weekend :) Travel safe! X\n",
            "Thank you  :)\n",
            " welcome to twitter world agaaaain sir :)\n",
            "Doli Siregar () retweeted your Tweet!   Many thanks! Cheers, Ninh. :) #Inspire\n",
            "#welcomeTweet  Thanks for following me.. :-) via \n",
            " :) It's a shoe box size of a shop, but the gems in there makes the geekiest of geeks cry at their wallets.\n",
            "  And by turning up, to be fair! It meant a lot that people came. :)\n",
            "Early birds are gone already das nice :)\n",
            "  i request those media Persons covering \"Rally\" please rotate their camera just once. :)\n",
            " OMG THAT SUCKS BUT AT LEAST YOU SEE ONE DIRECTION!:)\n",
            " am driving up tomorrow if you still wanted to come? :)\n",
            " fback? :)\n",
            "Eek! Its all got to go in the red suitcase.\n",
            "Beijing and China festival of science here I come :)\n",
            " \n",
            " Visit my blog  thanks :D\n",
            "  awesome! :)\n",
            "Twitter meni tebrik etdi :))\n",
            "Congratulations, you have over 700 followers\n",
            "Tomorrow back to school :D\n",
            "Someone bring me   sunshine a bob will love you forever?💗 :-)\n",
            "******************** \n",
            "No hashtags\n",
            " -)\n",
            "  Its a perfect Satyajit Ray's movie 'Porosh Pathor' situation\n",
            " Run for cover :-)) \n",
            "only I would drop 10 wine bottles   spill some of the wine all over myself @ work :-)\n",
            " jazmin Bonilla :), We Share the Latest Secret on 15000 Stars Kim Hollywood, Check the only secret on my Bio\n",
            " thanks for sharing! Wishing you a wicked weekend :)\n",
            "ROFL! Shades of Grey - netsec edition :-) \n",
            "Let's enjoy this day! :)\n",
            " oh will certainly be watching it anyway!! :D\n",
            " hi ate.  :)\n",
            "Stats for the day have arrived. 1 new follower and NO unfollowers :) via \n",
            " whenever we have another movie night :)\n",
            "  follow     follow u back :)\n",
            " fback? :)\n",
            " another amazing video Kev :) we are so proud to have such wonderful support from such a wonderful person :) \n",
            "Been listening to these two  on repeat. :D  and  More collab please? \n",
            "I kinda wanna fangirl over not an apology again even tho i already fangirled a lot when i first heard it :D\n",
            "  Yummy :D\n",
            " why thank you kind sir :)\n",
            " we are all good here. :) My sisters told me that they m\n",
            "******************** \n",
            "No digits\n",
            " th them :)\n",
            " I won't :)\n",
            " selamat sore,like model :)\n",
            " Sorry about this Nichola! Glad its back working now :) Always here if you need us. MKa\n",
            "Found someone I met long ago in Malta to come to  with me with the extra ticket I had :)   \n",
            "\": :)\"why baby?😘😘😘\n",
            " HA yes, make it quick tho :D\n",
            "Shobs incomplete :))\n",
            "   \n",
            " Thanks for being with us for so long :)\n",
            " i was :p\n",
            " hey i hope u have / are having a good day :)\n",
            " follow     follow u back :)\n",
            "What a great day to be at Silverstone today!  Beautiful Classic cars - and BONUS! We are there too! :D \n",
            " Im streaming now dude i pull about  views all the time :)\n",
            "Mum asked me if I wanted to go to the bookstore? Not sure if she knows me at all :) I'm off book shopping\n",
            " mind to follow back? :)\n",
            "I feel like I haven't been keeping up with my friends lately like I should be smh. Ganna text them to see if they're all doing good :-)\n",
            "  YEP - I would vote for her if I could ;-)) Just imagine Hillary Clinton as first female US president :-))\n",
            " thanks :)\n",
            " fback? :)\n",
            "Every tim\n",
            "******************** \n",
            "No emojis\n",
            " ks for being with us for so long :)\n",
            " i was :p\n",
            " hey i hope u have / are having a good day :)\n",
            " follow     follow u back :)\n",
            "What a great day to be at Silverstone today!  Beautiful Classic cars - and BONUS! We are there too! :D \n",
            " Im streaming now dude i pull about  views all the time :)\n",
            "Mum asked me if I wanted to go to the bookstore? Not sure if she knows me at all :) I'm off book shopping\n",
            " mind to follow back? :)\n",
            "I feel like I haven't been keeping up with my friends lately like I should be smh. Ganna text them to see if they're all doing good :-)\n",
            "  YEP - I would vote for her if I could ;-)) Just imagine Hillary Clinton as first female US president :-))\n",
            " thanks :)\n",
            " fback? :)\n",
            "Every time I visit a court I notice something funny and slice of life-so many stories hidden there untapped :)\n",
            "\" just had a dinner with my love  love you babe ! :) x\"\n",
            " thank you! :D\n",
            " thank you I really appreciate you taking the time to watch the video :)\n",
            "lmaoo me and my mca team gettin HELLA money dm me now if u wana \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = tokenizer.tokenize(text.lower()) # all in lower case \n",
        "find = nltk.BigramCollocationFinder.from_words(tokens) # of course there is a Trigram one in NTLK\n",
        "pmi = find.score_ngrams(nltk.BigramAssocMeasures().pmi) # score with PMI"
      ],
      "metadata": {
        "id": "jcx8xyp6QibA"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "OK. We can now use this to look at the last word of whatever a user gives as input to suggest, let's say *k* = 3 options, for the following word. First we have to figure out how to get from there the pairs that start with the word for which we want to find a follow-up."
      ],
      "metadata": {
        "id": "YrgH7JgeUr_N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "word = 'hey'\n",
        "for (pair, score) in pmi:\n",
        "  (first, second) = pair\n",
        "  if first == word:\n",
        "    print(second, score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kaXfMfxyWsKz",
        "outputId": "dbabd941-fd37-48a5-f647-3a797dc1f700"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "christine 9.858313013444844\n",
            "dazzle 9.858313013444844\n",
            "granny 9.858313013444844\n",
            "halla 9.858313013444844\n",
            "juaquin 9.858313013444844\n",
            "lynne 9.858313013444844\n",
            "obakeng 9.858313013444844\n",
            "bud 8.858313013444844\n",
            "cam 8.858313013444844\n",
            "mama 8.858313013444844\n",
            "marty 8.858313013444844\n",
            "hunny 8.273350512723688\n",
            "lesley 8.273350512723688\n",
            "paul 8.273350512723688\n",
            "raspberry 8.273350512723688\n",
            "tweeps 8.273350512723688\n",
            "james 7.983843895528706\n",
            "abby 7.858313013444844\n",
            "emily 7.858313013444844\n",
            "sarah 7.858313013444844\n",
            "dave 7.5363849185574825\n",
            "jay 7.5363849185574825\n",
            "buddy 7.0509580913872405\n",
            "fam 7.0509580913872405\n",
            "folks 6.858313013444844\n",
            "hun 6.858313013444844\n",
            "liam 6.688388012002532\n",
            "remember 6.536384918557484\n",
            "iphone 6.273350512723688\n",
            "girl 5.8886866624883645\n",
            "guys 5.4387741219310595\n",
            "babe 5.334751057387832\n",
            "mom 5.273350512723688\n",
            "found 5.157873295303752\n",
            "r 4.770850172194505\n",
            "there 4.16482605594552\n",
            "check 3.9275756758819576\n",
            "here 3.806650893622354\n",
            "long 3.648859647815895\n",
            "take 3.648859647815895\n",
            "welcome 3.648859647815895\n",
            "yeah 3.398881394807548\n",
            "someone 3.3191542023368132\n",
            "who 3.2584001712577173\n",
            "hey 3.1718124862616257\n",
            "or 2.9634952501369014\n",
            "wanna 2.9041167030579693\n",
            "thanks 2.5667585676010027\n",
            "would 2.563692264553218\n",
            "please 2.3269315529285315\n",
            "really 2.1303925588816455\n",
            "if 1.7393719407213375\n",
            "now 1.6635561590225958\n",
            "do 1.5273961353302266\n",
            "it 1.444685084420673\n",
            "all 1.1269939824197799\n",
            "follow 1.0574131135245395\n",
            "no 1.03176452615393\n",
            "and 0.9675420831996036\n",
            "i 0.8551465067796755\n",
            "can 0.7162059561422947\n",
            "you 0.6337083319063446\n",
            "that 0.3584671263616386\n",
            "have 0.27899707586482947\n",
            "d 0.27711243151988896\n",
            "a 0.21716441603361147\n",
            "is 0.02859027835878436\n",
            "for -0.34258559159360047\n",
            "my -0.4870922332729499\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The simplest way would be just to always suggest the best *k* options. Now, are the *best* options the lowest or the highest PMI? Let's see. Also note that many options have the same exact value of PMI which will make choosing complex. We will just, for now, favor the last one seen."
      ],
      "metadata": {
        "id": "mCEln1-wad4m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def autocomplete(word, scoring, k = 3):\n",
        "  top = []\n",
        "  bot = []\n",
        "  for (pair, score) in scoring:\n",
        "   (first, second) = pair\n",
        "   if first == word and second != word: # no word repetitions, those would just loop\n",
        "       if len(top) < k or score >= min(top, key = lambda o: o[1])[1]:\n",
        "         top.append((second, score))\n",
        "       if len(bot) < k or score <= max(bot, key = lambda o: o[1])[1]:\n",
        "         bot.append((second, score))\n",
        "       if len(bot) > k:\n",
        "         # ditch the HIGHEST score from bot(tom)\n",
        "         bot.remove(max(bot, key = lambda o: o[1]))\n",
        "       if len(top) > k:\n",
        "         # ditch the LOWEST score from top\n",
        "         top.remove(min(top, key = lambda o: o[1]))\n",
        "  return (top, bot)\n",
        "\n",
        "print(autocomplete('hey', pmi, k = 3))\n",
        "print(autocomplete('what', pmi, k = 2))\n",
        "print(autocomplete('cats', pmi, k = 4))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-U2eEmkGajfn",
        "outputId": "c66b6a94-85f7-42e7-90cd-b998de6cf178"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "([('juaquin', 9.858313013444844), ('lynne', 9.858313013444844), ('obakeng', 9.858313013444844)], [('is', 0.02859027835878436), ('for', -0.34258559159360047), ('my', -0.4870922332729499)])\n",
            "([('sights', 8.410387220407136), ('ux', 8.410387220407136)], [('have', -0.1689287171728786), ('this', -0.8703835497234671)])\n",
            "([('itchy', 14.222885445740701), ('boooo', 13.222885445740701), ('have', 4.643569508160686), ('and', 3.7471520147743025)], [('boooo', 13.222885445740701), ('have', 4.643569508160686), ('and', 3.7471520147743025), ('i', 2.219718939075534)])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The top ones seem to be often names. The bottom ones are not. Let's try to autocomplete a longer sentence with both to see how they behave. When a word does not occur a lot, the top and the bottom will overlap."
      ],
      "metadata": {
        "id": "Jr37BFSoeGME"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "suggestions = dict() # to avoid recomputing\n",
        "\n",
        "def grow(seed, suggestions, n = 20):\n",
        "  sent = { 0: [ seed ], 1: [ seed ] } # two sentences starting with the same seed word\n",
        "  for step in range(n):\n",
        "   for i in sent:\n",
        "     sentence = sent[i]\n",
        "     word = sentence[-1]\n",
        "     if word not in suggestions:\n",
        "       suggestions[word] = autocomplete(word, pmi, k = 1) # just the best one each\n",
        "     best = suggestions[word][i][0][0]\n",
        "     sentence.append(best)\n",
        "  return sent\n",
        "     \n",
        "seeds = [ 'help', 'morning', 'perfect', 'work' ]\n",
        "for word in seeds:\n",
        "    result = grow(word, suggestions)\n",
        "    for s in result:\n",
        "      print(' '.join(result[s]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kcxCM3JFeRy9",
        "outputId": "760434c1-487d-4661-dfbc-f2d35e229726"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "help raising the floor dammit stop lovey dovey in zdps no worries wacha niende online winners yesterday choroo ga nahi hua\n",
            "help to for i the to for i the to for i the to for i the to for i the\n",
            "morning wears on twitch join betis saludos hey obakeng op chocolates and zoomed away teleportation is بندر العنزي heiyo visit naaahhh\n",
            "morning you me we you me we you me we you me we you me we you me we you me\n",
            "perfect satyajit ray dept such gd wknd u wana make twinx cry hakhakhak t_____________t didn t zoom into piles anyway bbz\n",
            "perfect i the to for i the to for i the to for i the to for i the to for\n",
            "work zaine zac and zoomed away teleportation is بندر العنزي heiyo visit naaahhh d مطعم_هاشم yummy ala yeke its veritably cold\n",
            "work to for i the to for i the to for i the to for i the to for i the\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Well, the top option makes little sense and the bottom option got kind of repetitive. We could of course penalize in the score each possible word with how many times it already appears or how long ago it last appeared.\n",
        "\n",
        "It looks like bigrams are not long enough to convey context so we produce incoherent babbling. In the second question of the weekly reflection, see if you can improve upon this crappy excuse of an autocomplete."
      ],
      "metadata": {
        "id": "QA2yUXD1ha8s"
      }
    }
  ]
}