applied_text_mining_in_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applied text mining in Python\n",
    "----\n",
    "\n",
    "**Applied text mining in Python** was the fourth course in the **Applied Data Science with Python** specialization. This notebook shows my work for the four assignments that I had to solve during this course:\n",
    "\n",
    "* ** Assignment 1: Cleaning messy medical data**\n",
    "\n",
    "In this assignment, I had to use *Pandas* and *regex* to clean and extract relevant information from medical notes. The final objective was to correctly identify all of the different date variants encoded in this dataset and to properly normalize and sort the dates.\n",
    " \n",
    "* ** Assignment 2:  Introduction to NLTK**\n",
    "\n",
    "This assignment was structured in two parts. In the first part of this assignment I've explored the Herman Melville novel Moby Dick with *nltk*. In the second part of the assignment, I have created a spelling recommender function that uses *nltk* to find words similar to the misspelling.\n",
    "\n",
    "* **Assignment 3:  Spam classifier**\n",
    "\n",
    "In this assignment, I've created a model to predict if a text message is spam or not. This assignment required  knowledge on both text mining as well as machine learning.\n",
    "\n",
    "* **Assignment 4: Document similarity & topic modelling**\n",
    "\n",
    "This assignment was split in two parts. In the first part of the assignment, I've created two functions *doc_to_synsets* and *similarity_score* to find the path similarity between two documents. In the second part of the assignment, I've used Gensim's LDA (Latent Dirichlet Allocation) model to model topics in text data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 1\n",
    "----\n",
    "In this assignment, you'll be working with messy medical data and using regex to extract relevant infromation from the data. \n",
    "\n",
    "Each line of the `dates.txt` file corresponds to a medical note. Each note has a date that needs to be extracted, but each date is encoded in one of many formats.\n",
    "\n",
    "The goal of this assignment is to correctly identify all of the different date variants encoded in this dataset and to properly normalize and sort the dates. \n",
    "\n",
    "Here is a list of some of the variants you might encounter in this dataset:\n",
    "* 04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
    "* Mar-20-2009; Mar 20, 2009; March 20, 2009;  Mar. 20, 2009; Mar 20 2009;\n",
    "* 20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
    "* Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
    "* Feb 2009; Sep 2009; Oct 2010\n",
    "* 6/2008; 12/2009\n",
    "* 2009; 2010\n",
    "\n",
    "Once you have extracted these date patterns from the text, the next step is to sort them in ascending chronological order accoring to the following rules:\n",
    "* Assume all dates in xx/xx/xx format are mm/dd/yy\n",
    "* Assume all dates where year is encoded in only two digits are years from the 1900's (e.g. 1/5/89 is January 5th, 1989)\n",
    "* If the day is missing (e.g. 9/2009), assume it is the first day of the month (e.g. September 1, 2009).\n",
    "* If the month is missing (e.g. 2010), assume it is the first of January of that year (e.g. January 1, 2010).\n",
    "\n",
    "With these rules in mind, find the correct date in each note and return a pandas Series in chronological order of the original Series' indices.\n",
    "\n",
    "For example if the original series was this:\n",
    "\n",
    "    0    1999\n",
    "    1    2010\n",
    "    2    1978\n",
    "    3    2015\n",
    "    4    1985\n",
    "\n",
    "Your function should return this:\n",
    "\n",
    "    0    2\n",
    "    1    4\n",
    "    2    0\n",
    "    3    1\n",
    "    4    3\n",
    "\n",
    "Your score will be calculated using [Kendall's tau](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient), a correlation measure for ordinal data.\n",
    "\n",
    "*This function should return a Series of length 500 and dtype int.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         03/25/93 Total time of visit (in minutes):\\n\n",
       "1                       6/18/85 Primary Care Doctor:\\n\n",
       "2    sshe plans to move as of 7/8/71 In-Home Servic...\n",
       "3                7 on 9/27/75 Audit C Score Current:\\n\n",
       "4    2/6/96 sleep studyPain Treatment Pain Level (N...\n",
       "5                    .Per 7/06/79 Movement D/O note:\\n\n",
       "6    4, 5/18/78 Patient's thoughts about current su...\n",
       "7    10/24/89 CPT Code: 90801 - Psychiatric Diagnos...\n",
       "8                         3/7/86 SOS-10 Total Score:\\n\n",
       "9             (4/10/71)Score-1Audit C Score Current:\\n\n",
       "dtype: object"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "doc = []\n",
    "with open('datasets/dates.txt') as file:\n",
    "    for line in file:\n",
    "        doc.append(line)\n",
    "\n",
    "df = pd.Series(doc)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def date_sorter():   \n",
    "    # Your code here\n",
    "    regex1 = '(\\d{1,2}[/-]\\d{1,2}[/-]\\d{2,4})'\n",
    "    regex2 = '((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[\\S]*[+\\s]\\d{1,2}[,]{0,1}[+\\s]\\d{4})'\n",
    "    regex3 = '(\\d{1,2}[+\\s](?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[\\S]*[+\\s]\\d{4})'\n",
    "    regex4 = '((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[\\S]*[+\\s]\\d{4})'\n",
    "    regex5 = '(\\d{1,2}[/-][1|2]\\d{3})'\n",
    "    regex6 = '([1|2]\\d{3})'\n",
    "    full_regex = '(%s|%s|%s|%s|%s|%s)' %(regex1, regex2, regex3, regex4, regex5, regex6)\n",
    "    parsed_date = df.str.extract(full_regex)\n",
    "    parsed_date = parsed_date.iloc[:,0].str.replace('Janaury', 'January').str.replace('Decemeber', 'December')\n",
    "    parsed_date = pd.Series(pd.to_datetime(parsed_date))\n",
    "    parsed_date = parsed_date.sort_values(ascending=True).index\n",
    "    return pd.Series(parsed_date.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 2 \n",
    "----\n",
    "\n",
    "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1 - Analyzing Moby Dick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# If you would like to work with the raw text you can use 'moby_raw'\n",
    "with open('datasets/moby.txt', 'r') as f:\n",
    "    moby_raw = f.read()\n",
    "    \n",
    "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
    "moby_tokens = nltk.word_tokenize(moby_raw)\n",
    "text1 = nltk.Text(moby_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1\n",
    "\n",
    "How many tokens (words and punctuation symbols) are in text1?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "254989"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_one():\n",
    "    return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
    "\n",
    "example_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "\n",
    "How many unique tokens (unique words and punctuation) does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20755"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_two():\n",
    "    return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
    "\n",
    "example_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3\n",
    "\n",
    "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16900"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "def example_three():\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
    "    return len(set(lemmatized))\n",
    "\n",
    "example_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "\n",
    "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08139566804842562"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    return len(set(text1))/len(text1)\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "What percentage of tokens is 'whale'or 'Whale'?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4125668166077752"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    return (len([x for x in text1 if (x=='Whale') | (x=='whale')])/len(text1))*100\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
    "\n",
    "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(',', 19204),\n",
       " ('the', 13715),\n",
       " ('.', 7308),\n",
       " ('of', 6513),\n",
       " ('and', 6010),\n",
       " ('a', 4545),\n",
       " ('to', 4515),\n",
       " (';', 4173),\n",
       " ('in', 3908),\n",
       " ('that', 2978),\n",
       " ('his', 2459),\n",
       " ('it', 2196),\n",
       " ('I', 2097),\n",
       " ('!', 1767),\n",
       " ('is', 1722),\n",
       " ('--', 1713),\n",
       " ('with', 1659),\n",
       " ('he', 1658),\n",
       " ('was', 1639),\n",
       " ('as', 1620)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    df = pd.value_counts(text1) \n",
    "    return [(x, y) for x, y in zip(df.index[:20], df[:20])]\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "What tokens have a length of greater than 5 and frequency of more than 150?\n",
    "\n",
    "*This function should return a sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Captain',\n",
       " 'Pequod',\n",
       " 'Queequeg',\n",
       " 'Starbuck',\n",
       " 'almost',\n",
       " 'before',\n",
       " 'himself',\n",
       " 'little',\n",
       " 'seemed',\n",
       " 'should',\n",
       " 'though',\n",
       " 'through',\n",
       " 'whales',\n",
       " 'without']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    df = pd.value_counts(text1)\n",
    "    ans = [x for x, y in zip(df.index, df) if (len(x)>5) & (y>150)]   \n",
    "    return sorted(ans)\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Find the longest word in text1 and that word's length.\n",
    "\n",
    "*This function should return a tuple `(longest_word, length)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"twelve-o'clock-at-night\", 23)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    len_words = [len(x) for x in text1]   \n",
    "    return (text1[np.argmax(len_words)], np.max(len_words))\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What unique words have a frequency of more than 2000? What is their frequency?\n",
    "\n",
    "\"Hint:  you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
    "\n",
    "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(13715, 'the'),\n",
       " (6513, 'of'),\n",
       " (6010, 'and'),\n",
       " (4545, 'a'),\n",
       " (4515, 'to'),\n",
       " (3908, 'in'),\n",
       " (2978, 'that'),\n",
       " (2459, 'his'),\n",
       " (2196, 'it'),\n",
       " (2097, 'I')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    df = pd.value_counts(text1)\n",
    "    df = df[df>2000]\n",
    "    return [(y, x) for x, y in zip(df.index, df) if x.isalpha()]\n",
    "\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "What is the average number of tokens per sentence?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.881952902963864"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    tokens_sent = nltk.tokenize.sent_tokenize(moby_raw)\n",
    "    num_tokens = [len(nltk.word_tokenize(x)) for x in tokens_sent]\n",
    "    return np.mean(num_tokens)\n",
    "\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
    "\n",
    "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', 32730), ('IN', 28657), ('DT', 25867), (',', 19204), ('JJ', 17620)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    pos = [y for x, y in nltk.pos_tag(text1)]\n",
    "    df = pd.value_counts(pos)\n",
    "    ans = [(x, y) for x, y in zip(df.index[:5], df[:5])]\n",
    "    return ans\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2 - Spelling Recommender\n",
    "\n",
    "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
    "\n",
    "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
    "\n",
    "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
    "\n",
    "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import words\n",
    "\n",
    "correct_spellings = words.words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mada/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['corpulent', 'indecence', 'validate']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    spellings = []\n",
    "    for entry in entries:\n",
    "        entry_spellings = [x for x in correct_spellings if x[0]==entry[0]]\n",
    "        jac_distance = [nltk.jaccard_distance(set(nltk.ngrams(entry, n=3)), set(nltk.ngrams(x, n=3))) for \n",
    "                        x in entry_spellings]\n",
    "        spellings.append(entry_spellings[np.argmin(jac_distance)])\n",
    "    return spellings\n",
    "    \n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mada/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['cormus', 'incendiary', 'valid']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    spellings = []\n",
    "    for entry in entries:\n",
    "        entry_spellings = [x for x in correct_spellings if x[0]==entry[0]]\n",
    "        jac_distance = [nltk.jaccard_distance(set(nltk.ngrams(entry, n=4)), set(nltk.ngrams(x, n=4))) for \n",
    "                        x in entry_spellings]\n",
    "        spellings.append(entry_spellings[np.argmin(jac_distance)])\n",
    "    return spellings\n",
    "   \n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['corpulent', 'intendence', 'validate']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    spellings = []\n",
    "    for entry in entries:\n",
    "        entry_spellings = [x for x in correct_spellings if x[0]==entry[0]]\n",
    "        edit_distance = [nltk.edit_distance(entry, x, transpositions=True) for x in entry_spellings]\n",
    "        spellings.append(entry_spellings[np.argmin(edit_distance)])\n",
    "    return spellings\n",
    "\n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 3\n",
    "----\n",
    "\n",
    "In this assignment you will explore text message data and create models to predict if a message is spam or not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>FreeMsg Hey there darling it's been 3 week's n...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Even my brother is not like to speak with me. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>As per your request 'Melle Melle (Oru Minnamin...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>WINNER!! As a valued network customer you have...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Had your mobile 11 months or more? U R entitle...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  target\n",
       "0  Go until jurong point, crazy.. Available only ...       0\n",
       "1                      Ok lar... Joking wif u oni...       0\n",
       "2  Free entry in 2 a wkly comp to win FA Cup fina...       1\n",
       "3  U dun say so early hor... U c already then say...       0\n",
       "4  Nah I don't think he goes to usf, he lives aro...       0\n",
       "5  FreeMsg Hey there darling it's been 3 week's n...       1\n",
       "6  Even my brother is not like to speak with me. ...       0\n",
       "7  As per your request 'Melle Melle (Oru Minnamin...       0\n",
       "8  WINNER!! As a valued network customer you have...       1\n",
       "9  Had your mobile 11 months or more? U R entitle...       1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "spam_data = pd.read_csv('datasets/spam.csv')\n",
    "\n",
    "spam_data['target'] = np.where(spam_data['target']=='spam',1,0)\n",
    "spam_data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(spam_data['text'], \n",
    "                                                    spam_data['target'], \n",
    "                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "What percentage of the documents in `spam_data` are spam?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_one():\n",
    "    return (len(spam_data[spam_data['target']==1])/len(spam_data))*100\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Fit the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "What is the longest token in the vocabulary?\n",
    "\n",
    "*This function should return a string.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "def answer_two():\n",
    "    vocabulary = CountVectorizer().fit(X_train).vocabulary_\n",
    "    vocabulary = [x for x in vocabulary.keys()]\n",
    "    len_vocabulary = [len(x) for x in vocabulary]\n",
    "    return vocabulary[np.argmax(len_vocabulary)]\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "Fit and transform the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "Next, fit a fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1`. Find the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "def answer_three():\n",
    "    cv = CountVectorizer().fit(X_train)\n",
    "    X_train_cv = cv.transform(X_train)\n",
    "    X_test_cv = cv.transform(X_test)\n",
    "    clf = MultinomialNB(alpha=0.1)\n",
    "    clf.fit(X_train_cv, y_train)\n",
    "    preds_test = clf.predict(X_test_cv)\n",
    "    return roc_auc_score(y_test, preds_test)\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer with default parameters.\n",
    "\n",
    "What 20 features have the smallest tf-idf and what 20 have the largest tf-idf?\n",
    "\n",
    "Put these features in a two series where each series is sorted by tf-idf value and then alphabetically by feature name. The index of the series should be the feature name, and the data should be the tf-idf.\n",
    "\n",
    "The series of 20 features with smallest tf-idfs should be sorted smallest tfidf first, the list of 20 features with largest tf-idfs should be sorted largest first. \n",
    "\n",
    "*This function should return a tuple of two series\n",
    "`(smallest tf-idfs series, largest tf-idfs series)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "def answer_four():\n",
    "    tf = TfidfVectorizer().fit(X_train)\n",
    "    feature_names = np.array(tf.get_feature_names())\n",
    "    X_train_tf = tf.transform(X_train)\n",
    "    max_tf_idfs = X_train_tf.max(0).toarray()[0]\n",
    "    sorted_tf_idxs = max_tf_idfs.argsort()\n",
    "    sorted_tf_idfs = max_tf_idfs[sorted_tf_idxs]\n",
    "    smallest_tf_idfs = pd.Series(sorted_tf_idfs[:20], index=feature_names[sorted_tf_idxs[:20]])                    \n",
    "    largest_tf_idfs = pd.Series(sorted_tf_idfs[-20:][::-1], index=feature_names[sorted_tf_idxs[-20:][::-1]])\n",
    "    return (smallest_tf_idfs, largest_tf_idfs)\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **3**.\n",
    "\n",
    "Then fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1` and compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_five():\n",
    "    tf = TfidfVectorizer(min_df=3).fit(X_train)\n",
    "    X_train_tf = tf.transform(X_train)\n",
    "    X_test_tf = tf.transform(X_test)\n",
    "    clf = MultinomialNB(alpha=0.1)\n",
    "    clf.fit(X_train_tf, y_train)\n",
    "    preds_test = clf.predict(X_test_tf)\n",
    "    return roc_auc_score(y_test, preds_test)\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What is the average length of documents (number of characters) for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average length not spam, average length spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_six():\n",
    "    len_spam = [len(x) for x in spam_data.loc[spam_data['target']==1, 'text']]\n",
    "    len_non_spam = [len(x) for x in spam_data.loc[spam_data['target']==0, 'text']]\n",
    "    return (np.mean(len_non_spam), np.mean(len_spam))\n",
    "\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "The following function has been provided to help you combine new features into the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_feature(X, feature_to_add):\n",
    "    \"\"\"\n",
    "    Returns sparse feature matrix with added feature.\n",
    "    feature_to_add can also be a list of features.\n",
    "    \"\"\"\n",
    "    from scipy.sparse import csr_matrix, hstack\n",
    "    return hstack([X, csr_matrix(feature_to_add).T], 'csr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "Fit and transform the training data X_train using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5**.\n",
    "\n",
    "Using this document-term matrix and an additional feature, **the length of document (number of characters)**, fit a Support Vector Classification model with regularization `C=10000`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "def answer_seven():\n",
    "    len_docs_train = [len(x) for x in X_train]\n",
    "    len_docs_test = [len(x) for x in X_test]\n",
    "    tf = TfidfVectorizer(min_df=5).fit(X_train)\n",
    "    X_train_tf = tf.transform(X_train)\n",
    "    X_test_tf = tf.transform(X_test)\n",
    "    X_train_tf = add_feature(X_train_tf, len_docs_train)\n",
    "    X_test_tf = add_feature(X_test_tf, len_docs_test)\n",
    "    clf = SVC(C=10000)\n",
    "    clf.fit(X_train_tf, y_train)\n",
    "    preds_test = clf.predict(X_test_tf)\n",
    "    return roc_auc_score(y_test, preds_test)\n",
    "    \n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What is the average number of digits per document for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average # digits not spam, average # digits spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_eight():\n",
    "    num_dig_spam = [sum(char.isnumeric() for char in x) for x in spam_data.loc[spam_data['target']==1,'text']]\n",
    "    num_dig_non_spam = [sum(char.isnumeric() for char in x) for x in spam_data.loc[spam_data['target']==0,'text']]\n",
    "    return (np.mean(num_dig_non_spam), np.mean(num_dig_spam))\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **word n-grams from n=1 to n=3** (unigrams, bigrams, and trigrams).\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* **number of digits per document**\n",
    "\n",
    "fit a Logistic Regression model with regularization `C=100`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "def answer_nine():\n",
    "    dig_docs_train = [sum(char.isnumeric() for char in x) for x in X_train]\n",
    "    dig_docs_test = [sum(char.isnumeric() for char in x) for x in X_test]\n",
    "    tf = TfidfVectorizer(min_df=5, ngram_range=(1, 3)).fit(X_train)\n",
    "    X_train_tf = tf.transform(X_train)\n",
    "    X_test_tf = tf.transform(X_test)\n",
    "    X_train_tf = add_feature(X_train_tf, dig_docs_train)\n",
    "    X_test_tf = add_feature(X_test_tf, dig_docs_test)\n",
    "    clf = LogisticRegression(C=100)\n",
    "    clf.fit(X_train_tf, y_train)\n",
    "    preds_test = clf.predict(X_test_tf)\n",
    "    return roc_auc_score(y_test, preds_test)\n",
    "\n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "What is the average number of non-word characters (anything other than a letter, digit or underscore) per document for not spam and spam documents?\n",
    "\n",
    "*Hint: Use `\\w` and `\\W` character classes*\n",
    "\n",
    "*This function should return a tuple (average # non-word characters not spam, average # non-word characters spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_ten():\n",
    "    return (np.mean(spam_data.loc[spam_data['target']==0,'text'].str.count('\\W')), \n",
    "            np.mean(spam_data.loc[spam_data['target']==1,'text'].str.count('\\W')))\n",
    "\n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 4\n",
    "----\n",
    "### Part 1 - Document Similarity\n",
    "\n",
    "For the first part of this assignment, you will complete the functions `doc_to_synsets` and `similarity_score` which will be used by `document_path_similarity` to find the path similarity between two documents.\n",
    "\n",
    "The following functions are provided:\n",
    "* **`convert_tag:`** converts the tag given by `nltk.pos_tag` to a tag used by `wordnet.synsets`. You will need to use this function in `doc_to_synsets`.\n",
    "* **`document_path_similarity:`** computes the symmetrical path similarity between two documents by finding the synsets in each document using `doc_to_synsets`, then computing similarities using `similarity_score`.\n",
    "\n",
    "You will need to finish writing the following functions:\n",
    "* **`doc_to_synsets:`** returns a list of synsets in document. This function should first tokenize and part of speech tag the document using `nltk.word_tokenize` and `nltk.pos_tag`. Then it should find each tokens corresponding synset using `wn.synsets(token, wordnet_tag)`. The first synset match should be used. If there is no match, that token is skipped.\n",
    "* **`similarity_score:`** returns the normalized similarity score of a list of synsets (s1) onto a second list of synsets (s2). For each synset in s1, find the synset in s2 with the largest similarity value. Sum all of the largest similarity values together and normalize this value by dividing it by the number of largest similarity values found. Be careful with data types, which should be floats. Missing values should be ignored.\n",
    "\n",
    "Once `doc_to_synsets` and `similarity_score` have been completed, submit to the autograder which will run `test_document_path_similarity` to test that these functions are running correctly. \n",
    "\n",
    "*Do not modify the functions `convert_tag`, `document_path_similarity`, and `test_document_path_similarity`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import nltk\n",
    "from nltk.corpus import wordnet as wn\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def convert_tag(tag):\n",
    "    \"\"\"Convert the tag given by nltk.pos_tag to the tag used by wordnet.synsets\"\"\"\n",
    "    \n",
    "    tag_dict = {'N': 'n', 'J': 'a', 'R': 'r', 'V': 'v'}\n",
    "    try:\n",
    "        return tag_dict[tag[0]]\n",
    "    except KeyError:\n",
    "        return None\n",
    "\n",
    "\n",
    "def doc_to_synsets(doc):\n",
    "    \"\"\"\n",
    "    Returns a list of synsets in document.\n",
    "\n",
    "    Tokenizes and tags the words in the document doc.\n",
    "    Then finds the first synset for each word/tag combination.\n",
    "    If a synset is not found for that combination it is skipped.\n",
    "\n",
    "    Args:\n",
    "        doc: string to be converted\n",
    "\n",
    "    Returns:\n",
    "        list of synsets\n",
    "\n",
    "    Example:\n",
    "        doc_to_synsets('Fish are nvqjp friends.')\n",
    "        Out: [Synset('fish.n.01'), Synset('be.v.01'), Synset('friend.n.01')]\n",
    "    \"\"\"\n",
    "    \n",
    "\n",
    "    tokens = nltk.word_tokenize(doc)\n",
    "    pos = nltk.pos_tag(tokens)\n",
    "    wn_tags = [convert_tag(x[1]) for x in pos]\n",
    "    synsets_list = [wn.synsets(x, y)[0] for x, y in zip(tokens, wn_tags) if len(wn.synsets(x,y))>0]\n",
    "    \n",
    "    return synsets_list\n",
    "\n",
    "\n",
    "def similarity_score(s1, s2):\n",
    "    \"\"\"\n",
    "    Calculate the normalized similarity score of s1 onto s2\n",
    "\n",
    "    For each synset in s1, finds the synset in s2 with the largest similarity value.\n",
    "    Sum of all of the largest similarity values and normalize this value by dividing it by the\n",
    "    number of largest similarity values found.\n",
    "\n",
    "    Args:\n",
    "        s1, s2: list of synsets from doc_to_synsets\n",
    "\n",
    "    Returns:\n",
    "        normalized similarity score of s1 onto s2\n",
    "\n",
    "    Example:\n",
    "        synsets1 = doc_to_synsets('I like cats')\n",
    "        synsets2 = doc_to_synsets('I like dogs')\n",
    "        similarity_score(synsets1, synsets2)\n",
    "        Out: 0.73333333333333339\n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "    # Your Code Here\n",
    "    max_sim_vals = []\n",
    "    for syn in s1:\n",
    "        sim_val = [syn.path_similarity(x) for x in s2 if syn.path_similarity(x) is not None]\n",
    "        if sim_val:\n",
    "            max_sim_vals.append(max(sim_val))\n",
    "    return np.mean(max_sim_vals)\n",
    "\n",
    "\n",
    "def document_path_similarity(doc1, doc2):\n",
    "    \"\"\"Finds the symmetrical similarity between doc1 and doc2\"\"\"\n",
    "\n",
    "    synsets1 = doc_to_synsets(doc1)\n",
    "    synsets2 = doc_to_synsets(doc2)\n",
    "\n",
    "    return (similarity_score(synsets1, synsets2) + similarity_score(synsets2, synsets1)) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### test_document_path_similarity\n",
    "\n",
    "Use this function to check if doc_to_synsets and similarity_score are correct.\n",
    "\n",
    "*This function should return the similarity score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_document_path_similarity():\n",
    "    doc1 = 'This is a function to test document_path_similarity.'\n",
    "    doc2 = 'Use this function to see if your code in doc_to_synsets \\\n",
    "    and similarity_score is correct!'\n",
    "    return document_path_similarity(doc1, doc2)\n",
    "\n",
    "test_document_path_similarity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`paraphrases` is a DataFrame which contains the following columns: `Quality`, `D1`, and `D2`.\n",
    "\n",
    "`Quality` is an indicator variable which indicates if the two documents `D1` and `D2` are paraphrases of one another (1 for paraphrase, 0 for not paraphrase)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Quality</th>\n",
       "      <th>D1</th>\n",
       "      <th>D2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Ms Stewart, the chief executive, was not expec...</td>\n",
       "      <td>Ms Stewart, 61, its chief executive officer an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>After more than two years' detention under the...</td>\n",
       "      <td>After more than two years in detention by the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>\"It still remains to be seen whether the reven...</td>\n",
       "      <td>\"It remains to be seen whether the revenue rec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>And it's going to be a wild ride,\" said Allan ...</td>\n",
       "      <td>Now the rest is just mechanical,\" said Allan H...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>The cards are issued by Mexico's consulates to...</td>\n",
       "      <td>The card is issued by Mexico's consulates to i...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Quality                                                 D1  \\\n",
       "0        1  Ms Stewart, the chief executive, was not expec...   \n",
       "1        1  After more than two years' detention under the...   \n",
       "2        1  \"It still remains to be seen whether the reven...   \n",
       "3        0  And it's going to be a wild ride,\" said Allan ...   \n",
       "4        1  The cards are issued by Mexico's consulates to...   \n",
       "\n",
       "                                                  D2  \n",
       "0  Ms Stewart, 61, its chief executive officer an...  \n",
       "1  After more than two years in detention by the ...  \n",
       "2  \"It remains to be seen whether the revenue rec...  \n",
       "3  Now the rest is just mechanical,\" said Allan H...  \n",
       "4  The card is issued by Mexico's consulates to i...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use this dataframe for questions most_similar_docs and label_accuracy\n",
    "paraphrases = pd.read_csv('datasets/paraphrases.csv')\n",
    "paraphrases.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### most_similar_docs\n",
    "\n",
    "Using `document_path_similarity`, find the pair of documents in paraphrases which has the maximum similarity score.\n",
    "\n",
    "*This function should return a tuple `(D1, D2, similarity_score)`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def most_similar_docs():\n",
    "    similarity_scores = [document_path_similarity(x, y) for x, y in zip(paraphrases['D1'], paraphrases['D2'])] \n",
    "    return (paraphrases.loc[np.argmax(similarity_scores), 'D1'], \n",
    "            paraphrases.loc[np.argmax(similarity_scores), 'D2'],\n",
    "            np.max(similarity_scores))\n",
    "\n",
    "most_similar_docs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### label_accuracy\n",
    "\n",
    "Provide labels for the twenty pairs of documents by computing the similarity for each pair using `document_path_similarity`. Let the classifier rule be that if the score is greater than 0.75, label is paraphrase (1), else label is not paraphrase (0). Report accuracy of the classifier using scikit-learn's accuracy_score.\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def label_accuracy():\n",
    "    from sklearn.metrics import accuracy_score\n",
    "    # Your Code Here\n",
    "    paraphrases['sim'] = [document_path_similarity(x, y) for x, y in zip(paraphrases['D1'], paraphrases['D2'])]\n",
    "    paraphrases['sim'] = np.where(paraphrases['sim']>0.75, 1, 0)\n",
    "    return accuracy_score(paraphrases['Quality'], paraphrases['sim'])\n",
    "\n",
    "label_accuracy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2 - Topic Modelling\n",
    "\n",
    "For the second part of this assignment, you will use Gensim's LDA (Latent Dirichlet Allocation) model to model topics in `newsgroup_data`. You will first need to finish the code in the cell below by using gensim.models.ldamodel.LdaModel constructor to estimate LDA model parameters on the corpus, and save to the variable `ldamodel`. Extract 10 topics using `corpus` and `id_map`, and with `passes=25` and `random_state=34`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import gensim\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Load the list of documents\n",
    "with open('datasets/newsgroups', 'rb') as f:\n",
    "    newsgroup_data = pickle.load(f)\n",
    "\n",
    "# Use CountVectorizor to find three letter tokens, remove stop_words, \n",
    "# remove tokens that don't appear in at least 20 documents,\n",
    "# remove tokens that appear in more than 20% of the documents\n",
    "vect = CountVectorizer(min_df=20, max_df=0.2, stop_words='english', \n",
    "                       token_pattern='(?u)\\\\b\\\\w\\\\w\\\\w+\\\\b')\n",
    "# Fit and transform\n",
    "X = vect.fit_transform(newsgroup_data)\n",
    "\n",
    "# Convert sparse matrix to gensim corpus.\n",
    "corpus = gensim.matutils.Sparse2Corpus(X, documents_columns=False)\n",
    "\n",
    "# Mapping from word IDs to words (To be used in LdaModel's id2word parameter)\n",
    "id_map = dict((v, k) for k, v in vect.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use the gensim.models.ldamodel.LdaModel constructor to estimate \n",
    "# LDA model parameters on the corpus, and save to the variable `ldamodel`\n",
    "\n",
    "# Your code here:\n",
    "ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=10, id2word=id_map, passes=25, random_state=34)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### lda_topics\n",
    "\n",
    "Using `ldamodel`, find a list of the 10 topics and the most significant 10 words in each topic. This should be structured as a list of 10 tuples where each tuple takes on the form:\n",
    "\n",
    "`(9, '0.068*\"space\" + 0.036*\"nasa\" + 0.021*\"science\" + 0.020*\"edu\" + 0.019*\"data\" + 0.017*\"shuttle\" + 0.015*\"launch\" + 0.015*\"available\" + 0.014*\"center\" + 0.014*\"sci\"')`\n",
    "\n",
    "for example.\n",
    "\n",
    "*This function should return a list of tuples.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lda_topics():\n",
    "    return list(ldamodel.show_topics(num_topics=10, num_words=10))\n",
    "\n",
    "lda_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### topic_distribution\n",
    "\n",
    "For the new document `new_doc`, find the topic distribution. Remember to use vect.transform on the the new doc, and Sparse2Corpus to convert the sparse matrix to gensim corpus.\n",
    "\n",
    "*This function should return a list of tuples, where each tuple is `(#topic, probability)`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new_doc = [\"\\n\\nIt's my understanding that the freezing will start to occur because \\\n",
    "of the\\ngrowing distance of Pluto and Charon from the Sun, due to it's\\nelliptical orbit. \\\n",
    "It is not due to shadowing effects. \\n\\n\\nPluto can shadow Charon, and vice-versa.\\n\\nGeorge \\\n",
    "Krumins\\n-- \"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def topic_distribution(): \n",
    "    # Your Code Here\n",
    "    vect_doc = vect.transform(new_doc)\n",
    "    sparse_doc = gensim.matutils.Sparse2Corpus(vect_doc, documents_columns=False)\n",
    "    return list(ldamodel[sparse_doc])[0]\n",
    "\n",
    "topic_distribution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### topic_names\n",
    "\n",
    "From the list of the following given topics, assign topic names to the topics you found. If none of these names best matches the topics you found, create a new 1-3 word \"title\" for the topic.\n",
    "\n",
    "Topics: Health, Science, Automobiles, Politics, Government, Travel, Computers & IT, Sports, Business, Society & Lifestyle, Religion, Education.\n",
    "\n",
    "*This function should return a list of 10 strings.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def topic_names():\n",
    "    return ['Education', 'Science', 'Computers & IT', 'Religion', 'Automobiles', \n",
    "            'Sports', 'Health', 'Random', 'Business', 'Travel']\n",
    "\n",
    "topic_names()"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "LvcWI",
   "launcher_item_id": "krne9",
   "part_id": "Mkp1I"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}