intro_to_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Natural Language Processing with Python\n",
    "\n",
    "## Info\n",
    "- Scott Bailey (CIDR), *scottbailey@stanford.edu*\n",
    "- Javier de la Rosa (CIDR), *versae@stanford.edu*\n",
    "- Ashley Jester (CIDR/SSDS), *ajester@stanford.edu*\n",
    "\n",
    "## What are we covering today?\n",
    "- What is NLP?\n",
    "- Options for NLP in Python\n",
    "- Tokenization\n",
    "- Part of Speech Tagging\n",
    "- Word transformations (lemmatization, pluralization)\n",
    "- Sentiment Analysis\n",
    "- Readability indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "\n",
    "By the end of the workshop, we hope you'll have a basic understanding of natural language processing, and enough familiarity with one NLP package, Textblob, to perform basic NLP tasks like tokenization and part of speech tagging. Through analyzing presidential speeches, we also hope you'll understand how these basic tasks open up a number of possibilities for textual analysis, such as readability indices. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is NLP\n",
    "\n",
    "NLP stands for Natual Language Processing and it involves a huge variety of tasks such as:\n",
    "- Automatic summarization.\n",
    "- Coreference resolution.\n",
    "- Discourse analysis.\n",
    "- Machine translation.\n",
    "- Morphological segmentation.\n",
    "- Named entity recognition.\n",
    "- Natural language understanding.\n",
    "- Part-of-speech tagging.\n",
    "- Parsing.\n",
    "- Question answering.\n",
    "- Relationship extraction.\n",
    "- Sentiment analysis.\n",
    "- Speech recognition.\n",
    "- Topic segmentation.\n",
    "- Word segmentation.\n",
    "- Word sense disambiguation.\n",
    "- Information retrieval.\n",
    "- Information extraction.\n",
    "- Speech processing.\n",
    "\n",
    "One of the key ideas is to be able to process text without reading it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP in Python\n",
    "\n",
    "Python is builtin with a very mature regular expression library, which is the building block of natural language processing. However, more advanced tasks need different libraries. Traditionally, in the Python ecosystem the Natural Language Processing Toolkit, abbreviated as `NLTK`, has been until recently the only working choice. Unfortunately, the library has not aged well, and even though it's updated to work with the newer versions of Python, it does not provide us the speed we might need to process large corpora.\n",
    "\n",
    "Another solution that appeared recently is called `spaCy`, and it is much faster since is written in a pseudo-C Python language optimized for speed called Cython.\n",
    "\n",
    "Both these libraries are complex and therefore there exist wrappers around them to simplify their APIs. The two more popular are `Textblob` for NLTK and CLiPS Parser, and `textacy` for spaCy. In this workshop we will be using Textblob since it is more well established and mature and provides with all we need to start learning some NLP basic tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Helper functions\n",
    "import requests\n",
    "from urllib.request import urlopen\n",
    "\n",
    "def get_text(url):\n",
    "    try:\n",
    "        return requests.get(url).text\n",
    "    except:\n",
    "        return urlopen(url).read().decode(\"utf8\")\n",
    "        \n",
    "def get_speech(url):\n",
    "    page = get_text(url)\n",
    "    full_text = page.split('\\n')\n",
    "    return \" \".join(full_text[2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_url = \"https://raw.githubusercontent.com/sul-cidr/python_workshops/master/data/clinton2000.txt\"\n",
    "clinton_speech = get_speech(clinton_url)\n",
    "clinton_speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob = TextBlob(clinton_speech[:446])\n",
    "clinton_blob.string == clinton_speech[:446]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "In NLP, the act of splitting text is called tokenization, and each of the individual chunks is called a token. Therefore, we can talk about word tokenization or sentence tokenization depending on what it is that we need to divide the text into."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.noun_phrases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A special way of dividing text in tuples of sequential words or letters is usualy referred to as N-Grams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.ngrams(n=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.ngrams(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Write a function `count_chars(text)` that receives `text` and returns the total number of characters ignoring spaces and punctuation marks. For example, `count_chars(\"Well, I am not 30 years old.\")` should return `20`.\n",
    "<br/>\n",
    "* **Hint**: You could count the characters in the words.*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def count_chars(text):\n",
    "    ...\n",
    "\n",
    "count_chars(\"Well, I am not 30 years old.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part of Speech Tagging\n",
    "\n",
    "Textblob also allows you to perform Part-Of-Speech tagging, a kind of grammatical chunking, out of the box. By default it uses the Penn U Treebank, but other taggers can be plugged in using NLTK classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for word, pos in clinton_blob.tags:\n",
    "    print(word, pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For what these tags mean, you might check out http://www.clips.ua.ac.be/pages/mbsp-tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.parse()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.sentences[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Sentence tree](data/tree.svg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.sentences[0].parse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from textblob import Word\n",
    "w = Word(\"octopi\")\n",
    "w.lemmatize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "w.lemma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "v = Word(\"is\")\n",
    "v.lemmatize(\"v\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for word in clinton_blob.words:\n",
    "    print(word, word.lemmatize())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for word in clinton_blob.words:\n",
    "    print(word, word.lemmatize(\"v\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for word, pos in clinton_blob.tags:\n",
    "    if pos == \"VBP\":\n",
    "        print(word, word.lemmatize(\"v\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for word in clinton_blob.words:\n",
    "    print(word, word.pluralize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.word_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.word_counts['congress']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.words.count('Mr', case_sensitive=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.noun_phrases.count('internal crisis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Let's define the lexicon of a person as the number of different words she uses to speak. Write a function `get_lexicon(text, n)` that receives `text` and `n` and returns the lemmas of nouns, verbs, and adjectives that are used at least `n` times. For example, `get_lexicon(clinton_speech, 10)` should return\n",
    "\n",
    "```\n",
    "{'A',\n",
    " 'America',\n",
    " 'New',\n",
    " 'So',\n",
    " 'Thank',\n",
    " 'Tonight',\n",
    " 'ask',\n",
    " 'be',\n",
    " 'child',\n",
    " 'do',\n",
    " 'have',\n",
    " 'help',\n",
    " 'make',\n",
    " 'more',\n",
    " 'new',\n",
    " 'people',\n",
    " 'thank',\n",
    " 'tonight',\n",
    " 'want',\n",
    " 'work',\n",
    " 'year'}\n",
    "```.\n",
    "<br/>\n",
    "* **Hint**: In Textblob, when a tag refers to nouns, verbs, or adjectives, the first letter of the tag starts with `n`, `v`, or `j`.*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_lexicon(text, n):\n",
    "    blob = TextBlob(text)\n",
    "    ...\n",
    "    \n",
    "get_lexicon(clinton_speech, 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment analysis\n",
    "\n",
    "Sentiment analysis is a basic form of classification of sentences, commonly into 2 categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_blob.sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for sentence in clinton_blob.sentences:\n",
    "    print(sentence, sentence.sentiment.polarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sad_sent = \"Life is sad.\"\n",
    "sad_blob = TextBlob(sad_sent)\n",
    "sad_blob.sentiment.polarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Textblob includes an alternate sentiment analyzer that you can use out of the box. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from textblob.sentiments import NaiveBayesAnalyzer\n",
    "blob = TextBlob(clinton_speech[:446], analyzer=NaiveBayesAnalyzer())\n",
    "for sentence in blob.sentences:\n",
    "    print(sentence, sentence.sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "para = \"Life is good. Life sucks. John hates soda. John hates nasty soda. John likes good soda. John loves soda. John loves sweet soda.\"\n",
    "sent_blob = TextBlob(para)\n",
    "for sent in sent_blob.sentences:\n",
    "    print(sent, sent.sentiment.polarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sent_blob_nb = TextBlob(para, analyzer=NaiveBayesAnalyzer())\n",
    "for sent in sent_blob_nb.sentences:\n",
    "    print(sent, sent.sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These examples used the built-in analyzers, but a Textblob analyzer can be built with a classifier object with its own methods. Some of them are very useful for model selection if you were building your own. The Textblob docs do give an example of how to build a basic sentiment classifier if you're interested."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Rather than just get the sentiment of individual sentences, we could try to calculate the average sentiment of a text by averaging the sentiment of its sentences. Write a function `avg_sentiment(text)` that receives `text` and returns the average positive sentiment as the sum of all probability of positive sentences divided by the number of sentences. For example, `avg_sentiment(para)` should return ~`0.3284`.\n",
    "<br/>\n",
    "* **Hint**: Remember to use the `NaiveBayesAnalyzer` analyzer.*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def avg_sentiment(text):\n",
    "    ...\n",
    "\n",
    "para = \"Life is good. Life sucks. John hates soda. John hates nasty soda. John likes good soda. John loves soda. John loves sweet soda.\"\n",
    "avg_sentiment(para)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Textblob also lets you simply get the sentiment of a whole text, but you'll notice that this and the average calculated from sentence sentiment are not the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sent_blob_nb.sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Readability indices\n",
    "\n",
    "Readability indices are ways of assessing how easy or complex it is to read a particular text based on the words and sentences it has. They usually output scores that correlate with grade levels.\n",
    "\n",
    "A couple of indices that are presumably easy to calculate are the Auto Readability Index (ARI) and the Coleman-Liau Index:\n",
    "\n",
    "$$\n",
    "ARI = 4.71\\frac{chars}{words}+0.5\\frac{words}{sentences}-21.43\n",
    "$$\n",
    "$$ CL = 0.0588\\frac{letters}{100 words} - 0.296\\frac{sentences}{100words} - 15.8 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def coleman_liau_index(blob):\n",
    "    chars = count_chars(blob.words)\n",
    "    words = len(blob.words)\n",
    "    sentences = len(blob.sentences)\n",
    "    return (0.0588 * letters_per_100(chars, words)) - (0.296 * sentences_per_100(sentences, words)) - 15.8\n",
    "\n",
    "def letters_per_100(chars, words):\n",
    "    return (chars / words) * 100\n",
    "    \n",
    "def sentences_per_100(sentences, words):\n",
    "    return (sentences / words) * 100\n",
    "\n",
    "def count_chars(words):\n",
    "    return sum(len(w) for w in words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "coleman_liau_index(sent_blob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Write a function `auto_readability_index(blob)` that receives a Textblob `blob` and returns the Auto Readability Index (ARI) score as defined above. For example, `auto_readability_index(sent_blob)` should return ~`0.2815`.\n",
    "<br/>\n",
    "* **Hint**: Rememer to use the `count_chars()` function we defined before.*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def auto_readability_index(blob):\n",
    "    chars = ...\n",
    "    words = ...\n",
    "    sentences = ...\n",
    "    ...\n",
    "\n",
    "auto_readability_index(sent_blob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpus\n",
    "  \n",
    "We will work with State of the Union speeches, each from their last year, for Barack Obama, George H.W. Bush, and Bill Clinton, and the recent address to Congress by Donald Trump."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clinton_url = \"https://raw.githubusercontent.com/sul-cidr/python_workshops/master/data/clinton2000.txt\"\n",
    "bush_url = \"https://raw.githubusercontent.com/sul-cidr/python_workshops/master/data/bush2008.txt\"\n",
    "obama_url = \"https://raw.githubusercontent.com/sul-cidr/python_workshops/master/data/obama2016.txt\"\n",
    "trump_url = \"https://raw.githubusercontent.com/sul-cidr/python_workshops/master/data/trump.txt\"\n",
    "clinton_speech = get_speech(clinton_url)\n",
    "bush_speech = get_speech(bush_url)\n",
    "obama_speech = get_speech(obama_url)\n",
    "trump_speech = get_speech(trump_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "speeches = {\n",
    "    \"clinton\": TextBlob(clinton_speech, analyzer=NaiveBayesAnalyzer()),\n",
    "    \"bush\": TextBlob(bush_speech, analyzer=NaiveBayesAnalyzer()),\n",
    "    \"obama\": TextBlob(obama_speech, analyzer=NaiveBayesAnalyzer()),\n",
    "    \"trump\": TextBlob(trump_speech, analyzer=NaiveBayesAnalyzer()),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get some basic data about the speeches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Name\", \"Chars\", \"Words\", \"Unique\", \"Sentences\", sep=\"\\t\")\n",
    "for speaker, speech in speeches.items():\n",
    "    print(speaker, count_chars(speech.words), len(speech.words), len(set(speech.words)), len(speech.sentences), sep=\"\\t\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate the average number of words per sentence for each speech."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Write a function `avg_sentence_length(blob)` that receives a Textblob `blob` and returns the average sentence length as the sum of all word lengths divided by the total number of sentences. For example, `avg_sentence_length(sent_blob)` should return ~`3.2857`.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def avg_sentence_length(blob):\n",
    "    ...\n",
    "\n",
    "avg_sentence_length(sent_blob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for speaker, speech in speeches.items():\n",
    "#     speech = speech.replace(\"Applause.\", \"\")\n",
    "    print(speaker, avg_sentence_length(speech))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the most used words. We are going to filter out some common stopwords first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stopwords_url = \"https://raw.githubusercontent.com/sul-cidr/python_workshops/master/data/english_stopwords.txt\"\n",
    "stopwords = get_text(stopwords_url).split(\"\\n\")\n",
    "stopwords[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def most_used_words(blob, n):\n",
    "    word_counts = sorted(blob.word_counts.items(), key=lambda p: p[1], reverse=True)\n",
    "    return list(filter(lambda p: p[0].lower() not in stopwords, word_counts))[:n]\n",
    "\n",
    "for speaker, speech in speeches.items():\n",
    "    print(speaker, most_used_words(speech, 10), \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sort of exploratory work is often the first step in figuring out how to clean a text for text analysis. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assess the lexical richness, defined as the ratio of number of unique words by the number of total words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lexical_richness(words):\n",
    "    return len(set(words)) / len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for speaker, speech in speeches.items():\n",
    "    print(speaker, lexical_richness(speech.words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about sentiment?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for speaker, speech in speeches.items():\n",
    "    print(speaker, avg_sentiment(speech.string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Readbility scores\n",
    "\n",
    "For the Automated Readability Index, you can get the appropriate grade level here: https://en.wikipedia.org/wiki/Automated_readability_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for speaker, speech in speeches.items():\n",
    "    print(speaker, \"ARI:\", auto_readability_index(speech), \"CL:\", coleman_liau_index(speech))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for speaker, speech in speeches.items():\n",
    "    speech = speech.replace(\"Applause.\", \"\")\n",
    "    print(speaker, \"ARI:\", auto_readability_index(speech), \"CL:\", coleman_liau_index(speech))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get some comparison, let's also look at some stats calculated through Textacy. You'll note several different scores here, such as the Flesh-Kincaid Grade Level and Readability Ease, the SMOG Index, and the Gunning-Fog Index. Each of these is a measure of readability, with each of them involving the number of syllables overall or number of polysyllabic words. We also see the ARI and CL scores, which use the same formulas we used. However, you might notice that the scores are different. To understand why, you have to dig into the source code for Textacy, where you'll find that it filters out punctuation in creating the word list, which affects the number of characters. It also lowercases the punctuation-filtered words before creating the set of unique words, decreasing that number as well compared to how we calculated it here. These changes affect both the ARI and CL scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{'obama': {'FK_level': 7.076928361323411, 'FK_ease': 73.1515946068819, 'CL': 8.082574134674179, 'GF': 10.361327601233576, 'ARI': 7.258372293175114}, \n",
    " 'bush': {'FK_level': 9.015548495431595, 'FK_ease': 63.533602854678094, 'CL': 10.373284975782742, 'GF': 12.31341855540742, 'ARI': 10.105743095660415}, \n",
    " 'trump': {'FK_level': 8.74771792073162, 'FK_ease': 65.47855524889772, 'CL': 9.922284358447495, 'GF': 11.973927886256654, 'ARI': 9.750467143001387}, \n",
    " 'clinton': {'FK_level': 8.3507883263192, 'FK_ease': 68.20265979605051, 'CL': 9.236949903852384, 'GF': 11.56165222833711, 'ARI': 9.172024279702171}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Write a function `stats(url)` that receives a `url` from a plain text version of a book in Project Gutenberg and returns the a dictionary with statistics (Auto Readability Index, Coleman-Lieu Index, lexical richness, average sentence length in words, average sentiment, number of characters, number of words, number of unique words, number of sentences, and 10 most used words) of the text contained in the URL. For example, `stats(\"http://www.gutenberg.org/cache/epub/345/pg345.txt\")` should return `{'ari': 7.051237118685233,\n",
    " 'average_sentiment': 0.6216963558545169,\n",
    " 'characters': 883114,\n",
    " 'cl': 6.151579188686984,\n",
    " 'lexical_richness': 15.130625285257873,\n",
    " 'sentence_length': 19.343680709534368,\n",
    " 'sentences': 8569,\n",
    " 'top_words': ['said',\n",
    "  'could',\n",
    "  'one',\n",
    "  'us',\n",
    "  'must',\n",
    "  'would',\n",
    "  'may',\n",
    "  'shall',\n",
    "  'see',\n",
    "  'know'],\n",
    " 'unique_words': 10955,\n",
    " 'words': 165756}`.\n",
    "<br/>\n",
    "* **Hint**: Rememer to use the `get_text()` function. Be careful with what parameters to pass in to each function.*\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def stats(url):\n",
    "    text = get_text(url)\n",
    "    blob = TextBlob(text)\n",
    "    return {\n",
    "        \"ari\": ...,\n",
    "        \"cl\": ...,\n",
    "        \"lexical_richness\": ...,\n",
    "        \"sentence_length\": ...,\n",
    "        \"average_sentiment\": ...,\n",
    "        \"characters\": ...,\n",
    "        \"words\": ...,\n",
    "        \"unique_words\": ...,\n",
    "        \"sentences\": ...,\n",
    "        \"top_words\": ...,\n",
    "    }\n",
    "\n",
    "stats(\"http://www.gutenberg.org/cache/epub/345/pg345.txt\")  # Dracula"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python (text)",
   "language": "python",
   "name": "text"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}