Topic Modeling Presidential Debates.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 12: Analyzing Presidential Debates\n",
    "\n",
    "The data from this project comes from\n",
    "the \n",
    "[UC Santa Barbara Presidency Project](https://www.presidency.ucsb.edu/documents/presidential-documents-archive-guidebook/presidential-campaigns-debates-and-endorsements-0)\n",
    "\n",
    "The data was scraped using a custom Python program. It contains most presidential and primary debates from 1960\n",
    "to 2020.\n",
    "\n",
    "The base dataframe has five columns:\n",
    "\n",
    "0. name : The name of the person speaking. There may be some inaccuracies due to inconsisitent formats on the website.\n",
    "1. text : The transcript of what the person said. \n",
    "2. debate_file : This corresponds to the individual transcript file (not included).\n",
    "3. party : This denotes the type of date. 'P' is for presidential, 'VP' is vice-presidential, 'D' is Democrat primaries,\n",
    "           and 'R' is Republican primaries.\n",
    "4. date : This is the date that the debate occurred. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from nltk.sentiment import vader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "Read in the data file and remove some descriptive rows\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debates = pd.read_csv('data/all_utterances.csv', index_col=0)\n",
    "debates.reset_index(inplace=True, drop=True)\n",
    "\n",
    "# replace missing values with empty strings\n",
    "debates = debates.fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic data information\n",
    "First, let's see who spoke the most. This includes moderators and candidates, and is not normalized by type of debate (primary or presidential."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debates.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fix dates and get debate year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debates['year'] = pd.DatetimeIndex(debates['date']).year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect the `name` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# distinct speakers\n",
    "print(debates['name'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# drop rows with names of speakers\n",
    "exclude_list = ['PARTICIPANTS', 'MODERATORS']\n",
    "debates = debates[~debates['name'].isin(exclude_list)]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Speaker-level measures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debate_counts = debates.groupby(['name']).count().sort_values(by='date', ascending=False)\n",
    "debate_counts['date'].head(15).plot.bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis\n",
    "Now, let's do some examples of sentiment analysis using the VADER algorithm.\n",
    "The VADER algorithm computes positive, negative, and neutral sentiment, and it produces\n",
    "a compound score that gives the overal polarity for each turn at talk.\n",
    "\n",
    "We will look at a few rows of data and see what we get.\n",
    "\n",
    "Compound scores greater than 0 indicate are considered positive, less than 0 are negative.\n",
    "The further the compound score is from 0, the more extreme the sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vader_analyzer = vader.SentimentIntensityAnalyzer()\n",
    "\n",
    "example1 = debates.loc[100, 'text']\n",
    "print(example1)\n",
    "vader_analyzer.polarity_scores(example1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example2 =  debates.loc[1000, 'text']\n",
    "print(example2)\n",
    "vader_analyzer.polarity_scores(example2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example3 = debates.loc[10000, 'text']\n",
    "print(example3)\n",
    "vader_analyzer.polarity_scores(example3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compute sentiment for all utterances\n",
    "Now, we will apply sentiment analysis to all of the turns-at-talk in the data.\n",
    "\n",
    "We will use the `.apply()` function in pandas. It runs the function on \n",
    "every row of the dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vader_analyzer = vader.SentimentIntensityAnalyzer()\n",
    "results = debates['text'].apply(vader_analyzer.polarity_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will format the results and add them to our original dataframe.\n",
    "We will use the `.concat()` function, which joins dataframes. `axis=1` \n",
    "joins them by columns rather than rows.\n",
    "\n",
    "See \n",
    "[this article](https://stackoverflow.com/questions/29681906/python-pandas-dataframe-from-series-of-dict) for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df = pd.DataFrame(list(results))\n",
    "\n",
    "# add the new columns\n",
    "debates = pd.concat([debates, results_df.reindex(debates.index)], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summarize sentiment data\n",
    "Now, let's compute some descriptive information for sentiment scores.\n",
    "\n",
    "First, check the distribution of sentiment scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debates['compound'].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see the mean compund sentiment score for the candidates that spoke the most."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speaker_summary = debates.groupby(['name']).agg(['mean', 'count'])\n",
    "\n",
    "display(speaker_summary['compound'][['mean', 'count']].sort_values(by='count', ascending=False).head(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.parsing.preprocessing import preprocess_string\n",
    "from gensim import corpora\n",
    "\n",
    "debates['clean_text'] = debates['text'].apply(preprocess_string)\n",
    "print(debates.loc[10, ['text', 'clean_text']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary(debates['clean_text'])\n",
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_corpus = [dictionary.doc2bow(text) for text in debates['clean_text']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting an LDA model\n",
    "\n",
    "Next, we will fit the model. One important consideration with LDA is that you must \n",
    "choose the number of topics in advance. The total number of topics allowed is not \n",
    "restricted, but too few topics and they will be too general interpret, too many topics \n",
    "and there may be considerable overlap. Later, we will see how to measure fit for different\n",
    "topic counts. This dataset is small, so model fitting is fast. Larger datasets could\n",
    "take minutes, hours, or days.\n",
    "\n",
    "For this first example, let's try with 25 topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim import models\n",
    "\n",
    "lda_25 = models.LdaModel(bow_corpus, num_topics=100, id2word=dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for topic in lda_25.show_topics(num_topics=20, ):\n",
    "    print(\"Topic\", topic[0], \":\", topic[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in bow_corpus[0:9]:\n",
    "    print(lda_25.get_document_topics(doc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "To evaluate topic model fit, we can use perplexity or coherence. These measures indicate improvement as they get \n",
    "closer to 0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Perplexity: ', lda_25.log_perplexity(bow_corpus))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add topics to data\n",
    "\n",
    "from gensim.matutils import corpus2csc\n",
    "all_topics = lda_25.get_document_topics(bow_corpus, minimum_probability=0.0)\n",
    "all_topics_csr = corpus2csc(all_topics)\n",
    "all_topics_numpy = all_topics_csr.T.toarray()\n",
    "all_topics_df = pd.DataFrame(all_topics_numpy)\n",
    "\n",
    "# make topic names easier to read\n",
    "topic_names = ['Topic ' + str(x) for x in all_topics_df.columns]\n",
    "all_topics_df.columns = topic_names\n",
    "\n",
    "\n",
    "debates = pd.concat([debates, all_topics_df], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Exercises\n",
    "1. Which candidate had the greatest percent of extreme positive turns-at-talk (compound score > 0.5)?\n",
    "2. Try building a model with 50 topics. Is it easier to interpret than one with 25 topics?\n",
    "3. How would you construct a classifier to predict a winner of an election?\n",
    "4. With text analysis, the variables that you can construct are only limited by your imagination.\n",
    "   Try creating a dictionary (a list of related words) and count the number of times each candidate\n",
    "   uses these words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "aa3e7337b8d36894b7a51014394139a0eba3728352dce4cced1005d81d5028b1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}