05_solr_assignments_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Indexing Reddit comments into Solr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* 53 million comments available at\n",
    "* /home/avjves/reddit/reddit_comments.gz on the course server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"score_hidden\":false,\"name\":\"t1_cnas8zv\",\"link_id\":\"t3_2qyr1a\",\"body\":\"Most of us have some family members like this. *Most* of my family is like this. \",\"downs\":0,\"created_utc\":\"1420070400\",\"score\":14,\"author\":\"YoungModern\",\"distinguished\":null,\"id\":\"cnas8zv\",\"archived\":false,\"parent_id\":\"t3_2qyr1a\",\"subreddit\":\"exmormon\",\"author_flair_css_class\":null,\"author_flair_text\":null,\"gilded\":0,\"retrieved_on\":1425124282,\"ups\":14,\"controversiality\":0,\"subreddit_id\":\"t5_2r0gj\",\"edited\":false}\n",
      "{\"distinguished\":null,\"id\":\"cnas8zw\",\"archived\":false,\"author\":\"RedCoatsForever\",\"score\":3,\"created_utc\":\"1420070400\",\"downs\":0,\"body\":\"But Mill's career was way better. Bentham is like, the Joseph Smith to Mill's Brigham Young.\",\"link_id\":\"t3_2qv6c6\",\"name\":\"t1_cnas8zw\",\"score_hidden\":false,\"controversiality\":0,\"subreddit_id\":\"t5_2s4gt\",\"edited\":false,\"retrieved_on\":1425124282,\"ups\":3,\"author_flair_css_class\":\"on\",\"gilded\":0,\"author_flair_text\":\"Ontario\",\"subreddit\":\"CanadaPolitics\",\"parent_id\":\"t1_cnas2b6\"}\n",
      "\n",
      "gzip: stdout: Broken pipe\n"
     ]
    }
   ],
   "source": [
    "zcat reddit_comments.gz | head -n 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "* The format is identical to the tweets in the previous assignment\n",
    "* Make a new core to your Solr, decide what fields you want to index, modify the schema and write another script that indexes the data into your new core\n",
    "* At least fields \"body\" (text_en), \"author\" (string), \"subreddit\" (string), \"gilded\" (boolean), \"ups\" (int) and \"downs\" (int) are useful\n",
    "* If you also want the URL to the actual comment, you need to form it from the subreddit, link_id, and id values\n",
    "* <strike>https://www.reddit.com/r/\"subreddit\"/comments/\"link_id\"/\"id\"/\"id\" </strike>\n",
    "* <b> EDIT: </b>\n",
    "    * \"t3_\" actually has to be stripped from the beginning of <i>link_id</i> \n",
    "    * Assuming that the comment JSON has been loaded into a dictionary called comment\n",
    "    * ```url = \"https://www.reddit.com/r/{}/comments/{}/{}/{}\".format(\n",
    "    comment[\"subreddit\"], comment[\"link_id\"].split(\"_\")[1],comment[\"id\"], comment[\"id\"]) ```\n",
    "* The URL is a bit weird, but it is really nice to be able to see the actual context instead of just the comment\n",
    "* The created_utc (date of creation) value is a UNIX timestamp, so to set it to a format that Solr accepts, use this command:<br>\n",
    "```import datetime\n",
    "    date = datetime.datetime.fromtimestamp(int(comment[\"created_utc\"])).isoformat()\n",
    "    ```\n",
    "* Now you can index this into Solr and specify the Solr field to be \"date\" type, so Solr knows that we are dealing with an actual date instead of just some random string\n",
    "\n",
    "\n",
    "* No need to index all 53 million comments (Takes a bit too much resources and a lot of time...)\n",
    "* Only index the first, say, 100 thousand comments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A skeleton for the script to read the data in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import pysolr, gzip, json\n",
    "\n",
    "def get_comments(filename):\n",
    "    with gzip.open(filename, \"rt\") as gzip_file:\n",
    "        for line in gzip_file:\n",
    "            comment = json.loads(line)\n",
    "            doc = {}\n",
    "            ## Gather the stuff you want here\n",
    "            yield doc\n",
    "\n",
    "\n",
    "## Get comments, index them to solr... \n",
    "for doc in get_comments(\"/home/avjves/reddit/reddit_comments.gz\"):\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Querying the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* So now we have indexed our reddit comments\n",
    "* Let's play around with them\n",
    "* Reddit comments have a couple of interesting fields: \n",
    "    - ups (upvotes, show how many upvotes the comment has gotten)\n",
    "    - gilded (if some other user has gilded this comment. Gilding = buying reddit gold for the user. This is bought with real money, so usually the comments are either funny or helpful :) )\n",
    "\n",
    "Let's play with these a bit.\n",
    " \n",
    "I suggest you use the query tab from the admin panel to query the data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "Few useful tools first:\n",
    "* Sorting - In the query panel there is an input field called sort, where you can specify which field to sort your results with.  Specify the field and then asc or desc, depending if you want the results to be in ascending or descending order. <i><b>ups desc</b></i> for sorting by upvotes in descending order.\n",
    "* Range - You can query for a value in range. You can, for example, look for posts that have 50 - 100 upvotes. Specify the field and then the range inside brackets. <i><b> ups:[50 TO 100] </b></i>\n",
    "* Facets - There is a checkbox for \"facets\" in the query panel. Clicking it will allow you to specify a field to use as a facet. Faceting categorizes the data based on the field you choose. We can use this to automatically see how many of the comments in our query appear in which subreddits. Specify the facets.field to be <i> subreddit </i>. The facets are shown at the end of the resulting JSON when you query, so scroll down the page a bit to see them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can try to look for something interesting. Try to look for:\n",
    "* Which comments have the most upvotes?\n",
    "* Which gilded comments have the most upvotes?\n",
    "* Which subreddit has the most gilded comments?\n",
    "* Pick an interesting phrase or a word (A meme for example) and find similar information about that. (most upvotes, how many gilded, subreddits it appeared in the most, etc.)\n",
    "* Whatever else you can think of :) Sentiment for example, so find comments that have some sentiment word and a topic of your choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}