07_topic_modeling_assignments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Playing with LDA:\n",
    "------------------------\n",
    "1. Take your favorite dataset from the previous demonstrations and read a sample from it in Python.\n",
    "2. Form a bag-of-words representation for each document. At first, do not use any preprocessing for your data.\n",
    "3. Now, train an LDA model with these documents. Start with very tiny dataset and a small number of topics as the training might take a long time on the virtual machine.\n",
    "\n",
    "4. Once you have the model trained, print the most common words from each topic to see whether the topics make any sense. Then start increasing your data and topic amount.\n",
    "\n",
    "5. Try preprocessing your input data by e.g. removing stop words. Do the results improve?\n",
    "\n",
    "6. Once you are happy with the topics, select a document and transform it into topic distributions. Do they make sense?\n",
    "\n",
    "7. As the last part, transform all your documents into topic distribution representations and search for the most similar documents for your example document (based on cosine similarity).\n",
    "\n",
    "Report the observation you made during these experiments. If the task seems too difficult feel free to use the code snippets provided in the lecture notes, all of the assignments should be doable by simply copying the code from the notes.\n",
    "\n",
    "2. Topic facets in Solr and Blacklight\n",
    "--------------------------------------\n",
    "\n",
    "In this task the goal is to integrate your topic modeling with the Solr search engine. For simplicity we are just going to use the most important topic for each document as a topical facet (i.e. we are hard assigning each document to a topic).\n",
    "\n",
    "Your task is as follows:\n",
    "1. Get files helper.py, pubmed_lda.pkl, pubmed_vect.pkl, pubmed2.txt.gz from /home/kahaka/ . helpers.py has some utility functions you can use for the task. .pkl files are pretrained models for bag-of-words and lda representations. pubmed2.txt.gz is a sample of biomedical abstract with 500K documents in total.\n",
    "2. Create a new Solr core for the data set with 2 fields: one text field for the actual content and one string field for the topic name\n",
    "3. Index a subset of the given pubmed documents using the provided helper functions and the pysolr library (the way you've done it during the previous weeks). You should combine the provided helper functions with the code you have from previous assignments (e.g. the one which indexes Reddit comments).\n",
    "4. Check that your Solr core now has some content\n",
    "5. Configure your Solr and Blacklight\n",
    "    1. Modify your Solrconfig.xml (follow the instructions from Blaclight lecture, modify the field names to match our current data)\n",
    "    2. Modify blacklight.yml to point to the correct core\n",
    "    3. Modify catalog_controller.rb to include correct fields (use text field for searching and topic field for facets)\n",
    "6. Play with your fancy search engine.\n",
    "\n",
    "Which parts were you able to complete?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}