02_boolean_model_asignments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise set nro 1\n",
    "\n",
    "* This is a mixture of programming and non-programming exercises\n",
    "* Maybe some of you can't do all of them, but that's fine\n",
    "* The exercise relies on the availability of the file /home/ginter/IR_Course/fiwiki-20140809-corpus.txt.gz\n",
    "* This file is there on the course server so if you do your exercises there, you don't have to do anything\n",
    "* The course server is vm0964.kaj.pouta.csc.fi so,\n",
    "* If you do your exercises on your own computer, you can do (note the dot at the end, it is a part of the command)\n",
    "  \n",
    "    scp yourusername@vm0964.kaj.pouta.csc.fi:/home/ginter/IR_Course/fiwiki-20140809-corpus.txt.gz .\n",
    "    \n",
    "# How to hand out\n",
    "\n",
    "* In ViLLE\n",
    "* Registration: http://tiny.cc/IR17\n",
    "* Course Key: IR17\n",
    "\n",
    "# Where to do\n",
    "\n",
    "* Check out [this summary](https://github.com/TurkuNLP/IR_Course/blob/master/01_classroom_and_computers.ipynb) of how to access the classroom computers and the course server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "--------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Below is gathered in one place the best IR system from the lecture. This, and the wikipedia data will be the basis of our first exercises.\n",
    "\n",
    "* It stores its data in the efficient, sparse matrix\n",
    "* Does the full tf.idf weighting\n",
    "* Can answer queries consisting of multiple terms\n",
    "* It cannot do the negations - we had to give up on that for a moment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**** 28.6950241217\n",
      "<article name=\"Lentokone\"> Lentokone on ilmassa liikkuva, ilmaa raskaampi kiinteäsiipinen ilma-alus. Lentokone pysyy ilmassa sen kantopintojen, kuten siipien aiheuttaman nostovoiman ansiosta, mutta le  (...)\n",
      "\n",
      "\n",
      "**** 24.9326224979\n",
      "<article name=\"Malév\"> Malév Hungarian Airlines oli unkarilainen lentoyhtiö. Malév liikennöi 34 maahan ja 50 kaupunkiin. Partner-yhtiöiden code-share-lennot mukaan laskien luvut nousevat 42 maahan ja   (...)\n",
      "\n",
      "\n",
      "**** 24.468528463\n",
      "<article name=\"Suomen ilmavoimat\"> Suomen ilmavoimat on yksi Suomen puolustusvoimien kolmesta puolustushaarasta. Muut kaksi ovat maavoimat ja merivoimat. Ilmavoimien perustaminen Pääartikkeli Suomen i  (...)\n",
      "\n",
      "\n",
      "**** 23.8049677357\n",
      "<article name=\"Wrightin veljekset\"> Wrightin veljeksiä, Orville Wrightia (19. elokuuta 1871 – 30. tammikuuta 1948) ja Wilbur Wrightia (16. huhtikuuta 1867 – 30. toukokuuta 1912), pidetään yleisesti en  (...)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import numpy as np\n",
    "import gzip\n",
    "\n",
    "def articles(gzipfile,max_articles=1000):\n",
    "    \"\"\"A function to yield documents from the wiki text dumps, one at a time\"\"\"\n",
    "    with gzip.open(gzipfile,\"rt\") as f:\n",
    "        article=[] #here we assemble the lines of the current article\n",
    "        for line in f:\n",
    "            line=line.strip()\n",
    "            article.append(line)\n",
    "            if line==\"</article>\": #end of article:\n",
    "                yield \" \".join(article) # yield it\n",
    "                max_articles-=1\n",
    "                if max_articles==0:\n",
    "                    break\n",
    "                article=[] # get ready for the next article\n",
    "\n",
    "def search(keywords,td_matrix,tfvec):\n",
    "    \"\"\"Carry out the search\"\"\"\n",
    "    term2row=tfvec.vocabulary_ #A more readable variable name\n",
    "    hits=np.sum(td_matrix[term2row[keyword]] for keyword in keywords) #Sum up the rows of the tf-idf weighted matrix\n",
    "    #Hits is a sparse matrix with a single row\n",
    "    #This is how you get the document indices and scores as arrays\n",
    "    document_indices=hits.nonzero()[1]\n",
    "    document_scores=hits[hits.nonzero()].A1 #A1 returns itself as flat array\n",
    "    # search done, return one array with document indices of the hits, and one array with their scores\n",
    "    return document_indices, document_scores\n",
    "\n",
    "def top_n_simple(document_indices, document_scores,top_N):\n",
    "    \"\"\"Sort the hits and return top N - simple and quite slow version for large collections\"\"\"\n",
    "    sorted_hits=sorted(zip(document_scores, document_indices),reverse=True) # Rank the results\n",
    "    #sorted_hits=list(zip(document_scores, document_indices)) # No ranking\n",
    "    return sorted_hits[:top_N] # Returns list of (score, doc_idx), (score, doc_idx), ...\n",
    "\n",
    "def index_wiki(doc_count):\n",
    "    # Read the documents into a list\n",
    "    # We need to remember them, so we can refer to them later\n",
    "    documents=list(articles(\"/home/ginter/IR_Course/fiwiki-20140809-corpus.txt.gz\",doc_count))\n",
    "    tfv_wiki=TfidfVectorizer(input=\"content\",lowercase=True,sublinear_tf=True,use_idf=True,norm=None)\n",
    "    td_matrix_wiki=tfv_wiki.fit_transform(documents)\n",
    "    td_matrix_wiki=td_matrix_wiki.T.tocsr() #Turn document-term into term-document sparse matrix\n",
    "    return td_matrix_wiki, tfv_wiki, documents #Returns the matrix, the learned vectorizer, and the documents\n",
    "\n",
    "td_matrix_wiki, tfv_wiki, documents=index_wiki(15000)\n",
    "document_indices,document_scores=search([\"ilma\",\"kone\"],td_matrix_wiki,tfv_wiki)\n",
    "top_n=top_n_simple(document_indices, document_scores,4)\n",
    "for score, doc_idx in top_n:\n",
    "    print(\"****\", score)\n",
    "    print(documents[doc_idx][:200],\" (...)\") #Print first 200 characters\n",
    "    print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "...it seems to work fine!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 1 - Read the code\n",
    "\n",
    "Read the source code above and do your best to understand what's going on. Try to put print statements in various places to make sure you know what it does. Then answer: how many unique terms you get when using the code to index the first 25,000 wiki articles, then 50,000 and then 100,000. How does the number of terms change with increasing number of documents?\n",
    "\n",
    "I'll be in the room so ask if you don't understand something."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 2 - Ranking on/off\n",
    "\n",
    "Cut'n'paste the code into a file, for example `ir.py` and test that it works with `python3 ir.py`. Then find the line in the code which ranks the results by their score and turn the ranking off. That will degrade the system into a simple keyword matching.\n",
    "\n",
    "1. When you search for [\"kissa\",\"koira\"] - is this degraded system returning documents for \"kissa AND koira\" or is it returning documents for \"kissa OR koira\"?\n",
    "2. Can you explain why the first hit you get is the page for Adolf Hitler?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 3 - Fix the system\n",
    "\n",
    "The system crashes if you query it with a word that is not in the vocabulary. Fix it. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'somewordwhichdoesnotexist'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-339c4df54cff>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"somewordwhichdoesnotexist\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtd_matrix_wiki\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtfv_wiki\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-1-161d692bf8c2>\u001b[0m in \u001b[0;36msearch\u001b[0;34m(keywords, td_matrix, tfvec)\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0;34m\"\"\"Carry out the search\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m     \u001b[0mterm2row\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtfvec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m \u001b[0;31m#A more readable variable name\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m     \u001b[0mhits\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtd_matrix\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mterm2row\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkeyword\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mkeyword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#Sum up the rows of the tf-idf weighted matrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     23\u001b[0m     \u001b[0;31m#Hits is a sparse matrix with a single row\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m     \u001b[0;31m#This is how you get the document indices and scores as arrays\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3/dist-packages/numpy/core/fromnumeric.py\u001b[0m in \u001b[0;36msum\u001b[0;34m(a, axis, dtype, out, keepdims)\u001b[0m\n\u001b[1;32m   1823\u001b[0m     \"\"\"\n\u001b[1;32m   1824\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_gentype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1825\u001b[0;31m         \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_sum_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1826\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1827\u001b[0m             \u001b[0mout\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-161d692bf8c2>\u001b[0m in \u001b[0;36m<genexpr>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0;34m\"\"\"Carry out the search\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m     \u001b[0mterm2row\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtfvec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary_\u001b[0m \u001b[0;31m#A more readable variable name\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m     \u001b[0mhits\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtd_matrix\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mterm2row\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkeyword\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mkeyword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#Sum up the rows of the tf-idf weighted matrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     23\u001b[0m     \u001b[0;31m#Hits is a sparse matrix with a single row\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m     \u001b[0;31m#This is how you get the document indices and scores as arrays\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'somewordwhichdoesnotexist'"
     ]
    }
   ],
   "source": [
    "search([\"somewordwhichdoesnotexist\"],td_matrix_wiki,tfv_wiki)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 4 - Does it scale?\n",
    "\n",
    "The \"system\" is just a handful of lines in Python. I wonder how well it scales? Let's see - here's your tasks:\n",
    "\n",
    "1. Index in it 10,000  50,000 and 100,000 articles from Finnish (or any other you like) Wikipedia\n",
    "2. Report for these three data sizes:\n",
    "   1. How long does the indexing take?\n",
    "   2. How long does it take to answer various queries of your choice - are there big differences?\n",
    "   3. How much of memory does the system roughly take (\"`top`\" command)?\n",
    "3. Based on your experience in (2) is our quick'n'dirty IR system totally sucky or not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Here is how you can measure the time spent doing something\n",
    "import time\n",
    "start=time.time() #Current time\n",
    "#... do something here\n",
    "x=sum(x for x in range(1000000)) #waste some time\n",
    "end=time.time()\n",
    "print(\"Spent\",end-start,\"seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 5 - Tf.idf and other options\n",
    "\n",
    "When creating the vectorizer in the function `index_wiki()`, you can turn on/off various options on its behavior. The full list is in the [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). Index yourself a decent number of documents from the wiki, say 25,000 and experiment with various queries and various combinations of parameters. At least IDF on/off and sub-linear TF on/off. Can you spot any differences in the results? Would you agree that the tf.idf weighting is superior? Write up your experiences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise 6 - Intersect lists\n",
    "\n",
    "On the lecture, I showed an algorithm to intersect several sorted lists of IDs. Write a function `intersect(list_of_lists)` which will compute a list containing their intersection. So `intersect([[1,3,4,6],[2,3,6,9],[1,3,4,5,6,9,27]])` would return `[3,6]`. If you want to do it fancy, then you can accept a list of iterators and return an iterator. In that way, you could chain these functions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}