04_solr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Real search engine\n",
    "\n",
    "* Upgrade from our toy example to something real\n",
    "* Test an industry-grade search engine\n",
    "* Useful skill to have:\n",
    "  * Work with own corpora - powerful search for free\n",
    "  * Custom search and indexing\n",
    "  * Especially for full-text search **much** better than SQL\n",
    "  \n",
    "## Solr\n",
    "\n",
    "* Apache project\n",
    "* Pure Java (yikes)\n",
    "* Search engine - not so much the interface\n",
    "  1. Index data\n",
    "  2. Query via HTTP API\n",
    "  3. Collect results in various formats http://evex.utu.fi/solr/ENCOW/select?fl=stext&indent=on&q=+stext:spin&wt=json\n",
    "* And you can use the built-in query page too: http://evex.utu.fi/solr/#/ENCOW/query\n",
    "* And of course you can talk to Solr also in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"responseHeader\":{\n",
      "    \"status\":0,\n",
      "    \"QTime\":0,\n",
      "    \"params\":{\n",
      "      \"q\":\"+stext:spin\",\n",
      "      \"indent\":\"on\",\n",
      "      \"fl\":\"stext\",\n",
      "      \"wt\":\"json\"}},\n",
      "  \"response\":{\"numFound\":123691,\"start\":0,\"docs\":[\n",
      "      {\n",
      "        \"stext\":\"spin spin spin spin spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin , spin , spin , spin , spin ...\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin and more spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin ...\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin .\"},\n",
      "      {\n",
      "        \"stext\":\"spin spin spin .\"}]\n",
      "  }}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "solr_user=\"solr\"\n",
    "solr_password=\"xxxxxxxxxx\" #password not shown\n",
    "params={\"fl\":\"stext\", \"q\":\"+stext:spin\",\"indent\":\"on\",\"wt\":\"json\"}\n",
    "result=requests.get(\"http://evex.utu.fi/solr/ENCOW/select\",data=params,auth=(solr_user,solr_password)) \n",
    "print(result.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Installing Solr\n",
    "\n",
    "* For what it is, installing Solr is surprisingly easy\n",
    "* Googling \"solr download\"\n",
    "* http://mirror.netinch.com/pub/apache/lucene/solr/6.5.0/solr-6.5.0.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  140M  100  140M    0     0  24.8M      0  0:00:05  0:00:05 --:--:-- 27.3M\n"
     ]
    }
   ],
   "source": [
    "# Download\n",
    "curl -O http://mirror.netinch.com/pub/apache/lucene/solr/6.5.0/solr-6.5.0.tgz\n",
    "# Unpack\n",
    "tar zxf solr-6.5.0.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01_classroom_and_computers.ipynb   README.md\n",
      "01_Intro.ipynb                     \u001b[0m\u001b[01;34msolr-6.4.2\u001b[0m\n",
      "02_boolean_model_asignments.ipynb  \u001b[01;31msolr-6.4.2.tgz\u001b[0m\n",
      "02_boolean_model.ipynb             SOLR-CoNLL-U.ipynb\n",
      "04_solr.ipynb                      Untitled.ipynb\n",
      "\u001b[01;31mfiwiki-20140809-corpus.txt.gz\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# List current directory\n",
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34mbin\u001b[0m          \u001b[01;34mcontrib\u001b[0m  \u001b[01;34mdocs\u001b[0m     \u001b[01;34mlicenses\u001b[0m     LUCENE_CHANGES.txt  README.txt\n",
      "CHANGES.txt  \u001b[01;34mdist\u001b[0m     \u001b[01;34mexample\u001b[0m  LICENSE.txt  NOTICE.txt          \u001b[01;34mserver\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Change to the directory solr-6.4.2\n",
    "cd solr-6.4.2\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* ...and that's all there is to it\n",
    "* let's run it, see what happens\n",
    "* ...but first we will need a little detour..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Internet addresses and ports\n",
    "\n",
    "* Every computer on Internet has an IP address\n",
    "* E.g. 193.166.24.207\n",
    "* DNS is a service which gives these human-readable names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "; <<>> DiG 9.9.5-3ubuntu0.11-Ubuntu <<>> -x 193.166.24.207\r\n",
      ";; global options: +cmd\r\n",
      ";; Got answer:\r\n",
      ";; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14419\r\n",
      ";; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3\r\n",
      "\r\n",
      ";; OPT PSEUDOSECTION:\r\n",
      "; EDNS: version: 0, flags:; udp: 4096\r\n",
      ";; QUESTION SECTION:\r\n",
      ";207.24.166.193.in-addr.arpa.\tIN\tPTR\r\n",
      "\r\n",
      ";; ANSWER SECTION:\r\n",
      "207.24.166.193.in-addr.arpa. 1800 IN\tPTR\tvm0964.kaj.pouta.csc.fi.\r\n",
      "\r\n",
      ";; AUTHORITY SECTION:\r\n",
      "24.166.193.in-addr.arpa. 86400\tIN\tNS\tns2.funet.fi.\r\n",
      "24.166.193.in-addr.arpa. 86400\tIN\tNS\tns-secondary.funet.fi.\r\n",
      "24.166.193.in-addr.arpa. 86400\tIN\tNS\tns.funet.fi.\r\n",
      "\r\n",
      ";; ADDITIONAL SECTION:\r\n",
      "ns.funet.fi.\t\t339\tIN\tA\t128.214.46.64\r\n",
      "ns-secondary.funet.fi.\t339\tIN\tA\t128.214.248.132\r\n",
      "\r\n",
      ";; Query time: 22 msec\r\n",
      ";; SERVER: 130.232.202.139#53(130.232.202.139)\r\n",
      ";; WHEN: Thu Mar 16 22:20:43 EET 2017\r\n",
      ";; MSG SIZE  rcvd: 193\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "**** Look up IP for a given name ****\r\n",
      "\r\n",
      "\r\n",
      "; <<>> DiG 9.9.5-3ubuntu0.11-Ubuntu <<>> vm0964.kaj.pouta.csc.fi\r\n",
      ";; global options: +cmd\r\n",
      ";; Got answer:\r\n",
      ";; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26167\r\n",
      ";; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3\r\n",
      "\r\n",
      ";; OPT PSEUDOSECTION:\r\n",
      "; EDNS: version: 0, flags:; udp: 4096\r\n",
      ";; QUESTION SECTION:\r\n",
      ";vm0964.kaj.pouta.csc.fi.\tIN\tA\r\n",
      "\r\n",
      ";; ANSWER SECTION:\r\n",
      "vm0964.kaj.pouta.csc.fi. 900\tIN\tA\t193.166.24.207\r\n",
      "\r\n",
      ";; AUTHORITY SECTION:\r\n",
      "csc.fi.\t\t\t10378\tIN\tNS\tns-secondary.funet.fi.\r\n",
      "csc.fi.\t\t\t10378\tIN\tNS\tns.funet.fi.\r\n",
      "csc.fi.\t\t\t10378\tIN\tNS\tns2.funet.fi.\r\n",
      "\r\n",
      ";; ADDITIONAL SECTION:\r\n",
      "ns.funet.fi.\t\t338\tIN\tA\t128.214.46.64\r\n",
      "ns-secondary.funet.fi.\t338\tIN\tA\t128.214.248.132\r\n",
      "\r\n",
      ";; Query time: 12 msec\r\n",
      ";; SERVER: 130.232.202.139#53(130.232.202.139)\r\n",
      ";; WHEN: Thu Mar 16 22:20:44 EET 2017\r\n",
      ";; MSG SIZE  rcvd: 168\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "# echo \"**** Look up name for a given IP ****\"\n",
    "echo\n",
    "dig -x 193.166.24.207\n",
    "echo\n",
    "echo\n",
    "echo \"**** Look up IP for a given name ****\"\n",
    "echo\n",
    "dig vm0964.kaj.pouta.csc.fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* Another address is 127.0.0.1 - local interface not accessible from the outside\n",
    "* Many local services use this one (safety, among other reasons)\n",
    "\n",
    "* Every interface has one address, but many more active services/connections\n",
    "* Traffic must not get mixed up between these\n",
    "* So every interface has numbered *ports*\n",
    "* ssh to a server -> talk to it on port 22\n",
    "* Get a web page from a server -> talk to it on port 80\n",
    "* Standard services have pre-agreed port numbers in the 1-1024 range (needs admin access to serve on them)\n",
    "* Any other service can open itself a port (if free) and answer requests\n",
    "* Jupyter Notebook tends to use 8888, Solr like 8983, etc..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-03-13 11:02:47.824 INFO  (main) [   ] o.e.j.s.Server jetty-9.3.14.v20161028\n",
      "2017-03-13 11:02:48.168 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.4.2\n",
      "2017-03-13 11:02:48.168 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port 16667\n",
      "2017-03-13 11:02:48.168 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \\__ \\/ _ \\ | '_|  Install dir: /home/ginter/IR_Course/solr-6.4.2\n",
      "2017-03-13 11:02:48.189 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter |___/\\___/_|_|    Start time: 2017-03-13T11:02:48.172Z\n",
      "2017-03-13 11:02:48.209 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using system property solr.solr.home: /home/ginter/IR_Course/solr-6.4.2/server/solr\n",
      "2017-03-13 11:02:48.217 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/ginter/IR_Course/solr-6.4.2/server/solr/solr.xml\n",
      "2017-03-13 11:02:48.598 INFO  (main) [   ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true\n",
      "2017-03-13 11:02:48.659 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/ginter/IR_Course/solr-6.4.2/server/solr\n",
      "2017-03-13 11:02:48.759 INFO  (main) [   ] o.e.j.s.Server Started @1440ms\n",
      "2017-03-13 11:54:09.426 INFO  (Thread-0) [   ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1990160809\n",
      "2017-03-13 11:54:09.428 INFO  (Thread-0) [   ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node\n"
     ]
    }
   ],
   "source": [
    "# Tells solr to start in foreground and listen on port 16667\n",
    "# foreground -> occupies the command line and is killed when you exit it (please use!)\n",
    "# background -> releases the command line, doesn't get killed when you exit it -> not what we want!\n",
    "bin/solr start -f -p 16667"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "* After start, I can visit solr at http://127.0.0.1:16667"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Port-forwarding\n",
    "\n",
    "* But if our service runs on 127.0.0.1:someport on a server, how can we access it?\n",
    "* ssh -L 56789:127.0.0.1:8983 someserver\n",
    "* Forwards all traffic from my own port 56789 to localhost:8983 on someserver\n",
    "* So now I can head to http://127.0.0.1:56789 and will see there what I would see if I went to 127.0.0.1:8983 on the server\n",
    "* Makes sense?  - That is how we will access our own solrs on the vm0964.kaj.pouta.csc.fi machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Solr cores\n",
    "\n",
    "* Data stored in *cores*\n",
    "* One core - one dataset\n",
    "* (collectios not covered here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Creating a core \n",
    "\n",
    "* Cores are like projects for Solr\n",
    "* One instance of Solr can have multiple cores running simultaneously\n",
    "* Each core has its own data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "bin/solr create_core -c mytest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "http://127.0.0.1:16667/solr/#/mytest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Adding data\n",
    "\n",
    "* bin/post -p 16667 -c mytest /home/ginter\n",
    "* .... wait for a while, kill\n",
    "* curl http://127.0.0.1:16667/solr/mytest/update?commit=true\n",
    "* ...and enjoy the result..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Adding data #2\n",
    "\n",
    "* Let's try to index Wiki Quotes\n",
    "* Do so programmatically to have full control\n",
    "* And use that to learn a bit about the query language\n",
    "\n",
    "* Ready-made config files prepared for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "curl -O http://bionlp-www.utu.fi/.avjves/config_files.tgz\n",
    "tar -zxf config_files.tgz\n",
    "mv conf example_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Assuming we are now in the solr-6.4.2 directory\n",
    "bin/solr create_core -p 16667 -c ir_course -d ~/example_config#ir_course will be the name of the core, change it to whatever you want"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* and now we are ready to index data to our core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Indexing data with Solr\n",
    "\n",
    "* Using the data we extracted from wikidumps\n",
    "* Each document will have a title and a quote"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Reminder of what our data looked like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "###C:Title:Main Page\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      " \r\n",
      "\r\n",
      " \r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "###C:Start Section\r\n",
      "Wikiquote's sister projects\r\n",
      "\r\n",
      "###C:Start Section\r\n",
      "Wikiquote languages\r\n",
      "\r\n",
      "\r\n",
      "###C:Title:Albert Einstein\r\n",
      "A hundred times every day I remind myself that my inner and outer life are based on the labors of other men, living and dead, and that I must exert myself in order to give in the same measure as I have received and am still receiving...\r\n",
      "A happy man is too satisfied with the present to dwell too much on the future.\r\n",
      "Albert Einstein (14 March 1879 – 18 April 1955) was a theoretical physicist who published the special and general theories of relativity and contributed in other areas of physics. He won the Nobel Prize in physics for his explanation of the photoelectric effect.\r\n",
      "See also\r\n",
      "Albert Einstein and politics\r\n",
      "\r\n",
      "###C:Start Section\r\n",
      "Quotes\r\n",
      "Unthinking respect for authority is the greatest enemy of truth.\r\n",
      "\r\n",
      "###C:Start Section\r\n",
      "1890s\r\n",
      " Un homme heureux est trop content de la présence [du présent] pour penser beaucoup à l'avenir. \r\n",
      " A happy man is too satisfied with the present to dwell too much on the future.\r\n",
      " From \"Mes Projets d'Avenir\", a French essay written at age 17 for a school exam (18 September 1896). The Collected Papers of Albert Einstein Vol. 1 (1987) Doc. 22.\r\n",
      "\r\n",
      "###C:Start Section\r\n",
      "1900s\r\n",
      "Einstein (right) with friends Conrad Habicht and Maurice Solovine, ca. 1903\r\n",
      "The mass of a body is a measure of its energy content.\r\n",
      "Autoritätsdusel ist der größte Feind der Wahrheit.\r\n",
      "Unthinking respect for authority is the greatest enemy of truth.\r\n",
      "Letter to Jost Winteler (1901), quoted in The Private Lives of Albert Einstein by Roger Highfield and Paul Carter (1993),  p. 79. Einstein had been annoyed that Paul Drude, editor of Annalen der Physik, had dismissed out of hand some criticisms Einstein made of Drude's electron theory of metals.\r\n",
      "Lieber Habicht! / Es herrscht ein weihevolles Stillschweigen zwischen uns, so daß es mir fast wie eine sündige Entweihung vorkommt, wenn ich es jetzt durch ein wenig bedeutsames Gepappel unterbreche... / Was machen Sie denn, Sie eingefrorener Walfisch, Sie getrocknetes, eingebüchstes Stück Seele...?\r\n",
      "Dear Habicht, / Such a solemn air of silence has descended between us that I almost feel as if I am committing a sacrilege when I break it now with some inconsequential babble... / What are you up to, you frozen whale, you smoked, dried, canned piece of soul...?\r\n",
      "Opening of a letter to his friend Conrad Habicht in which he describes his four revolutionary Annus Mirabilis papers (18 or 25 May 1905)  Doc. 27\r\n",
      "E = mc²\r\n",
      "The equivalence of mass and energy was originally expressed by the equation m = L/c², which easily translates into the far more well known E = mc² in  Does the Inertia of a Body Depend Upon Its Energy Content? published in the Annalen der Physik (27 September 1905) : \"If a body gives off the energy L in the form of radiation, its mass diminishes by L/c².\"\r\n",
      "In a later statement explaining the ideas expressed by this equation, Einstein summarized: \"It followed from the special theory of relativity that mass and energy are both but different manifestations of the same thing — a somewhat unfamiliar conception for the average mind. Furthermore, the equation E = mc², in which energy is put equal to mass, multiplied by the square of the velocity of light, showed that very small amounts of mass may be converted into a very large amount of energy and vice versa. The mass and energy were in fact equivalent, according to the formula mentioned before. This was demonstrated by Cockcroft and Walton in 1932, experimentally.\" \r\n",
      "cat: write error: Broken pipe\r\n"
     ]
    }
   ],
   "source": [
    "cat extracted.txt | head -n 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example quote: {'text': 'A hundred times every day I remind myself that my inner and outer life are based on the labors of other men, living and dead, and that I must exert myself in order to give in the same measure as I have received and am still receiving...', 'source': 'Albert Einstein'}\n",
      "Indexing quotes...\n",
      "91713 quotes indexed to Solr.\n"
     ]
    }
   ],
   "source": [
    "import pysolr\n",
    "\n",
    "def get_quote(filename):\n",
    "    with open(filename, \"r\") as text_file:\n",
    "        iterator = iter(text_file) #An iterator to go through the lines\n",
    "        for line in iterator:\n",
    "            if line.startswith(\"###\"):\n",
    "                ## Title lines that have more than 2 :'s  are quotes that we don't want, e.g. C:Title:User:X\n",
    "                if \"Title\" in line and line.count(\":\") == 2: \n",
    "                    title = line.split(\":\")[-1].strip()\n",
    "                elif \"Title\" in line:\n",
    "                    # No title for the quotes we don't want, so we can easily skip them all\n",
    "                    title = \"\"\n",
    "                else:\n",
    "                    # Skipping one extra line if no Title in the ### line, e.g. C:Start Section, as the next line is not a quote\n",
    "                    next(iterator)\n",
    "            # Skipping empty lines, lines that are too short and all lines when we don't have a title set\n",
    "            elif not line or len(line) < 20 or not title: \n",
    "                continue\n",
    "            else:\n",
    "                # Yielding a quote and a title for it\n",
    "                yield (line.strip(), title)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "\n",
    "    # Connecting to solr\n",
    "    solr = pysolr.Solr(\"http://localhost:8983/solr/ir_course\")\n",
    "\n",
    "    quotes = []\n",
    "    for line, title in get_quote(\"extracted.txt\"):\n",
    "        quotes.append({\"source\" : title, \"text\": line})\n",
    "        \n",
    "    print(\"Example quote:\", quotes[0])\n",
    "    print(\"Indexing quotes...\")\n",
    "    solr.add(quotes)\n",
    "    print(\"%d quotes indexed to Solr.\" % len(quotes))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# ...now that we have some real data...\n",
    "\n",
    "* http://www.solrtutorial.com/solr-query-syntax.html\n",
    "* text:cat\n",
    "* text:dep*tion\n",
    "* text:dog*\n",
    "* text:dog?\n",
    "* text:deprivation~\n",
    "* text:bitter~0.7\n",
    "* text:\"salt and\"\n",
    "* text:\"cat dog\"~7 http://evex.utu.fi/solr/ir_course/browse?q=text%3A%22cat+dog%22~7\n",
    "* text:\"dog AND (cat OR fish))\"\n",
    "* +text:cat -text:dog\n",
    "* +text:cat +source:Hamlet\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}