007_web_scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/007_web_scraping.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# web scraping and `http`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "let's talk about the internet. specifically, how people communicate using the internet. super specifically, how you can get data and request that \"things\" be done for you using the internet.\n",
    "\n",
    "this is by no means an exhaustive summary. there are 7.2 bajillion articles about web scraping and they're probably all better. [especially the Mozilla ones](https://developer.mozilla.org/en-US/). also, people always freak out about how bad the W3 pages are (and they are), but they're still (in my opinion) a good beginner's resource. so dont' be afraid to try them out, but be aware that there are some problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the only thing that is special about this one is that I wrote it and you've been duped into reading it.\n",
    "\n",
    "so enjoy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `http`, `request`s and `response`s\n",
    "\n",
    "the basic problem here is that some one out there in the ol' interwebs has put up a pretty page with some data we want, and we need to go get it.\n",
    "\n",
    "we are trying to make our computer yell at their computer and send us that neat data. so we need to figure out how to yell in a way that makes their computer happy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note that we've already done some internet based communication using different protocols: we've yelled from one computer to another via `ssh`. we've used programs like `ssh`, `scp`, and `putty` to manage this yelling. we're going to do the same thing, but with a new **protocol** -- a new yelling language.\n",
    "\n",
    "our computers all yell at each other in a language called `http` -- you have perhaps heard of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `request`s\n",
    "\n",
    "[there are a ton of details about how to yell](https://developer.mozilla.org/en-US/docs/Web/HTTP), what's appropriate and what isn't, etc. but the basic idea is that you make a block of text like the below and then send it over the internet to the other server\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://mdn.mozillademos.org/files/13821/HTTP_Request_Headers2.png\" width=\"700px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "these messages basically contain 3 pieces:\n",
    "\n",
    "+ the start line (above, `POST / HTTP/1.1`)\n",
    "+ the `headers`\n",
    "    + metadata about the request. things like where it is coming from, the type of browser requesting it, what sorts of format that `request`-ing agent understands (e.g. `html`, `json`)\n",
    "    + the red, green, and blue boxes above are three flavors of headers\n",
    "+ the `body`\n",
    "    + a generic space for the `request`er to add any content (text) that might be meaningful to the server on the other end of that `request`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "right now, let's zero in on the first line (called the \"start line\"):\n",
    "\n",
    "```\n",
    "POST / HTTP/1.1\n",
    "```\n",
    "\n",
    "this has three pieces:\n",
    "\n",
    "1. the [`request` `method`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) (here, `POST`)\n",
    "1. a url to which this message should be sent (here, `/`, but it will usually be a full fledged public `url`)\n",
    "1. the `http` protocol version (here, `HTTP/1.1`)\n",
    "\n",
    "let's zero in on that first element -- the `method`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are several different types of yells (`request` `method`s) which \"you\" (the client) can make to \"them\" (the `http` server):\n",
    "\n",
    "+ `GET` -- you want to get a thing and they should give it to you\n",
    "+ `POST` -- you want to send them a thing and they should at least write you a thank you card; if they want to give you something back that's cool but it's not expected (but it's polite, so, that's basically the same as expectation, but with all sorts of passive aggressiveness tacked on)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ `HEAD` -- you want to get a thing but you would prefer it if they just told you about the thing first, like sent you a picture of it or told you how big it was. You may still want to get it at some point but you're just testing the water. it's a fun, no commitment thing -- why rush in?\n",
    "+ `PUT` -- a really pushy `POST` request, basically they tell us they have a cool painting and we say OUR PAINTING HERE IS BETTER FORGET YOURS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ `DELETE` -- actually that painting was dumb. you were right. our bad\n",
    "+ `CONNECT`, `OPTIONS`, `TRACE`, `PATCH` -- stuff your parents yell to their friends but no one really does any more outside of really formal settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "all of these have their time and place and the web wouldn't work without them.\n",
    "\n",
    "that being said, you generally only need `GET` and `POST` messages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `response`s\n",
    "\n",
    "suppose you found some server out on the internet and you decided to compose a `GET` message like the one above and send it to that computer (using its `ip` address). that server spins its wheels determining what it wants to say and then sends you back a `response` -- something *similar* to a `request`, but with simple, important differences:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://mdn.mozillademos.org/files/13823/HTTP_Response_Headers2.png\" width=\"700px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "these messages basically contain 3 pieces:\n",
    "\n",
    "+ the start line (above, `HTTP/1.1 200 OK`)\n",
    "+ the `headers`\n",
    "    + metadata about the `response`. things like where it is coming from, the format of the material it is sending back (if anything), `cookie` values (cached, computed values for future use)\n",
    "+ the `body`\n",
    "    + a generic space for the server to put the content that was `request`ed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "right now, let's zero in on the first line (called the \"start line\"):\n",
    "\n",
    "```\n",
    "HTTP/1.1 200 OK\n",
    "```\n",
    "\n",
    "1. the `http` protocol version (here, `HTTP/1.1`)\n",
    "1. the [`http` status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) (here, `200`)\n",
    "    + 100s: information responses (rare in web scraping)\n",
    "    + 200s: success! (what we want)\n",
    "    + 300s: redirection (go look *here* instead)\n",
    "    + 400s: **client** error (*you* did something wrong)\n",
    "    + 500s: **server** error (you did everything right but the server borked)\n",
    "1. a short summary of the response (here, `OK`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## what's in the `body`\n",
    "\n",
    "so, suppose you got your computer to yell at their computer and everyone's on the same page.\n",
    "\n",
    "you say HEY CAN I `GET` OR WHAT and the server sends a `response`. the body is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```\n",
    "01100101 01100001 01110011 01110100 01100101 01110010 00100000 01100101\n",
    "01100111 01100111 01110011 00100000 01100001 01110010 01100101 00100000\n",
    "01100111 01110010 01100101 01100001 01110100 00100000 01001001 00100000\n",
    "01101100 01101111 01110110 01100101 00100000 01100101 01100001 01110011\n",
    "01110100 01100101 01110010 00100000 01100101 01100111 01100111 01110011\n",
    "00101110 00100000 01110011 01100011 01110010 01100001 01101101 01100010\n",
    "01101100 01100101 01100100 00101100 00100000 01101111 01110110 01100101\n",
    "01110010 00100000 01100101 01100001 01110011 01111001 00101100 00100000\n",
    "01100100 01101111 01101110 00100111 01110100 00100000 01100101 01110110\n",
    "01100101 01101110 00100000 01100011 01100001 01110010 01100101 00101110\n",
    "00100000 01110011 01100101 01110010 01101001 01101111 01110101 01110011\n",
    "01101100 01111001 00100000 01110100 01100001 01110011 01110100 01111001\n",
    "00101110\n",
    "```\n",
    "\n",
    "<!--a href=\"http://www.rapidtables.com/convert/number/binary-to-ascii.htm\">funtimes<a/ -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "not cool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "okay so things are actually a *little* cool: while there's nothing that says the `body` will come in any particular format, there are some very common formats.\n",
    "\n",
    "it looks like many of the other computers out there speak `json` and `html`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `json` format and `javascript`\n",
    "\n",
    "one of the most common `http` yelling languages is [JavaScript Object Notation](http://json.org/), or `json` for short.\n",
    "\n",
    "this is the basic way that computers communicate to each other through `api`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if you're familiar with `python` default data types, `json` can be converted directly to lists, dictionaries, numbers, and strings. my personal favorite example is [the Magic the Gathering `json` webpage](https://mtgjson.com/) ([direct link to example](https://www.mtgjson.com/json/decks/FaerieSchemes_ELD.json)), but [there are a ton out there](https://github.com/toddmotto/public-apis)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are a million web tutorials out there. use any of them. they're all better than this.\n",
    "\n",
    "`json` is everywhere. it [has it's detractors](https://codepunk.io/xml-vs-json-why-json-sucks/), but it is essentially the *de facto* data and information transfer protocol on the web."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as I wrote above, `json` is the \"JavaScript Object Notation.\" the reason it's so ubiquitous is that [`javascript`](https://developer.mozilla.org/en-US/docs/Web/JavaScript) is ubiquitous.\n",
    "\n",
    "but **`json` is not `javascript`**. it's a *notation*, a agreed-upon format for writing out hierarchical objects as text.\n",
    "\n",
    "`javascript` is a programming language which has a ton of ~~bugs~~ features resulting from its use case as the primary web language for client-side computation (where heavy lifting is done by your laptop (via your browser) rather than the server sending you information)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "some people treat `javascript` as if it's not a chaos land of abominations and for the most part we all let them go on thinking those things, because they make us really pretty and fast webpages. Sometimes [everyone realizes that the whole enterprise is castles built on sand](https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/) and the super l33t trolls of the internet have a chuckle or two."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the main reason I bring it up is this:\n",
    "\n",
    "+ getting `json` as the body of a `response` to your `request` is trivially easy -- the best case scenario for web scraping\n",
    "+ needing `javascript` to *compute* something so that it can be scraped is much harder -- the worst case scenario for web scraping (but still doable!!)\n",
    "\n",
    "sometimes the only way to access the data you want is via `javascript`. in about 99% of all cases I've come across, I find this is something you can avoid with some digging and some advance knowledge of how web pages are built / how to use the browser inspect tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "why avoid `javascript`? because it requires breaking out of the simple `requests` paradigm and using a full-fledged `javascript` engine, and that's a big jump in complexity.\n",
    "\n",
    "**whenever possible, I try and just `GET` things.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `html` and `xml`\n",
    "\n",
    "`json` isn't the only way that computers yell at each other, though. there are other formats they yell in, perhaps you've heard of `html` ([HyperText Markup Language](https://developer.mozilla.org/en-US/docs/Web/HTML))\n",
    "\n",
    "`html` is a special amalgam of xml-like tag structures that is the language which is used by your browser to render webpages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "check out [this example page with simple `html`](http://www.columbia.edu/~fdc/sample.html).\n",
    "\n",
    "specifically, open it in your browser and then right click and select \"view source.\" you can do that on any webpage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "of course, you usually will have more luck using the \"developers tools\" than just viewing source\n",
    "\n",
    "+ in google chrome, right click > \"Inspect\"\n",
    "+ in firefox, right click > \"Inspect Element\"\n",
    "+ in explorer or edge, press F12\n",
    "+ in safari, go to www.google.com, search for, and then download something else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so suppose you sent a `GET` request to [that simple `url` from above](http://www.columbia.edu/~fdc/sample.html) and received something (I've added indentations) that looked like:\n",
    "\n",
    "```html\n",
    "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\n",
    "<html>\n",
    "  <head>\n",
    "    <!-- THIS IS A COMMENT -->\n",
    "    <title>Sample Web Page</title>\n",
    "    <META http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\n",
    "  </head>\n",
    "  <body bgcolor=\"#ffffff\" text=\"#000000\">\n",
    "    <h2>Sample Web Page</h2>\n",
    "    <!-- How to insert an image -->\n",
    "    <img src=\"picture-of-something.jpg\" alt=\"Brief description\" width=\"100%\"><br>\n",
    "    <small><i>A random photo, maximize your browser to enlarge.</i></small>\n",
    "    <p>\n",
    "        <a href=\"http://www.columbia.edu/~fdc/\">Frank da Cruz</a><br>\n",
    "        ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### EXTREMELY brief `xml` refresher\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://i1.wp.com/img.c4learn.com/2012/05/What-is-an-XML-ELement1.png\" width=\"600px\"><br><p>picture from <a href=\"https://premaseem.wordpress.com/2015/01/25/xml-elements-vs-attributes/\">this blog post</a></p></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we will learn two ways of parsing `html` for *elements* of data we wish to obtain: `css` selectors and the `xpath` query language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### searching `html` with `css`\n",
    "\n",
    "Cascading Style Sheets (*aka* [`css`](https://developer.mozilla.org/en-US/docs/Web/CSS)) is a language for describing how a browser should format and style the `html` documents it renders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`css` files are not a pre-requisite for creating a webpage (styling can be done directly in `html` docs), but websites that *don't* utilize `css`(or some more advanced variant) are few and far between."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you want an example of `css`, you can again refer to the \"Inspect\" screen / \"developers tools.\"\n",
    "\n",
    "+ The entire right panel is dedicated to `css` and `javascript` properties of the highlighted `html` document elements.\n",
    "+ the `Sources` tab of the Inspect window has a lot of files, but all the delivered `css` files are there\n",
    "+ check out the `css` file for [the Storyblocks landing page](https://www.storyblocks.com): [source is here](https://d3g7htsbjjywiv.cloudfront.net/assets/build/storyblocks~storyblocksAboutUs.e8fa4b843dab436694b1.css), but that's not that useful, is it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "why does this matter?\n",
    "\n",
    "developers are already using `css` to find and style \"things like this\" in `html` documents. we can do the same! if we want to find elements that look like\n",
    "\n",
    "```html\n",
    "<ul class=\"zachs_list\">\n",
    "```\n",
    "\n",
    "we can use `css` selector shorthand:\n",
    "\n",
    "```css\n",
    "ul.zachs_list\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`css` is a remarkably flexible way of specifying elements within a hierarchy and [this MDN tutorial](https://developer.mozilla.org/en-US/docs/Learn/CSS/Introduction_to_CSS/Selectors) is an excellent intro. there are a few rules that are most important to know:\n",
    "\n",
    "+ you don't *have* to provide the element tag\n",
    "+ `class` values are selected with the `.` character (e.g. `.class_name`)\n",
    "+ `id` values are selected with the `#` character (e.g. `#id_name`)\n",
    "+ a basic element in a selector is a combo of the `html` tag elements (e.g. `div`, `p`, `li`) and/or the above (e.g. `div.class1`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ the basic elements can be combined in several ways\n",
    "    + a comma between elements (e.g. `div, p`) will select any element of either the first or second type\n",
    "    + a space (e.g. `div p`) will select any second element underneath the first\n",
    "    + a `>` (e.g. `div > p`) will select any second element which is a direct child of the first\n",
    "    + a `+` (e.g. `div + p`) will select any second element which is an immediate sibling of the first\n",
    "    + a `~` (e.g. `div ~ p`) will select any second element which is any sibling of the first"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "+ attributes can be selected via\n",
    "    + `[attr]`: any element which has a given attribute\n",
    "    + `[attr=val]`: any element which has the given attribute with the given value\n",
    "    + several other options performing logical test on the values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### examples of each of the above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a comma between elements will select any element of either the first or second type\n",
    "\n",
    "selector:\n",
    "```css\n",
    "ul, li\n",
    "```\n",
    "\n",
    "matches:\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">  <!-- selected -->\n",
    "            <li>hello</li>       <!-- selected -->\n",
    "            <li>world</li>       <!-- selected -->\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a space between elements will select any element of the *second* type which is a descendent of the *first* at *any* depth\n",
    "\n",
    "selector\n",
    "```css\n",
    "div li\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>                        <!-- not selected, but looking under here -->\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>       <!-- selected -->\n",
    "            <li>world</li>       <!-- selected -->\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a `>` between two elements will select any element of the *second* type which is a *direct child* of an element of the *first* type\n",
    "\n",
    "selector\n",
    "```css\n",
    "ul > li\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>       <!-- selected -->\n",
    "            <li>world</li>       <!-- selected -->\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note that the `>` selector statement is pickier than the corresponding space (` `). `div li` worked above, but `div > li` does not work\n",
    "\n",
    "selector\n",
    "```css\n",
    "div > li\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>                        <!-- not selected, but looking under here -->\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>       <!-- not selected, not a direct child! -->\n",
    "            <li>world</li>       <!-- not selected, not a direct child! -->\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a `+` between two elements will select any element of the *second* type which is an *immediately-following sibling* of an element of the *first* type\n",
    "\n",
    "selector\n",
    "```css\n",
    "li + li\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>       <!-- not selected, but element of first type -->\n",
    "            <li>world</li>       <!-- selected, immediate sibling -->\n",
    "            <li>for fun</li>     <!-- selected, immediate sibling -->\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a `~` between two elements will select any element of the *second* type which is *any following sibling* of an element of the first type\n",
    "\n",
    "selector\n",
    "```css\n",
    "li ~ li\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>       <!-- not selected, not a *following* sibling -->\n",
    "            <p>...</p>\n",
    "            <li>world</li>       <!-- selected -->\n",
    "        </ul>\n",
    "        <ul class=\"zachs_second_list\">\n",
    "            <li>world</li>       <!-- not selected, not a *following* sibling -->\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "writing an attribute name within brackets (e.g. `[attr]`) or a tag element plus that string (e.g. `div[attr]`) will select any element which has that tag (if provided) and has that attribute (regardless of value)\n",
    "\n",
    "selector\n",
    "```css\n",
    "li[myattr]\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li myattr=\"myval\">hello</li>  <!-- selected -->\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and without specifying that it must be a `ul` element:\n",
    "\n",
    "selector\n",
    "```css\n",
    "/* note: no leading element! */\n",
    "[myattr]\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li myattr=\"myval\">hello</li>    <!-- selected -->\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "    <p myattr=\"myval\">example paragraph</p>  <!-- selected -->\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note that the this selector doesn't care if the *attribute* has anything for a *value*:\n",
    "\n",
    "selector\n",
    "```css\n",
    "/* note: no leading element! */\n",
    "[myattr]\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li myattr>hello</li>             <!-- selected, even without a value -->\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "    <p myattr=\"myval\"> example paragraph</p>  <!-- selected -->\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "finally, you can also specify that in addition to *having* an attribute that attribute has a specific value (`[attr=val]`). the same rules re: including or not including the tabs apply\n",
    "\n",
    "selector\n",
    "```css\n",
    "li[myattr=\"myval\"]\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li myattr=\"myval\">hello</li>  <!-- selected -->\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and an example in which the tag is not provided but only one element has a matching attribute value\n",
    "\n",
    "selector\n",
    "```css\n",
    "[myattr=\"myval\"]\n",
    "```\n",
    "\n",
    "matches\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li myattr>hello</li>             <!-- not selected, no value -->\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "    <p myattr=\"myval\"> example paragraph</p>  <!-- selected -->\n",
    "    <p myattr=\"blurb\"> example paragraph</p>  <!-- not selected, wrong value -->\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### searching `html` with `xpath` [advanced]\n",
    "\n",
    "[`xpath`](https://msdn.microsoft.com/en-us/library/ms256086) is a query language for `xml` documents. Given that `html` is effectively a refined subset of `xml`, it's a natural fit for parsing `html` documents.\n",
    "\n",
    "it's a bit older than an alternative selecting language option (`css`) (time-tested in years, but perhaps not in projects or eyeballs) and has a broader use case (all of `xml`) and can do some things `css` selectors cannot. **however**, it has a steeper learning curve and is less commonly used in the `web`-scraping community.\n",
    "\n",
    "by analogy, `xpath` is to `css` selectors what `C` or `C++` are to `python`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "that being said, it's by far my favorite of the two, and almost always my go-to. this is probably only because I learned it first, which is a terrible way to make a decision.\n",
    "\n",
    "technically, if you're learning both `xpath` and `css` selectors here, you learned them in the opposite order -- I have broken the cycle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in many respects, `xpath` is similar to describing paths of files on a linux file system. let's learn via this analogy.\n",
    "\n",
    "suppose that you know that somewhere several levels deep inside your root directory there is a folder called `important_files` and that it has a file in it where the extension is `.txt` and you want to list information about it.\n",
    "\n",
    "if you knew the *exact path*, you could use the `ls` command:\n",
    "\n",
    "```\n",
    "ls /path/to/directory/important_files/my_file.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if you *don't* know the *exact path*, though, maybe you could use some wildcard / `glob` expressions to find it. for example, you could use the find tool:\n",
    "\n",
    "```bash\n",
    "find . -ipath \"*important_files/*.txt\"\n",
    "\n",
    "# real example:\n",
    "find ~/miniconda3 -ipath \"*/pkgs/*.yaml\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this example is a little convoluted. typically you don't suddenly have an instinct that there are `yaml` files you need and they are all under the `pkgs` directory; rather, you identify the files you need and you build as generic a `glob` expression as you can to search for only those, e.g.:\n",
    "\n",
    "> all the files I want end in `.yaml`, but that's too broad -- I want only those that had `pkgs` as one of the ancestor directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`xpath` is similar. take the example below, and suppose we wanted to \"select\" the `ul` item.\n",
    "\n",
    "you know things about the hierarchy of a document (it's a list item `ul`) and the attributes (attribute `class` has value `zachs_list`)\n",
    "\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can specify the *exact path*:\n",
    "\n",
    "```\n",
    "path = /body/div/ul\n",
    "```\n",
    "\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it's also possible to discuss *relative paths*. suppose we want to describe `ul` *relative* to `div`:\n",
    "\n",
    "```\n",
    "path = ./ul\n",
    "```\n",
    "\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can also represent unknown elements along some longer hierarchy (similar to our `*` in our patterns in the file path examples above. we do this in `xpath` by writing two consecutive `/` characters. suppose we want to find any `ul` element under the `root` (top of the document) with any intermediate elements:\n",
    "\n",
    "```\n",
    "path = //ul\n",
    "```\n",
    "\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "finally, it's possible to select elements using their attributes (those items inside the `<>` characters such as `class`, `id`, *etc.*).\n",
    "\n",
    "you specify these elements using the following notation:\n",
    "\n",
    "```\n",
    "/path/to/element[@attr=\"attr_value\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for example, we could find all `ul` elements with `class` value of `zachs_list` via the path:\n",
    "\n",
    "```\n",
    "path = //ul[@class=\"zachs_list\"]\n",
    "```\n",
    "\n",
    "```html\n",
    "<body>\n",
    "    <div>\n",
    "        <ul class=\"zachs_list\">\n",
    "            <li>hello</li>\n",
    "            <li>world</li>\n",
    "        </ul>\n",
    "        <ul></ul>\n",
    "    </div>\n",
    "</body>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: create some `xpath` expressions</div>**\n",
    "\n",
    "given the source `html` from this webpage: https://anaconda.org/anaconda/repo, develop some `xpath` expressions to select the values in the \"Package Name\" column.\n",
    "\n",
    "*hint: in \"inspect\" mode, you can find an \"element selector\" button that will allow you to click on the visible element and it will isolate that piece in the source code*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "here are some reasonable options:\n",
    "\n",
    "1. absolute: `/html/body/div/div/div/div/div/div/div/div/div/table/tbody/tr/td/a/span`\n",
    "2. at any depth (over-inclusive): `//a/span`\n",
    "3. using attributes: `//span[@class=\"packageName\"]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "one final note on \"narrowing down\" the number of items which will match a given `xpath` expression: there are two very common attributes in `html` documents, and because they serve different purposes in `html` and `css`, they can be pretty useful in selecting items:\n",
    "\n",
    "1. `class`: this attribute marks that an element is \"in a class with\" other elements, so it often defines elements which are conceptually related. This is particularly common for groups of things that are specially formatted (e.g. our \"Package Name\" elements, which are all green links\n",
    "2. `id`: this is a *globally unique* attribute, so if we want to get one and only one element (*e.g.* `<table class=\"full-width\" id=\"repo-packages-table\">`) this is a guaranteed way to do so"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### a quick diversion: installing selector gadget\n",
    "\n",
    "generally speaking, it never *hurts* to check out the source code when building an `xpath` expression or a `css` selector expression. but there's a pretty great tool for cutting short, if you're interested..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: install selector gadget</div>**\n",
    "\n",
    "go find the browser extension [selector gadget](http://selectorgadget.com/) and install it.\n",
    "\n",
    "note: this is optional. if you don't like cluttering your browser I get it and you can just follow along"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "up above we built an `xpath` expression to find the \"Package Name\" element in the table at (https://anaconda.org/anaconda/repo). let's look at how selector gadget helps us here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: use selector gadget to build a `css` path and an `xpath` expression</div>**\n",
    "\n",
    "1. go to the [`anaconda` package repo page](https://anaconda.org/anaconda/repo)\n",
    "1. activate selector gadget by clicking on the extension button\n",
    "1. select the package name element\n",
    "    1. this will activate (highlight yellow) elements which match that selected item under *some* `css` selector path\n",
    "1. click on any \"wrong\" items until only the desire elements are highlighted\n",
    "1. investigate the `css` selector item at the bottom, and click the `xpath` button\n",
    "1. repeat with the Summary field items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## setup `python` to make `request`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">mini-exercise: create a web scraping `conda` environment</div>**\n",
    "\n",
    "using `conda`, let's create a new environment and install the most basic items for our first web scraping tests. use `ec2` or your laptop, wherever you prefer to access `conda` from a command line\n",
    "\n",
    "```bash\n",
    "conda create -n scrapesville python=3\n",
    "conda activate scrapesville\n",
    "conda install -y requests lxml cssselect pandas beautifulsoup4\n",
    "```\n",
    "\n",
    "that's it! we should be good to go"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "fun side story: check out [this cute little trick](https://www.theregister.co.uk/2017/09/15/pretend_python_packages_prey_on_poor_typing/), and make sure you always install `requests` and not `reqeusts`\n",
    "\n",
    "update: this has actually been addressed by `pypi` admins, and they are supposedly preventing this in the future. how? not sure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `GET` requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so far we have learned:\n",
    "\n",
    "+ communication via the `http` protocol is done by a *client* sending a `request` to a *server*, and a *server* sending a `response` back to the *client*\n",
    "+ `request`s and `response`s have a fixed format that includes a title line, headers with meta info, and a body\n",
    "+ most `response` messages contain a body that is formatted as `json`, `html`, or `xml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "one thing we *haven't* done yet is talked about how to build `request` messages. let's do that now!\n",
    "\n",
    "we will start by building a `GET` `request` message.\n",
    "\n",
    "as a reminder, a `GET` request is a request we send to a server to ask for some information, and the returned message is whatever content we asked for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `json` is almost too easy\n",
    "\n",
    "let's start with the easy case: a `GET` request that returns `json`.\n",
    "\n",
    "my go-to `json` example, as discussed above, is [mtgjson.com](https://mtgjson.com/). however, I can understand why maybe that's not the most useful resource for everyone.\n",
    "\n",
    "let's work with the (debatably) more useful [`github` software developers jobs board `json` api](https://jobs.github.com/api)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "take, for example, the below url:\n",
    "\n",
    "```\n",
    "https://jobs.github.com/positions.json?description=data&location=washington%20dc&full_time=true\n",
    "```\n",
    "[(link)](https://jobs.github.com/positions.json?description=data&location=washington%20dc&full_time=true)\n",
    "\n",
    "when working with web apis, you can thing of a URL as being a call to a function, perhaps passing that function some parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's break the url from this request down into pieces. I'll use whitespace to emphasize the pieces:\n",
    "\n",
    "```\n",
    "https://jobs.github.com/positions.json\n",
    "    ?\n",
    "    description=data\n",
    "    &location=washington%20dc\n",
    "    &full_time=true\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```\n",
    "https://jobs.github.com/positions.json  <-- the endpoint\n",
    "    ?\n",
    "    description=data\n",
    "    &location=washington%20dc\n",
    "    &full_time=true\n",
    "```\n",
    "\n",
    "the first part is the familiar looking `url`. this is known as the api \"endpoint,\" a `url` to which we can send `request`s. in an api call, this serves the name of the function call"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```\n",
    "https://jobs.github.com/positions.json\n",
    "    ?                                   <-- the start of parameters\n",
    "    description=data\n",
    "    &location=washington%20dc\n",
    "    &full_time=true\n",
    "```\n",
    "\n",
    "`?` is a common delimiter (most common? only?) indicating that what follows is a list of key-value pairs of parameters and their values. think of this is the open-parentheses of a function declaration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```\n",
    "https://jobs.github.com/positions.json\n",
    "    ?\n",
    "    description=data           <-- parameter=value\n",
    "    &location=washington%20dc  <-- &parameter=value\n",
    "    &full_time=true            <-- &parameter=value\n",
    "```\n",
    "\n",
    "each of the `key=value` strings are parameter and value pairs passed to the api function.\n",
    "\n",
    "the allowed keys are defined by the api endpoint, and a really good api will [tell you what the possibilities are](https://jobs.github.com/api) (but some won't)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```\n",
    "https://jobs.github.com/positions.json\n",
    "    ?\n",
    "    description=data           <-- parameter=value\n",
    "    &location=washington%20dc  <-- &parameter=value\n",
    "    &full_time=true            <-- &parameter=value\n",
    "```\n",
    "\n",
    "in addition to the `key=value` pairs there are `&` characters -- these symbols separate key-value pairs, like commas in an `R` or `python` function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "all together:\n",
    "\n",
    "```\n",
    "https://jobs.github.com/positions.json?description=data&location=washington%20dc&full_time=true\n",
    "```\n",
    "\n",
    "+ ask the `github` `positions.json` endpoint, and\n",
    "+ set the `description` to be `\"data\"`\n",
    "+ set the `location` to be `washington%20dc`\n",
    "    + here `%20` is a [`url`-encoding](https://en.wikipedia.org/wiki/Percent-encoding) of the space character. many characters are escaped and encoded this way\n",
    "+ set the `full_time` parameter to be `true`\n",
    "    + note: `javascript` and hence most `api`s have a lowercase `t` in the `true` boolean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in theory, you could look up the `api` endpoints, parameters, and possible values, and construct these `url` values yourself. you could build the entire `url` as a string and use `curl` or `wget` just like we have to download things in the past in this course.\n",
    "\n",
    "in practice, you will just let `python` (and the `requests` package we installed above) handle all of that for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T19:46:28.554830Z",
     "start_time": "2017-05-29T19:46:28.551277Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "help(requests.get)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T19:57:42.970066Z",
     "start_time": "2017-05-29T19:57:42.962608Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "response = requests.get(\n",
    "    # note: no ? or key-value pairs in the url -- requests will do that\n",
    "    url='https://jobs.github.com/positions.json',\n",
    "    # the key-value pairs go here, and note we don't %20 (requests does\n",
    "    # that for us, too)\n",
    "    params={\n",
    "        'description': 'data',\n",
    "        'location': 'washington dc',\n",
    "        'full_time': True,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T19:57:57.275483Z",
     "start_time": "2017-05-29T19:57:56.981163Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "remember that the 200-series [`http` status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) are *success* codes, so our request above was successful\n",
    "\n",
    "this code is useful information for more automated process, but also sometimes helps us understand what went wrong. for the most part, we like 200s and we hate 400s and 500s. the others -- supposedly -- exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-28T20:58:36.635583Z",
     "start_time": "2017-05-28T20:58:36.630604Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "very cool. now what about the content? as the [documentation](https://jobs.github.com/api) said, this `api` endpoint responded with `json`, which is accessible via the `json` method of this `response` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T19:50:04.093796Z",
     "start_time": "2017-05-29T19:50:04.086461Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "j = response.json()\n",
    "j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can dig into that as we would with any `python` dict of dicts of dicts of dicts of lists of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T19:50:56.966925Z",
     "start_time": "2017-05-29T19:50:56.959787Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "j[0]['title']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML(j[0]['description']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "not too bad!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `html` is a little harder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as a working example, let's keep focusing on that `conda` repo table: https://anaconda.org/anaconda/repo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we'll use the `requests` library to get the `html` document, and then a few different ways of parsing / searching that document\n",
    "\n",
    "+ `css` selectors with the `lxml` library\n",
    "+ `xpath` with the `lxml` library\n",
    "+ `beautifulsoup4` find functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:08:47.928238Z",
     "start_time": "2017-05-29T20:08:47.899689Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:08:48.636707Z",
     "start_time": "2017-05-29T20:08:48.058919Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "response = requests.get(url='https://anaconda.org/anaconda/repo')\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:08:48.648337Z",
     "start_time": "2017-05-29T20:08:48.639302Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(response.text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### loading `html` into `python`\n",
    "\n",
    "with `response.text` we have a string we *could* parse to find things. we know from our selector gadget work above that the package name `html` elements all have a class of `.packageName`. we could find that via normal string functions, or regular expressions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i_first_class = response.text.find('packageName')\n",
    "response.text[i_first_class - 100: i_first_class + 100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "m = re.search(\"<span class='packageName'>\", response.text)\n",
    "response.text[m.start(): m.end()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you *could* do this. you *could*. but you definitely ***should not do this***. to understand why, start with [one of the most popular stack overflow answers of all time](https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454)\n",
    "\n",
    "in short: parsing `html` is very hard! don't do it by yourself. rely on thousands of better programmers to do it for you, using an external `xml` parsing library `libxml`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in `python`, this is available as a package `lxml`\n",
    "\n",
    "the `lxml` package, and specifically `lxml.html` library, will read in arbitrary (even broken!) `html` and `xml` strings and build a `python` object that you can use as an interface to that `html` document.\n",
    "\n",
    "the object we create this way knows how to \"move around\" in the returned `html`, and can very quickly search through `html` documents using the query languages we described above (`css` selectors, `xpath`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:08:49.164770Z",
     "start_time": "2017-05-29T20:08:49.155322Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import lxml.html\n",
    "\n",
    "root = lxml.html.fromstring(response.text)\n",
    "root"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the `Element` object is nested: it has \"children\" which are, themselves, `lxml.html Element` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:09:19.186413Z",
     "start_time": "2017-05-29T20:09:19.179952Z"
    }
   },
   "outputs": [],
   "source": [
    "root.getchildren()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that this matches the general structure of all `html` documents:\n",
    "```html\n",
    "<!doctype html>\n",
    "<html lang=\"en\">\n",
    "    <head>...</head>\n",
    "    <body>...</body>\n",
    "</html>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the `head` and `body` elements themselves have children:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head, body = root.getchildren()\n",
    "body.getchildren()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and so on throughout the document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `css` selection\n",
    "\n",
    "`css` selectors are a concise and flexible query language for specifying elements in arbitrary `html` documents. we can use this query language to search \"within\" any one `lxml.html` `Element` object using the `cssselect` method (*three* `s` characters there).\n",
    "\n",
    "*note: `css` selectors are not supported out of the box by the `lxml` module, only `xpath` is. to use `css` selectors you must also install the `python` package named `cssselect`, which we did with `conda` up above*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in the selector gadget mini-exercise above, we found a simple `css` selector expression for identifying all the package name cells: `.packageName`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "packageElems = root.cssselect('.packageName')\n",
    "print('we have {} spans'.format(len(packageElems)))\n",
    "\n",
    "print('the first one is:\\n')\n",
    "pkg0 = packageElems[0]\n",
    "print(lxml.html.tostring(pkg0).decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the text contained \"within\" this first `span` element is `isort` (note: will change day-to-day). the `lxml.html` object that we named `pkg0` above has that text string available to you as an attribute `.text`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pkg0.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "using that `text` object attribute we can get all of the package names from a simple list comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:29:01.666896Z",
     "start_time": "2017-05-29T20:29:01.644927Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# list comprehension to iterate over packageElems list\n",
    "packages = [elem.text.strip() for elem in packageElems]\n",
    "packages[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "that's pretty good so far, but what if we want to actually pull down *all* the contents of that table -- not just the package name?\n",
    "\n",
    "we *could* create a separate parser for each column and zip them together\n",
    "\n",
    "there are issues with this sort of approach, though -- any idea what would be problematic with this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "first, what would happen if one of the rows was missing an element, and we zipped lists of different lengths? or a future table had multiple sub-elements in a cell in a row (e.g. [here](https://en.wikipedia.org/wiki/Help:Table#Cells_spanning_multiple_rows_or_columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "second, that approach actually throws away quite a bit of information. every cell element within *any one row* of the table is actually a *child* of a table row element (`tr`).\n",
    "\n",
    "we could use a `css` selector expression to get the 50 `tr` table row elements of this table, and then iterate through the elements (`td`) that are children of those rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's start by building an expression to get the table rows. this is helped a great deal by the fact that we have an `id` (globally unique in an `html` document!) identifying our table for us. we can then select any `tr` element underneath that table\n",
    "\n",
    "```css\n",
    "table#repo-packages-table tr\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = root.cssselect('table#repo-packages-table tr')\n",
    "print('we found {} rows'.format(len(rows)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "notice anything weird?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "that `css` selection grabbed 51 rows, but we were only supposed to have 50. our \"any descendant\" selector picked up the header as well.\n",
    "\n",
    "not cool!\n",
    "\n",
    "let's be more specific"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:29:02.281904Z",
     "start_time": "2017-05-29T20:29:02.272074Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "rows = root.cssselect('table#repo-packages-table > tbody > tr')\n",
    "print('we found {} rows'.format(len(rows)))\n",
    "\n",
    "row0 = rows[0]\n",
    "row0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "print(lxml.html.tostring(row0).decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this `tr` \"table row\" element has four `td` children -- one for each column in the table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row0.getchildren()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's unpack this row and parse out the info we want from each element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "packageTd, accessTd, summaryTd, updatedTd = row0.getchildren()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for `packageTd`, let's get the link to and name of the package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "packageTd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = packageTd.cssselect('a')[0]\n",
    "print(a)\n",
    "\n",
    "a.attrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "packagelink = packageTd.cssselect('a')[0].attrib['href']\n",
    "packagelink"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "as for the name, that's the `text` element on the `span` (we saw this up above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "packagename = packageTd.cssselect('a > span')[0].text\n",
    "packagename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the other three are much easier -- each is simply a `td` element with a text item we'd like to pull out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "access = accessTd.text\n",
    "summary = summaryTd.text\n",
    "updated = updatedTd.text\n",
    "\n",
    "access, summary, updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the extra whitespace on the `summary` is annoying -- we can `strip` that off. also, let's parse the `updated` date string into a datetime object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "access = accessTd.text\n",
    "summary = summaryTd.text.strip()\n",
    "updated = datetime.datetime.strptime(updatedTd.text, '%Y-%m-%d')\n",
    "\n",
    "access, summary, updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's unpack these four elements all in one place and put them together into a dictionary for this record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "packageTd, accessTd, summaryTd, updatedTd = row0.getchildren()\n",
    "d = {\n",
    "    'packagelink': packageTd.find('a').attrib['href'],\n",
    "    'packagename': packageTd.find('a/span').text,\n",
    "    'access': accessTd.text,\n",
    "    'summary': summaryTd.text.strip(),\n",
    "    'updated': datetime.datetime.strptime(updatedTd.text, '%Y-%m-%d'),\n",
    "}\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's turn that into a function we can apply to all of the rows in our parsed table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_row(elem):\n",
    "    packageTd, accessTd, summaryTd, updatedTd = elem.getchildren()\n",
    "    d = {'packagelink': packageTd.cssselect('a')[0].attrib['href'],\n",
    "         'packagename': packageTd.cssselect('a > span')[0].text,\n",
    "         'access': accessTd.text,\n",
    "         'summary': summaryTd.text.strip(),\n",
    "         'updated': datetime.datetime.strptime(updatedTd.text, '%Y-%m-%d'), }\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and finally, we can take our list of `tr` elements we called `rows` from wayyyyyyyy back and use a list comprehension to parse each row in that list to a dictionary of useful information. in fact, it'll be a list of dictionaries -- let's just toss it into a `pandas` dataframe while we're at it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "rows = root.cssselect('table#repo-packages-table tbody tr')\n",
    "packageinfo = [parse_row(row) for row in rows]\n",
    "dfpackage = pd.DataFrame(packageinfo)\n",
    "dfpackage.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "from end to end, then, we have the following `python` code to parse this entire table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def parse_row(elem):\n",
    "    packageTd, accessTd, summaryTd, updatedTd = elem.getchildren()\n",
    "    d = {'packagelink': packageTd.cssselect('a')[0].attrib['href'],\n",
    "         'packagename': packageTd.cssselect('a > span')[0].text,\n",
    "         'access': accessTd.text,\n",
    "         'summary': summaryTd.text.strip(),\n",
    "         'updated': datetime.datetime.strptime(updatedTd.text, '%Y-%m-%d'), }\n",
    "    return d\n",
    "\n",
    "def get_packages():\n",
    "    response = requests.get(url='https://anaconda.org/anaconda/repo')\n",
    "    root = lxml.html.fromstring(response.text)\n",
    "    return pd.DataFrame([\n",
    "        parse_row(row)\n",
    "        for row in root.cssselect('table#repo-packages-table tbody tr')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "dfpackage = get_packages()\n",
    "dfpackage.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `xpath` selection with `lxml` [advanced]\n",
    "\n",
    "anything we can do with `css` selectors we can also do with `xpath` -- plus, the `lxml` library is specifically built for `xpath` search, so we have some (very minor) shortcuts at our disposal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "just like `lxml.html` objects have a `.cssselect` method for `css` selection queries, they also have a `.xpath` method for `xpath` queries.\n",
    "\n",
    "*note: we are re-using the `lxml.html` object from above*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "by definition, all `xpath` expressions executed this way are relative to the particular object whose method we are using. this means that while we could search for `div` elements anywhere in a document with\n",
    "\n",
    "```python\n",
    "root.cssselect('div')\n",
    "```\n",
    "\n",
    "the `xpath` searching is more explicit -- `root.xpath('div')` would only find `div` elements that were **immediate children** of the `root` element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:10:31.310437Z",
     "start_time": "2017-05-29T20:10:31.303160Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# root *does* have a body element child:\n",
    "root.xpath('body')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:10:31.310437Z",
     "start_time": "2017-05-29T20:10:31.303160Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# root *does not* have a div element child:\n",
    "root.xpath('div')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's take the full `xpath` to the \"Package Name\" elements we developed in the `xpath` mini-exercise above:\n",
    "\n",
    "```\n",
    "body/div/div/div/div/div/div/div/div/div/table/tbody/tr/td/a/span\n",
    "```\n",
    "\n",
    "and find elements at this `xpath` using the `xpath` method, one element at a time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-29T20:10:37.561101Z",
     "start_time": "2017-05-29T20:10:37.553486Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# build it up one element at a time\n",
    "#root.xpath('body/div/div/div/div/div/div/div/div/div/table/tbody/tr/td/a/span')\n",
    "root.xpath('body/div/div/div/div/div/div/div/div/div/table/tbody/tr/td/a/span')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "note that `tbody` element in the path above. it turns out that modern browsers add the `tbody` element in there whether it's in the source code or not.\n",
    "\n",
    "this means that we can often end up in a situation where `tbody` is what we see in the chrome devtools, for example, but there's actually no `tbody` element in the `lxml` object.\n",
    "\n",
    "why is still mostly a mystery to me, and [I'm comforted by the fact that I'm not alone](https://stackoverflow.com/questions/27918086/why-tbody-will-be-added-automatically-by-browser)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "root.xpath('body/div/div/div/div/div/div/div/div/div/table/tbody/tr/td/a/span')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "SO USEFUL, RIGHT?\n",
    "\n",
    "this absolute path is tremendous overkill, of course. we saw in the `css` selectors section and using selector gadget that we could identify package name spans by class -- we can do that in `xpath` as well\n",
    "\n",
    "```\n",
    "//span[@class=\"packageName\"]\n",
    "│ │    └─has 'class' attr with value \"packageName\"\n",
    "│ └─is a span element\n",
    "└─is at arbitrary depth in the tree below root\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "packageElems = root.xpath('//span[@class=\"packageName\"]')\n",
    "print('we have {} spans'.format(len(packageElems)))\n",
    "\n",
    "pkg0 = packageElems[0]\n",
    "print(lxml.html.tostring(pkg0).decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "same elements with about 1/80th of the work. cool.\n",
    "\n",
    "at this point we can directly reproduce the table parsing code from above, replacing all `cssselect` statements with `xpath` statements. the selection of the table element goes from the `css` selector\n",
    "\n",
    "```css\n",
    "table#repo-packages tbody tr\n",
    "```\n",
    "\n",
    "to `xpath` expression\n",
    "\n",
    "```\n",
    ".//table[@id=\"repo-packages-table\"]/tbody/tr\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "rows = root.xpath('.//table[@id=\"repo-packages-table\"]/tbody/tr')\n",
    "print('we found {} rows'.format(len(rows)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "one additional perk of using `xpath` is that `lxml.html` objects have a special method `.find()` which will take an `xpath` expression (*not* `cssselect`) and will return the very first match -- no need to get a list and select the `[0]`th element. additionally, this will return `None` if no match is found, whereas grabbing the 0th element of an empty list will raise an `IndexError`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-05-28T21:55:39.828425Z",
     "start_time": "2017-05-28T21:55:39.821335Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# find is the same as getting the first matching element\n",
    "assert root.find('.//div') == root.xpath('.//div')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this simplifies the link and name parsing in the package element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# packagelink = packageTd.xpath('a')[0].attrib['href']\n",
    "packagelink = packageTd.find('a').attrib['href']\n",
    "packagelink"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "packagename = packageTd.find('a/span').text\n",
    "packagename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "altogether, we get identical behavior with some slight tweaks to the `parse_row` and `get_packages` functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def parse_row(elem):\n",
    "    packageTd, accessTd, summaryTd, updatedTd = elem.getchildren()\n",
    "    return {'packagelink': packageTd.find('a').attrib['href'],\n",
    "            'packagename': packageTd.find('a/span').text,\n",
    "            'access': accessTd.text,\n",
    "            'summary': summaryTd.text.strip(),\n",
    "            'updated': datetime.datetime.strptime(\n",
    "                updatedTd.text, '%Y-%m-%d'), }\n",
    "\n",
    "def get_packages():\n",
    "    response = requests.get(url='https://anaconda.org/anaconda/repo')\n",
    "    root = lxml.html.fromstring(response.text)\n",
    "    return pd.DataFrame([\n",
    "        parse_row(row)\n",
    "        for row in root.xpath('//table[@id=\"repo-packages-table\"]/tbody/tr') ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "dfpackage = get_packages()\n",
    "dfpackage.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### a great alternative: beautiful soup\n",
    "\n",
    "in the last section we covered parsing `html` documents using the base `lxml.html` library and either the `css` selector or `xpath` query language. that method is considered pretty low-level -- you will need to invest a lot of overhead in doing simple common tasks, but the basic tools are all there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "similar to how some plotting libraries (e.g. `seaborn`, `plotly.express`) exist as smarter, more convenient wrappers to lower-level plotting libraries (e.g. `matplotlib`, base `plotly`), there is a very useful wrapper library for parsing `html` called [`beautifulsoup`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/). we installed this package (called `beautifulsoup4` on `pypi` / `conda` repos) earlier\n",
    "\n",
    "consider this library as a more user-friendly way of doing `html` parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's start by creating a `soup` object from `html` text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "response = requests.get(url='https://anaconda.org/anaconda/repo')\n",
    "soup = BeautifulSoup(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(soup.a.prettify()[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "first of all, we can do all of the `css` selector statements from above exactly as before, this time via"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "trs = soup.select('table#repo-packages-table tbody tr')\n",
    "print('we found {} rows\\n'.format(len(trs)))\n",
    "tr0 = trs[0]\n",
    "print(type(tr0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the objects we are picking out are not `lxml.html` elements any more but now `bs4.element.Tag` objects. they have better (usually preferable) behaviors. for example, when we print them to the screen they will actually show us the tag contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "tr0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "compare this to prior behavior for `lxml.html` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "root"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in addition, the `soup` object has a `find_all` method which accepts a flexible set of inputs for defining elements.\n",
    "\n",
    "the first of those inputs is the `name` argument, and can be any one of what the author calls [\"filters\"](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#kinds-of-filters), and there are a few types:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a string: search for *element tags* that exactly match the provided string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "soup.find_all('span')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a regular expression: search for *element tags* that match a regular expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "soup.find_all(re.compile('spa\\w'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a list: take a list of elements and find all elements that match any of them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "len(soup.find_all(['span', 'a']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "additional options are filters which always return `True` (the constant filter, include everything) and the filter which looks at the passed in `tag` element and invokes some code (check the docs for details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "assuming `find_all` can filter down the *tags* using the above filters as the value of the parameter `name`, we can go one step further and filter based on the `attributes` by providing key-value pairs to `python`. for example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# class is special -- it's reserved in `python`, so if we are looking for\n",
    "# elements such that `class=\"packageName\"` in `html`, add the underscore\n",
    "soup.find_all('span', class_=\"packageName\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the same goes for `id` values, `href` values, etc. in particular, this allows you to **use regular expressions** when matching the values of things -- e.g. you can find all the elements which contain a certain `class_` value, or all the `href` values which are external."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `POST` requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the entire long prior section was about how to handle the response message for a `GET` request when the body of the `response` is `json` or `html`. the actual `GET` request itself was very easy to build in `python`. parsing a `json` response was also pretty easy -- almost all of the complication above came from parsing the `response` when it was `html`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`GET` isn't our only method option for sending `request`s, though -- we can also `POST`. while a `GET` request implies that the server has some data and we are simply asking to read that data, a `POST` message could mean a few things:\n",
    "\n",
    "+ we are actively creating something (`POST`ing a message to a message board, e.g.)\n",
    "+ we are asking for some data, but our parameters for that `api` won't be passed as regular `api` endpoint parameters (e.g. they are too complex, or the `api` must remain the same regardless of parameters)\n",
    "    + this is how almost all `html` forms work, e.g."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "sometimes a `POST` request is basically the same as `GET` request -- point at a `url` and yell `POST` instead of `GET` and you're done.\n",
    "\n",
    "generally speaking, though, we are *also* sending data along with our `POST` request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "far and away, the most common use case for `POST` requests is submitting data via `html` forms (e.g. login forms, uploads, submission forms).\n",
    "\n",
    "as a quick and simple example of a `POST` request, let's check out the submission api for [github gist](https://gist.github.com/RZachLamberty) (a place to host small snippets of code instead of full repos):\n",
    "\n",
    "https://developer.github.com/v3/gists/#create-a-gist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so, we have the ability to create a new gist via an automated `POST` message. let's do that!\n",
    "\n",
    "our first task is to build the data object in that gist. It looks like a dictionary, so let's just see if that works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "data = {\n",
    "    'description': 'a test post of a gist',\n",
    "    'public': True,\n",
    "    'files': {\n",
    "        'my_test_file.py': {\n",
    "            'content': \"print('hello world')\"\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's try and post this information using the `requests.post` function. we could either convert the `data` element to `json` using the `python json.dumps` function, or we could let `requests` do it for us via the `json` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "resp = requests.post(url=\"https://api.github.com/gists\",\n",
    "                     json=data)\n",
    "resp.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "dang. 400s are no good. we did receive a `json` response, for what it's worth:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "resp.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### once more, with ~~feeling~~ authentication\n",
    "\n",
    "fundamentally, a `POST` request is allowing users to add some data to some data source. it is rare for a site to allow a `POST` request without also requesting that we authenticate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a paradigm we will repeat often in the remainder of the class -- one that is ubiquitous in web communication -- is the idea of creating something called a `session`.\n",
    "\n",
    "a `session` is a persistent connection with some other computer which serves a number of purposes, but in the web client context it mainly serves to cache authentication and preferred behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we are going to add our `github` credentials to a re-usable `session` object available directly in the `requests` library (and almost any other web request library)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# check out `auth` below\n",
    "help(requests.Request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "github gists support this as well, so let's do it! enter your user name and password to `username` and `pw` (I will use `getpass` here so I can type mine and not save it forever in a `notebook`, which would *not* be cool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import getpass\n",
    "\n",
    "username = 'rzl5'\n",
    "pw = getpass.getpass('Password: ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and now, just use that user name and password directly in the call to the github `POST` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "resp = requests.post(url=\"https://api.github.com/gists\",\n",
    "                     json=data,\n",
    "                     auth=(username, pw))\n",
    "resp.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "note: you may still get a 401 if you have enabled multi-factor authentication. if so, good for you, but also, sucks for you, because I won't cover that just now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and our results (click on the link!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "j = resp.json()\n",
    "print(j['html_url'])\n",
    "j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it's great that that worked for us, but imagine if you had to program hundreds of similar but slightly different requests. what if you don't want to pass your credentials to each one of them?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "session = requests.Session()\n",
    "session.auth = (username, pw)\n",
    "\n",
    "# note *session* below, not requests.post\n",
    "resp = session.post(url=\"https://api.github.com/gists\",\n",
    "                    json=data)\n",
    "\n",
    "resp.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "j = resp.json()\n",
    "print(j['html_url'])\n",
    "j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## using browser developer tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "when doing web scraping, the modern browser developer tools can be pretty much invaluable. I'm going to do a *very* cursory summary of the most important tricks to know when using developer tools. [the full documentation](https://developers.google.com/web/tools/chrome-devtools/) is excellent.\n",
    "\n",
    "remember the main goal: finding elements that have the data we want, or a static endpoint that contains those elements in a more compact form"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">walkthrough: google chrome \"Inspect\" mode</div>**\n",
    "\n",
    "let's check out the use of the chrome developer tools on [the `anaconda` repo page](https://anaconda.org/anaconda/repo).\n",
    "\n",
    "*note to self: exit presentation mode to get list of things to cover in notes slide*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "the walkthrough covered the following topics, broken down by tab name in the \"Inspect\" mode dashboard\n",
    "\n",
    "+ elements\n",
    "    + main window\n",
    "        + this is a DOM explorer, allowing you to search for elements, expand and collapse, etc\n",
    "        + selecting a line item highlights the corresponding rendered space\n",
    "            + this is nested, so you can use this to find which element does a thing, or...\n",
    "    + find (`Ctrl + f`)\n",
    "        + search is done using `css` selectors or `xpath`\n",
    "    + bottom \"breadcrumbs\" banner\n",
    "        + this can be used to find the full `xpath` expression or the individual `css` selector expressions\n",
    "    + the \"element selector\" button (top left of the Inspect menu)\n",
    "        + can be used to point and click select\n",
    "    + element properties menu (right panel)\n",
    "        + generally, more important for identifying `css` properties\n",
    "        + let's edit a value, and click on a page to identify the source of that value\n",
    "+ console\n",
    "    + this is a `javascript` console\n",
    "        + this is an interactive shell for executing `javascript` code\n",
    "        + it has already effectively loaded *all* of the `javascript` code that was used by your browser to build and render this webpage\n",
    "    + example: `console.log('hello world')`\n",
    "    + it will display pretty frequent error messages\n",
    "+ sources\n",
    "    + many files are used to build a webpage\n",
    "        + the contents returned by the single `GET` of the first `url` will include instructions on how to acquire many other files (*e.g.* `javascript` files, `css` files, other `url`s)\n",
    "    + this menu lists them\n",
    "    + files are grouped by\n",
    "        + the domain that sent them\n",
    "        + the path in the url\n",
    "        + a common construct is to have `css`, `js`, and `img` directories to separate those files by type\n",
    "    + most files are \"minified\"\n",
    "        + white spaces is removed to make transfer faster\n",
    "        + you can have chrome add that white spacing back in\n",
    "            + click the `{}` character in the bottom-left corner of the display window\n",
    "+ network\n",
    "    + one use case: seeing what files are sent when\n",
    "    + a better use case: looking at the content individual pages we pulled in\n",
    "        + many times, a *complicated* webpage will be built out of *much simpler* and *more programmatically friendly* data (e.g. simple `json` objects)\n",
    "    + top panel: filter by file type\n",
    "        + this is a high-pass filter and not guaranteed\n",
    "        + it's by extension -- some `javascript` or `json` requests will not have `js` as an extension, so it's not perfect\n",
    "    + bottom-left panel: file name\n",
    "    + bottom-right panel: `request` details\n",
    "        + this is the real bread and butter!\n",
    "        + [`headers`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers)\n",
    "            + `headers` are packets of information that are sent along with the actual `request` or `response` (basically, meta-data about the request itself)\n",
    "                + example: the data type of the content we're requesting or responding with (`json`)\n",
    "            + this panel contains the actual structure of the actual `request` that was made\n",
    "            + subsections:\n",
    "                + general\n",
    "                    + these are the most basic details about the `request` which was actually made and the status of the response\n",
    "                + response headers\n",
    "                    + these are the `header` elements of the `response` to our browser's request\n",
    "                + request headers\n",
    "                    + the `header` our browser sent along with our original `request`\n",
    "                    + often some sub-section of these are *required* by the responding server for your content to be received\n",
    "                + query string parameters\n",
    "                    + if the `url` contained a query string (`?key1=val1&key2=val2`, etc.), it's parsed here\n",
    "                    + can be \"unparsed\" as well\n",
    "            + if you want *just this piece* of the full url building process request, you should try to replicate this page\n",
    "        + preview\n",
    "            + mostly used for rendering images\n",
    "        + response\n",
    "            + this is the rendered content of the actual response (*i.e.* usually what you're looking for)\n",
    "    + back to the file name panel\n",
    "        + right click one of the items\n",
    "        + \"copy >\"\n",
    "        + check out the various options, including `curl` statements!\n",
    "    + finally, *reload* the page to see the entire request stream re-built"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">walkthrough: a practical application</div>**\n",
    "\n",
    "let's open [the Eversource power company's outage reporting map](https://outagemap.eversource.com/external/default.html) and use some of the developer tools to find the data in the table of current outage information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "for the walkthrough, do the following\n",
    "\n",
    "+ click on the \"Customer Outages > Connecticut\" button on the left nav\n",
    "+ observe that outage stats are in a table (e.g. for city \"Avon\")\n",
    "+ open the inspect panel\n",
    "+ (re)load [the outage map](https://outagemap.eversource.com/external/default.html)\n",
    "+ explore\n",
    "    + look for the current outage information (search for \"Avon\") -- should not find it!\n",
    "    + click the \"Customer Outages > Connecticut\" button\n",
    "+ in the \"region report\" popup, select the table row elements\n",
    "    + `css` selector: `table#report-panel-conn-table tr`\n",
    "    + `xpath` selector: `//table[@id=\"report-panel-conn-table\"]//tr`\n",
    "    + copy one of these\n",
    "+ try this in python with the code in the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "resp = requests.get('https://outagemap.eversource.com/external/default.html')\n",
    "root = lxml.html.fromstring(resp.text)\n",
    "root.cssselect('table#report-panel-conn-table tr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "+ it didn't work -- why?\n",
    "+ reloading the page\n",
    "    + maybe the data is not in the *original* request, but is loaded afterward. it is *built* by the page\n",
    "    + reload the webpage and search for that same `xpath` or `css select` statement rapidly\n",
    "    + observe: they are *not found* when the page starts to render, but then are found after we click the Connecticut link\n",
    "        + this screams [ajax](https://developer.mozilla.org/en-US/docs/AJAX/Getting_Started) (Asynchronous Javascript And Xml) -- the data doesn't show up until we click a button, so it probably was sent that data\n",
    "    + maybe we can see the request that obtained those elements?\n",
    "+ Inspect > Network tab\n",
    "    + reload the page\n",
    "    + clear the previous requests\n",
    "    + check the \"preserve log\" box\n",
    "    + hit the \"Customer Outages > Connecticut\" button\n",
    "    + click around through those files\n",
    "        + try limiting to `js` first, on a whim\n",
    "            + no dice\n",
    "        + try just `xhr`\n",
    "            + awwwwwwwwww yisssssssssss\n",
    "            + note the request url: https://outagemap.eversource.com/resources/data/external/interval_generation_data/YYYY_mm_dd_HH_MM_SS/report_conn.json\n",
    "            + and the request method: `GET`\n",
    "        + try the code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# update this in class with current value\n",
    "url = ('https://outagemap.eversource.com/resources/data/external/interval_generation_data/'\n",
    "       '2020_10_04_22_29_30'\n",
    "       '/report_conn.json')\n",
    "resp = requests.get(url)\n",
    "j = resp.json()\n",
    "j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it would be reasonable here to ask how we would automate this, if I need to get that date string `YYYY_mm_dd_HH_MM_SS` every time. fortunately, if you let the inspect tools run long enough, you will see that there is a second `api` endpoint constantly getting polled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_url = 'https://outagemap.eversource.com/resources/data/external/interval_generation_data/metadata.json'\n",
    "metadata_j = requests.get(metadata_url).json()\n",
    "metadata_j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so, all together, we can repeatably scrape this table's values with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_url = 'https://outagemap.eversource.com/resources/data/external/interval_generation_data/metadata.json'\n",
    "directory = requests.get(metadata_url).json()['directory']\n",
    "\n",
    "connecticut_url = 'https://outagemap.eversource.com/resources/data/external/interval_generation_data/{}/report_conn.json'\n",
    "connecticut_url = connecticut_url.format(directory)\n",
    "\n",
    "j = requests.get(connecticut_url).json()\n",
    "df = pd.DataFrame(j['file_data']['areas'][0]['areas'][0]['areas'][0]['areas'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `javascript` engines and `selenium` webdrivers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in the previous example we experienced something pretty tricky -- an `ajax` (Asynchronous Javascript And Xml) request.\n",
    "\n",
    "the response to our initial request contained instructions on how to *keep* building the webpage, and our browser knew how to take those instructions and turn them into additional `requests` (smart little cookie)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "because of the way that the developers who put together the Eversource cite constructed their webpage, we got lucky -- we were able to find a *single, `static` url* which had all the content we need in one go. it's not always that simple, though.\n",
    "\n",
    "sometimes it's just not possible to get the information you need without running some `javascript` code (just like your smart little internet browser does)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "when this is required, we must do something more complicated than our previous simple `python` requests -- we must use a `javascript engine`s like those used in our web browsers.\n",
    "\n",
    "we can use `webdrivers` -- programs which interface with browsers on our behalf -- to do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are many packages for running `webdrivers` in multiple languages, but the most common (across languages) is [`selenium`](http://www.seleniumhq.org/)\n",
    "\n",
    "you can think of it like this: people write code in various languages (e.g. `python`) that use the `selenium webdriver` to create and then interact with a browser's `javascript engine`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### installing `selenium`, a web browser, and a webdriver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we need:\n",
    "\n",
    "1. the `python selenium` library\n",
    "2. a web browser\n",
    "3. a web driver\n",
    "\n",
    "as always, the first question: am I installing this *locally* or *on my `ec2` server*?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the answer: it depends. we can make both work.\n",
    "\n",
    "+ on both local and remote, the install steps are basically the same\n",
    "+ *if* we go with *remote*, we have an additional complication when we want to *use* selenium (more on that later)\n",
    "\n",
    "you do you. I will do the easier *local* setup in class (just to show the basics of how `selenium` works). the steps involved in doing the harder *remote* setup are in the lecture notes below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">walkthrough: installing `selenium` parts (either *local* or *remote*)</div>**\n",
    "\n",
    "1. open a terminal\n",
    "1. install `selenium`\n",
    "    1. activate some `conda` environment (e.g. the `scrapesville` one we've been using)\n",
    "    1. run `pip install selenium`\n",
    "1. download and unpack [the webdriver for your chosen browser](http://selenium-python.readthedocs.io/installation.html#drivers)\n",
    "    1. you will likely have to pick one, download (`wget`) a `zip` or `tar`ball, and `unzip` or `tar -xvzf` it\n",
    "    1. note the path where you saved it\n",
    "    1. there are some complications with `chrome` at this time, so I'd recommend using `firefox`\n",
    "1. if needed, install a browser (none is installed by default in `ec2`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in my (local) case:\n",
    "\n",
    "```bash\n",
    "conda activate scrapesville\n",
    "pip install selenium\n",
    "\n",
    "mkdir -p ~/selenium_example\n",
    "cd ~/selenium_example\n",
    "# mac:\n",
    "wget https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-macos.tar.gz\n",
    "tar -xvzf geckodriver*\n",
    "ls -alh ~/selenium_example/\n",
    "\n",
    "# finally, install firefox -- chrome is actually pretty annoying for this example\n",
    "# mac:\n",
    "# install by hand\n",
    "# linux:\n",
    "# sudo apt install firefox\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so we now should have access to the `selenium python` code library, an executable `webdriver`, and a browser that `webdriver` can drive.\n",
    "\n",
    "what can (should) we actually *do* with them?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "well, `selenium` was originally constructed not for web scraping but for UAT processes -- we could use it to automatically replicate the experience of a user coming to a webpage, clicking buttons, typing things in fields, etc.\n",
    "\n",
    "it can be used to create `test`s of a webpage's functionality, ensuring that the webpage behaves as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this means, in particular, that it can *launch an actual browser*, and control it with commands. that's pretty awesome.\n",
    "\n",
    "in turn, *that* means that we may need to be able to see an actual browser (or launching it will fail). if you're working *locally*, no problem! if you're working *remotely*, though, that's not something we've ever done via the command line yet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "#### `x11` forwarding graphical interfaces (*remote only*)\n",
    "\n",
    "the general idea here is straightforward: if *you* have some `x11`-speaking service running, and the *remote server* has some `x11`-speaking service running, you can have guis sent over your `ssh` connection from the *remote* to your local laptop.\n",
    "\n",
    "your `ubuntu` servers already have what they need -- let's install what *we* need"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "for a full walkthrough, check out [this description from Indiana University](https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?pageId=280461906). the basics are simple though:\n",
    "\n",
    "1. mac: download [`xquartz`](https://www.xquartz.org/) and run it\n",
    "2. windows: download [`xming`](http://sourceforge.net/projects/xming/) and run it\n",
    "3. linux: you should be ok out of the box. if you see complaints about not having the `xorg` package installed, install it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "now that you have an `x11` service running on your machine, you should be able to have `x11` communications with your remote server, and display guis `forward`ed from that server.\n",
    "\n",
    "for mac and linux users, create an `ssh` connection with `x11` forwarding by entering the *exact same command* you usually would to connect via `ssh` connection, but **add a `-Y` flag** (that's a *capital* `Y`):\n",
    "\n",
    "```bash\n",
    "ssh -Y -i /path/to/privatekey username@servername\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "for windows users, you will want to enable `x11` by loading your `putty` connection and navigating to the `connection > ssh > x11` side menu panel. after you've made that update, you can save if you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "`x11` forwarding is a somewhat common process, and you will do it *any* time you want to \"see\" an application running on a remote server. think of it as \"remote desktop\", but for a single application. it's also the same process for every application, so you only need to set it up and learn it one time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### where are we?\n",
    "\n",
    "at this point:\n",
    "\n",
    "1. you should have installed `selenium`\n",
    "2. you should have downloaded and unzipped a `webdriver`\n",
    "3. if you're doing all of this remotely\n",
    "    1. you should have downloaded an `x11` client\n",
    "    2. you should have made an `ssh` connection using the `-X` flag\n",
    "\n",
    "everybody there?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### an example\n",
    "\n",
    "let's try an example. suppose [our Eversource example page from before](https://www.eversource.com/clp/outage/outagemap.aspx) hadn't yielded that single convenient endpoint. we could still scrape that table if we wanted to because with `selenium` we can do *literally anything* we could do with a browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's start with a few simple `python` commands just to get things up and running.\n",
    "\n",
    "first, we will need to know or remember *where* on our filesystem we saved that `webdriver` file -- we will need to be able to point to it in order to use.\n",
    "\n",
    "I saved mine in a `~/selenium_example` directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "import os\n",
    "import pandas as pd\n",
    "import selenium.webdriver\n",
    "\n",
    "fdriver = os.path.join(os.path.expanduser('~'),\n",
    "                       'selenium_example',\n",
    "                       'geckodriver')\n",
    "\n",
    "# you may use Firefox, or Chrome, or Edge, or whatever\n",
    "driver = selenium.webdriver.Firefox(executable_path=fdriver)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "woah!\n",
    "\n",
    "first, that `driver = ...` command actually launched our browser. cool.\n",
    "\n",
    "did you notice, though, that the `python` session waited until the browser seemed done rendering? it does that. for everything we do, actually. which is good -- we are waiting until the web browser tells us it's done working on the current request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's try getting our eversource url:\n",
    "\n",
    "```python\n",
    "driver.get('https://outagemap.eversource.com/external/default.html')\n",
    "driver.current_url\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the `driver` item we have now has the ability to do all the things we might want to do -- including:\n",
    "\n",
    "1. clicking on elements\n",
    "2. typing (sending keystrokes)\n",
    "3. dragging, dropping, or highlighting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the way it does all of the above is by\n",
    "\n",
    "1. selecting an element it wishes to interact with (e.g. a text box we would type in)\n",
    "2. using `send_keys`, `click`, or other member functions on that element\n",
    "\n",
    "let's try an example -- we know we want to click on that table popup here. we need a `css` selector or `xpath` to isolate that clickable menu element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "look for that information in the `inspector` mode of firefox, chrome, etc.\n",
    "\n",
    "once you've found an `{xpath, css}`, try and select that element with `driver.find_element_by_{xpath, css_selector}()`\n",
    "\n",
    "if that works, try `click`ing that element and see what happens!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "menulink = driver.find_element_by_xpath('.//span[@id=\"menu-summary\"]')\n",
    "menulink\n",
    "menulink.click()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so let's keep repeating this process until we've done all the steps we want:\n",
    "\n",
    "1. click on the \"View Outage Report\" link\n",
    "    1. hint: there's a `driver.find_element_by_id` method\n",
    "2. select all the elements in the table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "outagelink = driver.find_element_by_id('view-summary-conn')\n",
    "outagelink.click()\n",
    "\n",
    "outageInfo = []\n",
    "for row in driver.find_elements_by_css_selector('tr.level2'):\n",
    "    town, cust, aff, pct = row.find_elements_by_tag_name('td')\n",
    "    outageInfo.append({'town_name': town.text,\n",
    "                       'customers': cust.text,\n",
    "                       'customers_affected': aff.text,\n",
    "                       'pct_customers_affected': pct.text, })\n",
    "\n",
    "df = pd.DataFrame(outageInfo)\n",
    "df.head()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "now, that's pretty inarguably cool, but in terms of doing something *practical*, I think it has some serious drawbacks:\n",
    "\n",
    "1. requires external software (a running browser program and driver)\n",
    "    1. **you don't have to do the `x11` forwarding**\n",
    "        1. it is possible to run the browser in \"headless\" mode\n",
    "        2. this makes the process considerably faster\n",
    "2. obviously slower (more overhead in rendering, e.g.)\n",
    "3. more resource (memory) intensive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## final summary\n",
    "\n",
    "so, in general, the workflow for doing directed webscraping of an identified data sources is:\n",
    "\n",
    "1. search, at least once, for a `json` api (e.g. google search \"api my.data.source\")\n",
    "    1. if you find a `json` api, use that api with the standard web requests\n",
    "2. see if the content is kept in obviously structured `html` in a static request\n",
    "    1. if it is, `GET` the `html` document with `requests`\n",
    "    2. convert the `html` to an `lxml.html` object\n",
    "    3. identify `xpath` or `css` selectors to iterate through data elements and find them on the `lxml.html` object\n",
    "3. look through the developer tools for a better sub-request\n",
    "    1. if so, repeat the steps under 2\n",
    "4. look into an automated `javascript` engine approach using `selenium`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<strong><div align=\"center\"><code>REST</code> up</div></strong>\n",
    "<div align=\"center\"><img src=\"http://www.softwaresamurai.org/wp-content/uploads/2017/12/RESTfil-API.png\" width=\"400px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [`aws` `iam`](008_iam.ipynb)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}