Drudge_Report_Headline_Scraper_One-Click.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.ibb.co/LS4N64c/george-hayward-data-scientist.jpg\" alt=\"George John Jordan Thomas Aquinas Hayward, Optimist\" width = 200 height = 200>\n",
    "\n",
    "## Hello World!\n",
    "\n",
    "This script pulls the top headlines from the Drudge Report and saves them into a Pandas data frame. It also saves the time the script pulled them since Drudge changes a lot. \n",
    "\n",
    "Best,\n",
    "<br>George John Jordan Thomas Aquinas Hayward, Optimist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example from June 29, 2019 at 1:20am:\n",
    "<img src=\"https://raw.githubusercontent.com/ghayward/drudge_report_headline_scraper/master/Drudge_Report_Headlines_06-29-19.png\" width = 700 height = 325>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>top_headline</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>G20: Beneath smiles and handshakes, tensions s...</td>\n",
       "      <td>2019-06-29</td>\n",
       "      <td>2019-06-29 01:21:02.147771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Saudi crown prince feted at summit...</td>\n",
       "      <td>2019-06-29</td>\n",
       "      <td>2019-06-29 01:21:02.147771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FRANCE HOTTEST DAY IN HISTORY!</td>\n",
       "      <td>2019-06-29</td>\n",
       "      <td>2019-06-29 01:21:02.147771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MANURE EXPLODES IN SPAIN</td>\n",
       "      <td>2019-06-29</td>\n",
       "      <td>2019-06-29 01:21:02.147771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RED WARNING ITALY</td>\n",
       "      <td>2019-06-29</td>\n",
       "      <td>2019-06-29 01:21:02.147771</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        top_headline        date  \\\n",
       "0  G20: Beneath smiles and handshakes, tensions s...  2019-06-29   \n",
       "1              Saudi crown prince feted at summit...  2019-06-29   \n",
       "2                     FRANCE HOTTEST DAY IN HISTORY!  2019-06-29   \n",
       "3                           MANURE EXPLODES IN SPAIN  2019-06-29   \n",
       "4                                  RED WARNING ITALY  2019-06-29   \n",
       "\n",
       "                        time  \n",
       "0 2019-06-29 01:21:02.147771  \n",
       "1 2019-06-29 01:21:02.147771  \n",
       "2 2019-06-29 01:21:02.147771  \n",
       "3 2019-06-29 01:21:02.147771  \n",
       "4 2019-06-29 01:21:02.147771  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import pandas as pd\n",
    "import datetime\n",
    "from datetime import date\n",
    "\n",
    "webpage_response = requests.get('https://www.drudgereport.com/')\n",
    "webpage = webpage_response.content\n",
    "soup = BeautifulSoup(webpage, \"html.parser\")\n",
    "#the key here is that Drudge's top headlines are under the 'tt' tag\n",
    "headers_raw = soup.tt.get_text()\n",
    "#this is a sensible place to split\n",
    "split_up = headers_raw.split(\"\\n\")\n",
    "\n",
    "#here, we just remove the spaces that get left in the split-up list\n",
    "lines = []\n",
    "for i in split_up:\n",
    "    if i != '':\n",
    "        lines.append(i)\n",
    "        \n",
    "#we make a dictionary to help us sort through what we have and prepare for moving to a pandas dataframe\n",
    "drudge_headlines = {}\n",
    "drudge_headlines['top_headline'] = lines\n",
    "#drudge headlines change all the time, so we'll want to know the day and the datetime of this scrape\n",
    "drudge_headlines['date'] = date.today()\n",
    "drudge_headlines['time'] = datetime.datetime.now()\n",
    "\n",
    "#pandas\n",
    "Drudge_Report = pd.DataFrame.from_dict(drudge_headlines)\n",
    "\n",
    "#getting the date into the file_name\n",
    "datetime_now_string = datetime.datetime.now().strftime(\"%m-%d-%Y_%H-%M-%p\")\n",
    "Drudge_Report.to_csv(\"drudge_report_\"+datetime_now_string+\".csv\",index=False)\n",
    "#!!--->I prefer to write to Excel for my own purposes, and if you don't have openpyxl, pandas won't write it\n",
    "#\"pip install openpyxl\"\n",
    "Drudge_Report.to_excel(\"drudge_report_\"+datetime_now_string+\".xlsx\",index=False)\n",
    "\n",
    "#here's the dataframe\n",
    "Drudge_Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thank you for stopping by!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}