Day013_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PTT 網路爬蟲實作練習\n",
    "\n",
    "\n",
    "* 能夠利用 Request + BeatifulSour 撰寫爬蟲，並存放到合適的資料結構\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標\n",
    "\n",
    "根據範例 ，完成以下問題：\n",
    "\n",
    "* ① 印出最新文章的「作者」「標題」「時間」\n",
    "* ② 印出第一頁所有文章的「作者」「標題」「時間」\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ① 印出最新文章的「作者」「標題」「時間」"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "url = 'https://www.ptt.cc/bbs/home-sale/index.html'\n",
    "r = requests.get(url)\n",
    "\n",
    "soup = BeautifulSoup(r.text, \"html5lib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                              作者         看板  \\\n",
       "0            extra312 (extra312)  home-sale   \n",
       "1                  kline (kline)  home-sale   \n",
       "2                Jack10 (Jack10)  home-sale   \n",
       "3              peng1022 (reeeed)  home-sale   \n",
       "4                a25042121 (阿水®)  home-sale   \n",
       "5            waeiwwww (waeiwwww)  home-sale   \n",
       "6            BuzzerBeatBB (Mr.M)  home-sale   \n",
       "7                 jdkj1110 (金俗西)  home-sale   \n",
       "8   angel07 (Dark Moon Princess)  home-sale   \n",
       "9   angel07 (Dark Moon Princess)  home-sale   \n",
       "10  angel07 (Dark Moon Princess)  home-sale   \n",
       "11  angel07 (Dark Moon Princess)  home-sale   \n",
       "12  angel07 (Dark Moon Princess)  home-sale   \n",
       "\n",
       "                                   標題                        時間  \n",
       "0                    [買/台南]徵台南皇龍天澍三房型  Tue Dec 29 22:47:15 2020  \n",
       "1                  [請益] 隔壁鑰匙竟然可以開我家大門  Tue Dec 29 22:56:00 2020  \n",
       "2                 [閒聊] 房子賠售就是爽版主被查水錶？  Tue Dec 29 22:57:20 2020  \n",
       "3                           [賣/高雄/鼓山]  Tue Dec 29 23:13:58 2020  \n",
       "4            [賣/新竹/竹北] 國賓大悅時尚裝潢美兩房+平車  Tue Dec 29 23:20:18 2020  \n",
       "5   [賣/新竹/竹北]新社國小 富源 沐學苑 中高樓朝東南 視野採光優  Tue Dec 29 23:59:53 2020  \n",
       "6                     [請益] 隔套地板大家都用哪種  Wed Dec 30 00:13:39 2020  \n",
       "7                         [請益] 該賣還是租人  Wed Dec 30 00:52:52 2020  \n",
       "8                     [公告] 檢舉申訴申請管理辦法  Mon May  1 13:13:00 2017  \n",
       "9                   [公告] home-sale 板規  Thu Jul 12 04:10:59 2018  \n",
       "10                   [公告] 180701置底檢舉區  Sun Jul  1 01:39:27 2018  \n",
       "11           [公告] 板務實行標準 (2018/07/25)  Wed Jul 25 22:24:51 2018  \n",
       "12                  [公告] 2020 過渡時期暫行律  Tue Oct 27 12:26:13 2020  "
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>作者</th>\n      <th>看板</th>\n      <th>標題</th>\n      <th>時間</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>extra312 (extra312)</td>\n      <td>home-sale</td>\n      <td>[買/台南]徵台南皇龍天澍三房型</td>\n      <td>Tue Dec 29 22:47:15 2020</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>kline (kline)</td>\n      <td>home-sale</td>\n      <td>[請益] 隔壁鑰匙竟然可以開我家大門</td>\n      <td>Tue Dec 29 22:56:00 2020</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Jack10 (Jack10)</td>\n      <td>home-sale</td>\n      <td>[閒聊] 房子賠售就是爽版主被查水錶？</td>\n      <td>Tue Dec 29 22:57:20 2020</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>peng1022 (reeeed)</td>\n      <td>home-sale</td>\n      <td>[賣/高雄/鼓山]</td>\n      <td>Tue Dec 29 23:13:58 2020</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>a25042121 (阿水®)</td>\n      <td>home-sale</td>\n      <td>[賣/新竹/竹北] 國賓大悅時尚裝潢美兩房+平車</td>\n      <td>Tue Dec 29 23:20:18 2020</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>waeiwwww (waeiwwww)</td>\n      <td>home-sale</td>\n      <td>[賣/新竹/竹北]新社國小 富源 沐學苑 中高樓朝東南 視野採光優</td>\n      <td>Tue Dec 29 23:59:53 2020</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>BuzzerBeatBB (Mr.M)</td>\n      <td>home-sale</td>\n      <td>[請益] 隔套地板大家都用哪種</td>\n      <td>Wed Dec 30 00:13:39 2020</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>jdkj1110 (金俗西)</td>\n      <td>home-sale</td>\n      <td>[請益] 該賣還是租人</td>\n      <td>Wed Dec 30 00:52:52 2020</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>angel07 (Dark Moon Princess)</td>\n      <td>home-sale</td>\n      <td>[公告] 檢舉申訴申請管理辦法</td>\n      <td>Mon May  1 13:13:00 2017</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>angel07 (Dark Moon Princess)</td>\n      <td>home-sale</td>\n      <td>[公告] home-sale 板規</td>\n      <td>Thu Jul 12 04:10:59 2018</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>angel07 (Dark Moon Princess)</td>\n      <td>home-sale</td>\n      <td>[公告] 180701置底檢舉區</td>\n      <td>Sun Jul  1 01:39:27 2018</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>angel07 (Dark Moon Princess)</td>\n      <td>home-sale</td>\n      <td>[公告] 板務實行標準 (2018/07/25)</td>\n      <td>Wed Jul 25 22:24:51 2018</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>angel07 (Dark Moon Princess)</td>\n      <td>home-sale</td>\n      <td>[公告] 2020 過渡時期暫行律</td>\n      <td>Tue Oct 27 12:26:13 2020</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "info_all = []\n",
    "for d in soup.find_all(class_=\"title\"):\n",
    "    try:\n",
    "        url = 'https://www.ptt.cc/' + d.find('a')['href']\n",
    "        # print(url)\n",
    "        soup_2 = BeautifulSoup(requests.get('https://www.ptt.cc'+d.find('a')['href']).text, \"html5lib\")\n",
    "        info = []\n",
    "        for dd in soup_2.find_all(class_='article-meta-value'):\n",
    "            info.append(dd.text)\n",
    "        info_all.append(info)\n",
    "    except:\n",
    "        continue\n",
    "df = pd.DataFrame(info_all, columns=['作者', '看板', '標題', '時間'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ② 印出第一頁所有文章的「作者」「標題」「時間」"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ③ 試著爬爬看其他版的文章"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}