Day_049_Blending.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 範例 : (Kaggle)房價預測\n",
    "***\n",
    "- 分數以網站評分結果為準, 請同學實際將提交檔(*.csv)上傳試試看  \n",
    "https://www.kaggle.com/c/house-prices-advanced-regression-techniques/submit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [教學目標]\n",
    "- 以下用房價預測資料, 觀查混合泛化 (Blending) 的寫法與效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [範例重點]\n",
    "- 觀察混合泛化的準確度 (In[12]), 是否比單一模型準確度為高 (In[9~11])  \n",
    "- 除了我們的權重, 同學也可以試著自行調整權重 (注意:權重和=1), 看看有什麼影響"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 79 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities LotConfig  ... ScreenPorch PoolArea PoolQC Fence  \\\n",
       "0         Lvl    AllPub    Inside  ...           0        0    NaN   NaN   \n",
       "1         Lvl    AllPub       FR2  ...           0        0    NaN   NaN   \n",
       "2         Lvl    AllPub    Inside  ...           0        0    NaN   NaN   \n",
       "3         Lvl    AllPub    Corner  ...           0        0    NaN   NaN   \n",
       "4         Lvl    AllPub       FR2  ...           0        0    NaN   NaN   \n",
       "\n",
       "  MiscFeature MiscVal  MoSold  YrSold  SaleType  SaleCondition  \n",
       "0         NaN       0       2    2008        WD         Normal  \n",
       "1         NaN       0       5    2007        WD         Normal  \n",
       "2         NaN       0       9    2008        WD         Normal  \n",
       "3         NaN       0       2    2006        WD        Abnorml  \n",
       "4         NaN       0      12    2008        WD         Normal  \n",
       "\n",
       "[5 rows x 79 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 做完特徵工程前的所有準備\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import copy, time\n",
    "from IPython.display import display\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "data_path = 'data/'\n",
    "df_train = pd.read_csv(data_path + 'house_train.csv')\n",
    "df_test = pd.read_csv(data_path + 'house_test.csv')\n",
    "\n",
    "train_Y = np.log1p(df_train['SalePrice'])\n",
    "ids = df_test['Id']\n",
    "df_train = df_train.drop(['Id', 'SalePrice'] , axis=1)\n",
    "df_test = df_test.drop(['Id'] , axis=1)\n",
    "df = pd.concat([df_train, df_test])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Missing Ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>PoolQC</th>\n",
       "      <td>99.657417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MiscFeature</th>\n",
       "      <td>96.402878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Alley</th>\n",
       "      <td>93.216855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fence</th>\n",
       "      <td>80.438506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FireplaceQu</th>\n",
       "      <td>48.646797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LotFrontage</th>\n",
       "      <td>16.649538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageFinish</th>\n",
       "      <td>5.447071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <td>5.447071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageQual</th>\n",
       "      <td>5.447071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageCond</th>\n",
       "      <td>5.447071</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Missing Ratio\n",
       "PoolQC            99.657417\n",
       "MiscFeature       96.402878\n",
       "Alley             93.216855\n",
       "Fence             80.438506\n",
       "FireplaceQu       48.646797\n",
       "LotFrontage       16.649538\n",
       "GarageFinish       5.447071\n",
       "GarageYrBlt        5.447071\n",
       "GarageQual         5.447071\n",
       "GarageCond         5.447071"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 檢查 DataFrame 空缺值的狀態\n",
    "def na_check(df_data):\n",
    "    data_na = (df_data.isnull().sum() / len(df_data)) * 100\n",
    "    data_na = data_na.drop(data_na[data_na == 0].index).sort_values(ascending=False)\n",
    "    missing_data = pd.DataFrame({'Missing Ratio' :data_na})\n",
    "    display(missing_data.head(10))\n",
    "na_check(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 以下 In[3]~In[7] 只是摘錄房產預測中的一組特徵工程, 並以此組特徵工程跑參數, 若更換其他特徵工程, In[8]的參數需要重新跑\n",
    "# 部分欄位缺值補 'None'\n",
    "none_cols = ['PoolQC', 'MiscFeature', 'Alley', 'Fence', 'FireplaceQu', 'FireplaceQu', 'FireplaceQu', 'FireplaceQu', \n",
    "            'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure', \n",
    "             'BsmtFinType1', 'BsmtFinType2', 'MasVnrType', 'Functional', 'MSSubClass']\n",
    "for col in none_cols:\n",
    "    df[col] = df[col].fillna(\"None\")\n",
    "    \n",
    "# 部分欄位缺值填補 0\n",
    "zero_cols = ['GarageYrBlt', 'GarageArea', 'GarageCars', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF','TotalBsmtSF', \n",
    "             'BsmtFullBath', 'BsmtHalfBath', 'MasVnrArea']\n",
    "for col in zero_cols:\n",
    "    df[col] = df[col].fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 部分欄位缺值補眾數\n",
    "mode_cols = ['MSZoning', 'Electrical', 'KitchenQual', 'Exterior1st', 'Exterior2nd', 'SaleType']\n",
    "for col in mode_cols:\n",
    "    df[col] = df[col].fillna(df[col].mode()[0])\n",
    "    \n",
    "# 'LotFrontage' 有空缺時, 以同一區 (Neighborhood) 的 LotFrontage 中位數填補 (可以視為填補一種群聚編碼 )\n",
    "df[\"LotFrontage\"] = df.groupby(\"Neighborhood\")[\"LotFrontage\"].transform(lambda x: x.fillna(x.median()))\n",
    "\n",
    "# Utilities 參考資訊很少, 所以直接捨棄\n",
    "df = df.drop(['Utilities'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Missing Ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [Missing Ratio]\n",
       "Index: []"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 做完各種補缺值, 確認一下有沒有遺漏\n",
    "na_check(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape df: (2919, 79)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>TotalSF</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>Inside</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>2566.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>FR2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>2524.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>Inside</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>2706.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>Corner</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>2473.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>FR2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>3343.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 79 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning  LotFrontage  LotArea  Street  Alley  LotShape  \\\n",
       "0           5       RL         65.0     8450       1      1         3   \n",
       "1           0       RL         80.0     9600       1      1         3   \n",
       "2           5       RL         68.0    11250       1      1         0   \n",
       "3           6       RL         60.0     9550       1      1         0   \n",
       "4           5       RL         84.0    14260       1      1         0   \n",
       "\n",
       "  LandContour LotConfig  LandSlope  ... PoolArea PoolQC Fence MiscFeature  \\\n",
       "0         Lvl    Inside          0  ...        0      3     4        None   \n",
       "1         Lvl       FR2          0  ...        0      3     4        None   \n",
       "2         Lvl    Inside          0  ...        0      3     4        None   \n",
       "3         Lvl    Corner          0  ...        0      3     4        None   \n",
       "4         Lvl       FR2          0  ...        0      3     4        None   \n",
       "\n",
       "  MiscVal  MoSold  YrSold  SaleType  SaleCondition TotalSF  \n",
       "0       0       1       2        WD         Normal  2566.0  \n",
       "1       0       4       1        WD         Normal  2524.0  \n",
       "2       0       8       2        WD         Normal  2706.0  \n",
       "3       0       1       0        WD        Abnorml  2473.0  \n",
       "4       0      11       2        WD         Normal  3343.0  \n",
       "\n",
       "[5 rows x 79 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 四個數值欄位, 因為相異值有限, 轉成文字\n",
    "label_cols = ['MSSubClass', 'OverallCond', 'YrSold', 'MoSold']\n",
    "for col in mode_cols:\n",
    "    df[col] = df[col].astype(str)\n",
    "    \n",
    "# 相異值不太具有代表性的, 做標籤編碼\n",
    "cols = ('FireplaceQu', 'BsmtQual', 'BsmtCond', 'GarageQual', 'GarageCond', \n",
    "        'ExterQual', 'ExterCond','HeatingQC', 'PoolQC', 'KitchenQual', 'BsmtFinType1', \n",
    "        'BsmtFinType2', 'Functional', 'Fence', 'BsmtExposure', 'GarageFinish', 'LandSlope',\n",
    "        'LotShape', 'PavedDrive', 'Street', 'Alley', 'CentralAir', 'MSSubClass', 'OverallCond', \n",
    "        'YrSold', 'MoSold')\n",
    "for c in cols:\n",
    "    lbl = LabelEncoder() \n",
    "    lbl.fit(list(df[c].values)) \n",
    "    df[c] = lbl.transform(list(df[c].values))\n",
    "    \n",
    "# 由地下室面積 + 1樓面積 + 2樓面積, 計算總坪數特徵   \n",
    "df['TotalSF'] = df['TotalBsmtSF'] + df['1stFlrSF'] + df['2ndFlrSF']    \n",
    "    \n",
    "# 看看目前特徵工程的結果\n",
    "print('Shape df: {}'.format(df.shape))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2919, 221)\n"
     ]
    }
   ],
   "source": [
    "# 把剩下少數重要的類別型欄位, 做獨熱編碼 (已變成數字的欄位, 會自動跳過)\n",
    "df = pd.get_dummies(df)\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 將前述轉換完畢資料 df , 重新切成 train_X, test_X\n",
    "train_num = train_Y.shape[0]\n",
    "train_X = df[:train_num]\n",
    "test_X = df[train_num:]\n",
    "\n",
    "# 使用三種模型 : 線性迴歸 / 梯度提升機 / 隨機森林, 參數使用 Random Search 尋找\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor\n",
    "linear = LinearRegression(normalize=False, fit_intercept=True, copy_X=True)\n",
    "gdbt = GradientBoostingRegressor(tol=0.1, subsample=0.37, n_estimators=200, max_features=20, \n",
    "                                 max_depth=6, learning_rate=0.03)\n",
    "rf = RandomForestRegressor(n_estimators=300, min_samples_split=9, min_samples_leaf=10, \n",
    "                           max_features='sqrt', max_depth=8, bootstrap=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 線性迴歸預測檔 (結果有部分隨機, 請以 Kaggle 計算的得分為準, 以下模型同理)\n",
    "linear.fit(train_X, train_Y)\n",
    "linear_pred = linear.predict(test_X)\n",
    "sub = pd.DataFrame({'Id': ids, 'SalePrice': np.expm1(linear_pred)})\n",
    "sub.to_csv('house_linear.csv', index=False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 梯度提升機預測檔 \n",
    "gdbt.fit(train_X, train_Y)\n",
    "gdbt_pred = gdbt.predict(test_X)\n",
    "sub = pd.DataFrame({'Id': ids, 'SalePrice': np.expm1(gdbt_pred)})\n",
    "sub.to_csv('house_gdbt.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 隨機森林預測檔\n",
    "rf.fit(train_X, train_Y)\n",
    "rf_pred = rf.predict(test_X)\n",
    "sub = pd.DataFrame({'Id': ids, 'SalePrice': np.expm1(rf_pred)})\n",
    "sub.to_csv('house_rf.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 混合泛化預測檔 (依 Kaggle 傳回分數調整比重, 越準確者比重越高, 依資料性質有所不同)\n",
    "blending_pred = linear_pred*0.30 + gdbt_pred*0.67 + rf_pred*0.03\n",
    "sub = pd.DataFrame({'Id': ids, 'SalePrice': np.expm1(blending_pred)})\n",
    "sub.to_csv('house_blending.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}