automl_house_prices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3826d24a-474c-431c-8913-6f2cdd7490f5",
   "metadata": {},
   "source": [
    "## Estimating house 🏠 prices 💸 in Iowa's (USA) Ames residential area\n",
    "#### AutoML 🏎️💨 approach"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdaada37-a7a7-47f7-9cff-5c69278e20e4",
   "metadata": {},
   "source": [
    "#### Goal:\n",
    "to estimate house prices using the AutoML regression approach\n",
    "\n",
    "#### Data:\n",
    "Ames house features and prices, originally prepared by Dean De Cock, later modified by Kaggle. Data are accessible at: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data\n",
    "\n",
    "#### Processing steps:\n",
    "1. Downloading the source data using Kaggle API and unzipping it \n",
    "2. Loading the training part into Pandas \n",
    "3. Performing AutoML\n",
    "4. Analyzing the results\n",
    "5. Evaluating the final model on test data\n",
    "\n",
    "#### Sources: \n",
    "[1] Kaggle API usage advices: https://stackoverflow.com/questions/55934733/documentation-for-kaggle-api-within-python <br>\n",
    "[2] AutoML mljar-supervised. API documentation. fit() https://supervised.mljar.com/api/#supervised.automl.AutoML.fit <br>\n",
    "[3] AutoML mljar-supervised. Features documentation. https://supervised.mljar.com/features/automl/<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "950df355-6ed5-4859-bd32-fb0a5e153456",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "91059bba-fa22-49ea-9ba3-bdebcdbfb7c8",
   "metadata": {},
   "source": [
    "### Step 0: Importing packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "459281b9-076a-43a9-b4a9-4228522bf484",
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "import os\n",
    "from math import sqrt\n",
    "import zipfile\n",
    "\n",
    "from kaggle.api.kaggle_api_extended import KaggleApi\n",
    "import pandas as pd\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from supervised.automl import AutoML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ffd0b95-e0ee-4f8b-a85c-896072eae83a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2880dea1-a35f-4713-a70c-3b74fe697e80",
   "metadata": {},
   "source": [
    "### Step 1: Downloading the source data using Kaggle API and unzipping it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bad1a3c-87b0-492c-bc16-c006dbcad27f",
   "metadata": {},
   "source": [
    "As the first step, we need to create the Kaggle account and download the Kaggle API token following instructions in [1]. Then, we are ready to download the source data using the API: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7674963b-885f-4700-93fc-45ac4af10428",
   "metadata": {},
   "outputs": [],
   "source": [
    "api = KaggleApi()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8e0423b9-fdb1-465a-a1df-d3844b4fb45a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /home/CENAGIS/aradecka/.kaggle/kaggle.json'\n"
     ]
    }
   ],
   "source": [
    "api.authenticate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "11ae9c46-2354-407f-9290-3e59224fdcbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "competition_name = 'house-prices-advanced-regression-techniques'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9f95c8fe-a9b0-4c58-b38f-4b25c11ea3aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.competition_download_files(competition_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d3a2c50-3042-48e7-b1ce-27498e4dfd5c",
   "metadata": {},
   "source": [
    "The downloaded *.zip* file can be unzipped with the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "13d808b6-0777-4dc0-bff4-85bf51df451f",
   "metadata": {},
   "outputs": [],
   "source": [
    "with zipfile.ZipFile(competition_name + '.zip', 'r') as zip_ref:\n",
    "    zip_ref.extractall(competition_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5951c3b2-3944-440d-ad89-22455fadfd1b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "604e9a4e-9e03-4b5f-addc-be2cdcfddc25",
   "metadata": {},
   "source": [
    "### Step 2: Loading the training part into Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af318cad-6e8e-4ada-be0d-1891686a1a4d",
   "metadata": {},
   "source": [
    "Let's check what is included in the unzipped folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1b5cd33c-1155-4578-a124-414256d01944",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample_submission.csv\n",
      "data_description.txt\n",
      "test.csv\n",
      "train.csv\n"
     ]
    }
   ],
   "source": [
    "for entry in os.scandir(competition_name):\n",
    "    if entry.is_file():\n",
    "        print(entry.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4f8b0bca-70e6-451d-bd7a-84abce34d773",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(os.path.join(competition_name, 'train.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "40c0dad3-aa36-4712-ac5b-13aa8470c168",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities  ... PoolArea PoolQC Fence MiscFeature MiscVal MoSold  \\\n",
       "0         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "1         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      5   \n",
       "2         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      9   \n",
       "3         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "4         Lvl    AllPub  ...        0    NaN   NaN         NaN       0     12   \n",
       "\n",
       "  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0   2008        WD         Normal     208500  \n",
       "1   2007        WD         Normal     181500  \n",
       "2   2008        WD         Normal     223500  \n",
       "3   2006        WD        Abnorml     140000  \n",
       "4   2008        WD         Normal     250000  \n",
       "\n",
       "[5 rows x 81 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e706aebd-45ca-433d-bbe2-14fed07035a9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7be1ac0d-f3df-4bd9-a3e1-e2038fab5826",
   "metadata": {},
   "source": [
    "### Step 3: Performing AutoML "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c293e450-ab64-4033-b324-32943498e25d",
   "metadata": {},
   "source": [
    "Training with MLjar's AutoML requires to input X (features) and y (target) in a form of Pandas DataFrame/Series or Numpy Array [2]:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "78be1b9d-77aa-47e4-a075-3d2c0418e790",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_df = train_df.drop(['Id', 'SalePrice'], axis=1)\n",
    "y_train_df = train_df[['SalePrice']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861202bf-3aa0-4712-8fe4-33bc7fc2a9a3",
   "metadata": {},
   "source": [
    "In order to start training, we first need to define an AutoML object with parameters of choice. Then *fit()* method needs to be applied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "00c50a89-d18c-4817-b006-d6a8edb280d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_explain = AutoML(mode=\"Explain\", results_path=\"AutoML_Ames_house_prices_explain\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "214faf59-a70b-443f-be59-18e8a7bc9908",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML directory: AutoML_Ames_house_prices_explain\n",
      "The task is regression with evaluation metric rmse\n",
      "AutoML will use algorithms: ['Baseline', 'Linear', 'Decision Tree', 'Random Forest', 'Xgboost', 'Neural Network']\n",
      "AutoML will ensemble available models\n",
      "AutoML steps: ['simple_algorithms', 'default_algorithms', 'ensemble']\n",
      "* Step simple_algorithms will try to check up to 3 models\n",
      "1_Baseline rmse 71466.688145 trained in 1.0 seconds\n",
      "2_DecisionTree rmse 39466.587016 trained in 17.37 seconds\n",
      "3_Linear rmse 42927.088618 trained in 8.37 seconds\n",
      "* Step default_algorithms will try to check up to 3 models\n",
      "4_Default_Xgboost rmse 25655.843849 trained in 10.97 seconds\n",
      "5_Default_NeuralNetwork rmse 26341.418205 trained in 89.79 seconds\n",
      "6_Default_RandomForest rmse 30556.950933 trained in 22.01 seconds\n",
      "* Step ensemble will try to check up to 1 model\n",
      "Ensemble rmse 22919.375107 trained in 0.41 seconds\n",
      "AutoML fit time: 162.44 seconds\n",
      "AutoML best model: Ensemble\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AutoML(results_path='AutoML_Ames_house_prices_explain')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "automl_explain.fit(X_train_df, y_train_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2cf219c-752f-41b9-8a9a-a428f25d9088",
   "metadata": {},
   "source": [
    "In the experiment above I assigned *\"Explain\"* to *mode* parameter in order to explore the learning process in more details. As the next step I would like to focus on finding the most accurate approach to estimating house prices. I therefore select the *\"Compete\"* mode as it allows to evaluate more algorithms, perform more advanced feature engineering and fine-tuning. I also specified *validation_strategy* to 5-fold cross-validation in order to gain an understanding of the consistency of the models' results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3dab5bd5-0fc4-4be6-9d31-7e49fd53879b",
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_compete = AutoML(mode=\"Compete\", results_path=\"AutoML_Ames_house_prices_compete\", \n",
    "                        total_time_limit=86400, \n",
    "                        algorithms=[\"Random Forest\", \n",
    "                                    \"Extra Trees\", \n",
    "                                    \"Xgboost\", \n",
    "                                    \"CatBoost\", \n",
    "                                    \"Neural Network\"],\n",
    "                        validation_strategy={\n",
    "                            \"validation_type\": \"kfold\", \n",
    "                            \"k_folds\": 5, \n",
    "                            \"shuffle\": True,\n",
    "                            \"stratify\": True,\n",
    "                            \"random_seed\": 1234\n",
    "                        }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "08e6b019-0a2b-423d-b510-db727544446e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML directory: AutoML_Ames_house_prices_compete\n",
      "The task is regression with evaluation metric rmse\n",
      "AutoML will use algorithms: ['Random Forest', 'Extra Trees', 'Xgboost', 'CatBoost', 'Neural Network']\n",
      "AutoML will stack models\n",
      "AutoML will ensemble available models\n",
      "AutoML steps: ['simple_algorithms', 'default_algorithms', 'not_so_random', 'mix_encoding', 'golden_features', 'kmeans_features', 'insert_random_feature', 'features_selection', 'hill_climbing_1', 'hill_climbing_2', 'boost_on_errors', 'ensemble', 'stack', 'ensemble_stacked']\n",
      "Skip simple_algorithms because no parameters were generated.\n",
      "* Step default_algorithms will try to check up to 5 models\n",
      "1_Default_Xgboost rmse 28423.938295 trained in 16.87 seconds\n",
      "2_Default_CatBoost rmse 27074.351564 trained in 44.18 seconds\n",
      "3_Default_NeuralNetwork rmse 40392.537148 trained in 473.92 seconds\n",
      "4_Default_RandomForest rmse 34860.934331 trained in 12.29 seconds\n",
      "5_Default_ExtraTrees rmse 36730.560323 trained in 10.46 seconds\n",
      "* Step not_so_random will try to check up to 45 models\n",
      "6_Xgboost rmse 30985.003154 trained in 11.5 seconds\n",
      "15_CatBoost rmse 26679.847333 trained in 267.37 seconds\n",
      "24_RandomForest rmse 34288.447027 trained in 12.85 seconds\n",
      "33_ExtraTrees rmse 37236.561451 trained in 12.02 seconds\n",
      "42_NeuralNetwork rmse 36449.287407 trained in 337.56 seconds\n",
      "7_Xgboost rmse 29461.485451 trained in 15.65 seconds\n",
      "16_CatBoost rmse 27737.451918 trained in 125.84 seconds\n",
      "25_RandomForest rmse 38870.538012 trained in 12.47 seconds\n",
      "34_ExtraTrees rmse 40408.745603 trained in 12.46 seconds\n",
      "43_NeuralNetwork rmse 39517.3603 trained in 585.82 seconds\n",
      "8_Xgboost rmse 28180.102916 trained in 10.97 seconds\n",
      "17_CatBoost rmse 27478.497985 trained in 92.65 seconds\n",
      "26_RandomForest rmse 33185.733763 trained in 9.51 seconds\n",
      "35_ExtraTrees rmse 33843.004069 trained in 10.94 seconds\n",
      "44_NeuralNetwork rmse 58491.166385 trained in 817.24 seconds\n",
      "9_Xgboost rmse 28858.49709 trained in 13.55 seconds\n",
      "18_CatBoost rmse 30835.061387 trained in 63.56 seconds\n",
      "27_RandomForest rmse 33027.808291 trained in 11.63 seconds\n",
      "36_ExtraTrees rmse 34323.384317 trained in 10.13 seconds\n",
      "45_NeuralNetwork rmse 36825.805829 trained in 309.36 seconds\n",
      "10_Xgboost rmse 28439.894402 trained in 18.61 seconds\n",
      "19_CatBoost rmse 27757.096801 trained in 32.77 seconds\n",
      "28_RandomForest rmse 31517.170232 trained in 14.33 seconds\n",
      "37_ExtraTrees rmse 32317.046007 trained in 10.18 seconds\n",
      "46_NeuralNetwork rmse 42081.183833 trained in 712.4 seconds\n",
      "11_Xgboost rmse 28018.546939 trained in 21.62 seconds\n",
      "20_CatBoost rmse 27512.111107 trained in 27.42 seconds\n",
      "29_RandomForest rmse 33035.779445 trained in 15.03 seconds\n",
      "38_ExtraTrees rmse 33299.837713 trained in 10.64 seconds\n",
      "47_NeuralNetwork rmse 41837.166369 trained in 445.72 seconds\n",
      "12_Xgboost rmse 27468.086837 trained in 21.23 seconds\n",
      "21_CatBoost rmse 28506.957618 trained in 288.17 seconds\n",
      "30_RandomForest rmse 34907.494276 trained in 10.36 seconds\n",
      "39_ExtraTrees rmse 36832.254466 trained in 10.0 seconds\n",
      "48_NeuralNetwork rmse 39198.353336 trained in 9.92 seconds\n",
      "13_Xgboost rmse 28821.97926 trained in 17.29 seconds\n",
      "22_CatBoost rmse 26993.018703 trained in 70.28 seconds\n",
      "31_RandomForest rmse 33756.882803 trained in 12.52 seconds\n",
      "40_ExtraTrees rmse 37129.356516 trained in 11.83 seconds\n",
      "49_NeuralNetwork rmse 51157.687465 trained in 598.65 seconds\n",
      "14_Xgboost rmse 28373.916058 trained in 24.18 seconds\n",
      "23_CatBoost rmse 26443.220351 trained in 237.1 seconds\n",
      "32_RandomForest rmse 30417.415411 trained in 28.7 seconds\n",
      "41_ExtraTrees rmse 32144.460242 trained in 12.42 seconds\n",
      "50_NeuralNetwork rmse 36346.799483 trained in 11.41 seconds\n",
      "* Step mix_encoding will try to check up to 1 model\n",
      "12_Xgboost_categorical_mix rmse 28606.735147 trained in 25.46 seconds\n",
      "* Step golden_features will try to check up to 3 models\n",
      "None 10\n",
      "Add Golden Feature: GarageCars_multiply_OverallQual\n",
      "Add Golden Feature: GarageCars_sum_OverallQual\n",
      "Add Golden Feature: YearRemodAdd_ratio_OverallQual\n",
      "Add Golden Feature: OverallQual_ratio_YearRemodAdd\n",
      "Add Golden Feature: TotRmsAbvGrd_multiply_OverallQual\n",
      "Add Golden Feature: YearRemodAdd_multiply_OverallQual\n",
      "Add Golden Feature: FullBath_sum_OverallQual\n",
      "Add Golden Feature: OverallQual_ratio_YrSold\n",
      "Add Golden Feature: YrSold_ratio_OverallQual\n",
      "Add Golden Feature: Fireplaces_sum_OverallQual\n",
      "Created 10 Golden Features in 23.71 seconds.\n",
      "23_CatBoost_GoldenFeatures rmse 26039.0678 trained in 298.92 seconds\n",
      "15_CatBoost_GoldenFeatures rmse 25911.097131 trained in 286.8 seconds\n",
      "22_CatBoost_GoldenFeatures rmse 26989.946919 trained in 72.33 seconds\n",
      "* Step kmeans_features will try to check up to 3 models\n",
      "23_CatBoost_KMeansFeatures rmse 26376.094542 trained in 251.28 seconds\n",
      "15_CatBoost_KMeansFeatures rmse 27171.575017 trained in 272.03 seconds\n",
      "22_CatBoost_KMeansFeatures rmse 30103.957589 trained in 81.51 seconds\n",
      "* Step insert_random_feature will try to check up to 1 model\n",
      "15_CatBoost_GoldenFeatures_RandomFeature rmse 26018.164597 trained in 351.77 seconds\n",
      "Drop features ['WoodDeckSF', 'MSSubClass', 'Electrical', 'BldgType', 'Condition2', 'random_feature', 'BsmtUnfSF', 'HeatingQC', 'LotFrontage', 'GarageType', 'BsmtCond', 'LotConfig', 'PoolArea', 'FullBath', 'ExterCond', 'BsmtHalfBath', 'GarageQual', 'MiscFeature', 'GarageCond', 'PoolQC', 'Heating', 'Utilities', 'MiscVal', 'BsmtFinType2', 'Street', '3SsnPorch', 'Fence', 'RoofStyle', 'LotShape', 'LowQualFinSF', 'RoofMatl', 'Exterior2nd', 'YrSold']\n",
      "* Step features_selection will try to check up to 5 models\n",
      "15_CatBoost_GoldenFeatures_SelectedFeatures rmse 25252.068649 trained in 132.73 seconds\n",
      "12_Xgboost_SelectedFeatures rmse 27350.062141 trained in 47.1 seconds\n",
      "32_RandomForest_SelectedFeatures rmse 29720.518553 trained in 13.37 seconds\n",
      "41_ExtraTrees_SelectedFeatures rmse 30789.049466 trained in 12.33 seconds\n",
      "50_NeuralNetwork_SelectedFeatures rmse 35666.375296 trained in 11.28 seconds\n",
      "* Step hill_climbing_1 will try to check up to 21 models\n",
      "51_CatBoost_GoldenFeatures_SelectedFeatures rmse 26090.586586 trained in 65.82 seconds\n",
      "52_CatBoost_GoldenFeatures_SelectedFeatures rmse 26143.455225 trained in 236.16 seconds\n",
      "53_CatBoost_GoldenFeatures rmse 26191.85502 trained in 150.17 seconds\n",
      "54_CatBoost_GoldenFeatures rmse 26840.03697 trained in 468.71 seconds\n",
      "55_CatBoost_GoldenFeatures rmse 25924.499257 trained in 108.48 seconds\n",
      "56_CatBoost_GoldenFeatures rmse 25695.69869 trained in 390.05 seconds\n",
      "57_Xgboost_SelectedFeatures rmse 27118.009535 trained in 31.69 seconds\n",
      "58_Xgboost rmse 27889.3399 trained in 18.32 seconds\n",
      "59_Xgboost rmse 27632.851465 trained in 19.22 seconds\n",
      "60_Xgboost rmse 27769.287833 trained in 15.53 seconds\n",
      "61_RandomForest_SelectedFeatures rmse 29990.271799 trained in 14.69 seconds\n",
      "62_RandomForest rmse 30181.362407 trained in 16.94 seconds\n",
      "63_ExtraTrees_SelectedFeatures rmse 31027.058762 trained in 13.13 seconds\n",
      "64_RandomForest rmse 31179.454932 trained in 13.73 seconds\n",
      "65_RandomForest rmse 31151.82732 trained in 13.72 seconds\n",
      "66_ExtraTrees rmse 31608.557196 trained in 12.74 seconds\n",
      "67_ExtraTrees rmse 32176.14232 trained in 14.46 seconds\n",
      "68_ExtraTrees rmse 32628.734801 trained in 15.87 seconds\n",
      "69_NeuralNetwork_SelectedFeatures rmse 36517.282846 trained in 10.57 seconds\n",
      "70_NeuralNetwork rmse 38653.710771 trained in 11.93 seconds\n",
      "71_NeuralNetwork rmse 39397.89486 trained in 332.23 seconds\n",
      "* Step hill_climbing_2 will try to check up to 17 models\n",
      "72_CatBoost_GoldenFeatures_SelectedFeatures rmse 25248.397843 trained in 124.12 seconds\n",
      "73_CatBoost_GoldenFeatures_SelectedFeatures rmse 25737.897614 trained in 163.87 seconds\n",
      "74_CatBoost_GoldenFeatures rmse 25946.126774 trained in 591.39 seconds\n",
      "75_CatBoost_GoldenFeatures rmse 26773.149736 trained in 450.34 seconds\n",
      "76_CatBoost_GoldenFeatures rmse 25986.595814 trained in 448.66 seconds\n",
      "77_Xgboost_SelectedFeatures rmse 26601.765071 trained in 30.98 seconds\n",
      "78_Xgboost_SelectedFeatures rmse 26845.946879 trained in 32.48 seconds\n",
      "79_Xgboost_SelectedFeatures rmse 27305.084026 trained in 36.68 seconds\n",
      "80_Xgboost_SelectedFeatures rmse 27192.310902 trained in 37.8 seconds\n",
      "81_Xgboost rmse 26787.402291 trained in 38.96 seconds\n",
      "82_Xgboost rmse 27321.04051 trained in 31.71 seconds\n",
      "83_RandomForest_SelectedFeatures rmse 30123.424222 trained in 18.07 seconds\n",
      "84_RandomForest_SelectedFeatures rmse 31013.941234 trained in 25.17 seconds\n",
      "85_RandomForest rmse 30716.50992 trained in 20.29 seconds\n",
      "86_ExtraTrees_SelectedFeatures rmse 32075.055742 trained in 15.81 seconds\n",
      "87_ExtraTrees_SelectedFeatures rmse 31211.680127 trained in 15.48 seconds\n",
      "88_ExtraTrees rmse 31463.8079 trained in 16.01 seconds\n",
      "* Step boost_on_errors will try to check up to 1 model\n",
      "72_CatBoost_GoldenFeatures_SelectedFeatures_BoostOnErrors rmse 25474.811986 trained in 197.97 seconds\n",
      "* Step ensemble will try to check up to 1 model\n",
      "Ensemble rmse 24805.691622 trained in 40.36 seconds\n",
      "* Step stack will try to check up to 49 models\n",
      "72_CatBoost_GoldenFeatures_SelectedFeatures_Stacked rmse 26382.446519 trained in 149.13 seconds\n",
      "77_Xgboost_SelectedFeatures_Stacked rmse 27652.526873 trained in 34.49 seconds\n",
      "32_RandomForest_SelectedFeatures_Stacked rmse 26652.930375 trained in 25.87 seconds\n",
      "41_ExtraTrees_SelectedFeatures_Stacked rmse 25286.78969 trained in 19.11 seconds\n",
      "50_NeuralNetwork_SelectedFeatures_Stacked rmse 83759.653736 trained in 253.52 seconds\n",
      "15_CatBoost_GoldenFeatures_SelectedFeatures_Stacked rmse 26356.199883 trained in 198.27 seconds\n",
      "81_Xgboost_Stacked rmse 28120.591144 trained in 39.04 seconds\n",
      "61_RandomForest_SelectedFeatures_Stacked rmse 26624.96216 trained in 33.77 seconds\n",
      "63_ExtraTrees_SelectedFeatures_Stacked rmse 25934.107723 trained in 17.57 seconds\n",
      "50_NeuralNetwork_Stacked rmse 50087278.499454 trained in 139.14 seconds\n",
      "56_CatBoost_GoldenFeatures_Stacked rmse 26551.833037 trained in 428.04 seconds\n",
      "78_Xgboost_SelectedFeatures_Stacked rmse 27799.812056 trained in 26.25 seconds\n",
      "83_RandomForest_SelectedFeatures_Stacked rmse 26051.592014 trained in 29.23 seconds\n",
      "87_ExtraTrees_SelectedFeatures_Stacked rmse 25644.97937 trained in 19.26 seconds\n",
      "42_NeuralNetwork_Stacked rmse 98124.584664 trained in 377.53 seconds\n",
      "73_CatBoost_GoldenFeatures_SelectedFeatures_Stacked rmse 26216.34316 trained in 224.54 seconds\n",
      "57_Xgboost_SelectedFeatures_Stacked rmse 28542.309596 trained in 28.41 seconds\n",
      "62_RandomForest_Stacked rmse 26745.263383 trained in 36.15 seconds\n",
      "88_ExtraTrees_Stacked rmse 26043.198417 trained in 16.91 seconds\n",
      "69_NeuralNetwork_SelectedFeatures_Stacked rmse 90830.326644 trained in 183.19 seconds\n",
      "15_CatBoost_GoldenFeatures_Stacked rmse 26779.648417 trained in 246.79 seconds\n",
      "80_Xgboost_SelectedFeatures_Stacked rmse 28211.114415 trained in 27.08 seconds\n",
      "32_RandomForest_Stacked rmse 26506.622944 trained in 26.11 seconds\n",
      "66_ExtraTrees_Stacked rmse 25961.040212 trained in 23.26 seconds\n",
      "45_NeuralNetwork_Stacked rmse 52176973.951104 trained in 952.46 seconds\n",
      "55_CatBoost_GoldenFeatures_Stacked rmse 26529.874906 trained in 123.74 seconds\n",
      "79_Xgboost_SelectedFeatures_Stacked rmse 27751.978556 trained in 32.45 seconds\n",
      "85_RandomForest_Stacked rmse 26386.28272 trained in 21.31 seconds\n",
      "86_ExtraTrees_SelectedFeatures_Stacked rmse 25347.495153 trained in 15.48 seconds\n",
      "70_NeuralNetwork_Stacked rmse 95255.348714 trained in 381.9 seconds\n",
      "74_CatBoost_GoldenFeatures_Stacked rmse 26530.047028 trained in 478.66 seconds\n",
      "82_Xgboost_Stacked rmse 28155.319404 trained in 38.56 seconds\n",
      "84_RandomForest_SelectedFeatures_Stacked rmse 26579.745764 trained in 28.87 seconds\n",
      "41_ExtraTrees_Stacked rmse 25941.378437 trained in 17.86 seconds\n",
      "48_NeuralNetwork_Stacked rmse 34530917.469272 trained in 1753.46 seconds\n",
      "76_CatBoost_GoldenFeatures_Stacked rmse 27110.95052 trained in 235.47 seconds\n",
      "12_Xgboost_SelectedFeatures_Stacked rmse 28258.478949 trained in 45.8 seconds\n",
      "65_RandomForest_Stacked rmse 26794.451451 trained in 38.16 seconds\n",
      "67_ExtraTrees_Stacked rmse 25635.650187 trained in 19.73 seconds\n",
      "71_NeuralNetwork_Stacked rmse 6222893.73779 trained in 608.27 seconds\n",
      "23_CatBoost_GoldenFeatures_Stacked rmse 26384.050442 trained in 250.07 seconds\n",
      "12_Xgboost_Stacked rmse 29063.209208 trained in 34.34 seconds\n",
      "64_RandomForest_Stacked rmse 26952.240027 trained in 31.27 seconds\n",
      "37_ExtraTrees_Stacked rmse 25491.487293 trained in 22.94 seconds\n",
      "43_NeuralNetwork_Stacked rmse 21753157.084047 trained in 571.53 seconds\n",
      "59_Xgboost_Stacked rmse 28015.291056 trained in 25.29 seconds\n",
      "28_RandomForest_Stacked rmse 26872.64696 trained in 29.25 seconds\n",
      "68_ExtraTrees_Stacked rmse 25523.612506 trained in 19.8 seconds\n",
      "3_Default_NeuralNetwork_Stacked rmse 2212708.90728 trained in 1547.38 seconds\n",
      "* Step ensemble_stacked will try to check up to 1 model\n",
      "Ensemble_Stacked rmse 24488.359141 trained in 87.89 seconds\n",
      "AutoML fit time: 22394.47 seconds\n",
      "AutoML best model: Ensemble_Stacked\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AutoML(algorithms=['Random Forest', 'Extra Trees', 'Xgboost', 'CatBoost',\n",
       "                   'Neural Network'],\n",
       "       mode='Compete', results_path='AutoML_Ames_house_prices_compete',\n",
       "       total_time_limit=86400,\n",
       "       validation_strategy={'k_folds': 5, 'random_seed': 1234, 'shuffle': True,\n",
       "                            'stratify': True, 'validation_type': 'kfold'})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "automl_compete.fit(X_train_df, y_train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7f68e32-64c6-4ab4-84d9-f90c3c0f7960",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6a1ea8c5-04c9-4185-83b0-9c77260b119f",
   "metadata": {},
   "source": [
    "The computation results have a form of a folder saved to '*results_path*', which was defined as a parameter of the AutoML object. <br>\n",
    "The folder includes:\n",
    "- subfolders for tested algorithms, including the '*Baseline*' (naive); in the case of the '*compete*' mode we can observe that there are a lot more subfolders as not only algorithms are being compared, but also the hyperparameters - ones related to the algorithm's design which can be found in '*framework.json*' and feature selection/extraction approaches applied which are denoted in the subfolder's name\n",
    "- '*.json*' files informing about input data characteristics ('*data.json*'), 'Features with lower importance than random_feature' [3] ('*drop_features.json*'), characteristics of the new features created ('*golden_features.json*'), parameters of the AutoML process started ('*params.json*') and completed ('*progress.json*')\n",
    "- '*.png*' and '*.csv*' files presenting the summary results enabling to compare algorithms\n",
    "- '*.npy*' files showing which samples were used for training and validation in each of the folds "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "720229f8-114a-4107-abda-6b0a21022f2b",
   "metadata": {},
   "source": [
    "<img src=\"image/mljar_explain_results.jpg\" width=\"400\"/> <img src=\"image/mljar_compete_results.jpg\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1b779c6-4247-446d-a0fc-1a1d0ea33c0f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3bbae657-67e2-4922-9650-fff6404653ea",
   "metadata": {},
   "source": [
    "### Step 4: Analyzing the results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a84d178-0267-4bef-88eb-8dcd528e403d",
   "metadata": {},
   "source": [
    "First, let's have a look at selected summary results - files named '*ldb_performance.png*'. The Y-axis show RMSE values, while X-axis the consecutive algorithms tested (name '*#iteration*' may be a bit misleading, but it just denotes the order of AutoML calculations). The plot on the left ('*explain*' mode) is a bit easier to interpret as there are only 7 results to compare. We can observe that '*Ensemble*' model (in this case a combination of Xgboost and Neural Networks) works best. Decision tree and Linear models are performing the worst. The plot on the right ('*compete*' mode), because of the wide scale of RMSE values, does not allow to compare best performing algorithms: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9bd10c2-55cc-4c5a-a82a-21d7f2e20e54",
   "metadata": {},
   "source": [
    "<img src=\"image/ldb_performance.png\" width=\"500\"/> <img src=\"image/ldb_performance_compete.png\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ef0934-9d30-4228-ab4d-36e8424d643a",
   "metadata": {},
   "source": [
    "We can however check this informatio by looking into '*leaderboard.csv*':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6993f98a-ddc0-46db-83a6-badddb876624",
   "metadata": {},
   "outputs": [],
   "source": [
    "lb_compete_df = pd.read_csv(os.path.join('AutoML_Ames_house_prices_compete', 'leaderboard.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b6f28f36-0324-4833-b3fa-a1dc9a450747",
   "metadata": {},
   "outputs": [],
   "source": [
    "lb_compete_df.sort_values(by='metric_value', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e2e04e42-3e29-4032-ad57-fae29875faed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>model_type</th>\n",
       "      <th>metric_type</th>\n",
       "      <th>metric_value</th>\n",
       "      <th>train_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>Ensemble_Stacked</td>\n",
       "      <td>Ensemble</td>\n",
       "      <td>rmse</td>\n",
       "      <td>24488.359141</td>\n",
       "      <td>87.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>Ensemble</td>\n",
       "      <td>Ensemble</td>\n",
       "      <td>rmse</td>\n",
       "      <td>24805.691622</td>\n",
       "      <td>40.36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>72_CatBoost_GoldenFeatures_SelectedFeatures</td>\n",
       "      <td>CatBoost</td>\n",
       "      <td>rmse</td>\n",
       "      <td>25248.397843</td>\n",
       "      <td>124.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>15_CatBoost_GoldenFeatures_SelectedFeatures</td>\n",
       "      <td>CatBoost</td>\n",
       "      <td>rmse</td>\n",
       "      <td>25252.068649</td>\n",
       "      <td>133.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>41_ExtraTrees_SelectedFeatures_Stacked</td>\n",
       "      <td>Extra Trees</td>\n",
       "      <td>rmse</td>\n",
       "      <td>25286.789690</td>\n",
       "      <td>19.90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            name   model_type metric_type  \\\n",
       "152                             Ensemble_Stacked     Ensemble        rmse   \n",
       "102                                     Ensemble     Ensemble        rmse   \n",
       "84   72_CatBoost_GoldenFeatures_SelectedFeatures     CatBoost        rmse   \n",
       "58   15_CatBoost_GoldenFeatures_SelectedFeatures     CatBoost        rmse   \n",
       "106       41_ExtraTrees_SelectedFeatures_Stacked  Extra Trees        rmse   \n",
       "\n",
       "     metric_value  train_time  \n",
       "152  24488.359141       87.89  \n",
       "102  24805.691622       40.36  \n",
       "84   25248.397843      124.72  \n",
       "58   25252.068649      133.33  \n",
       "106  25286.789690       19.90  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lb_compete_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff71ee5f-70c2-4f7d-8d2a-3f856d7257fb",
   "metadata": {},
   "source": [
    "We can see that similarly to '*explain*' mode's results, results got for '*compete*' indicate '*Ensemble*' model as best. This time however it is created from 8 elements, not 2. In order to check what is the absolute value of the difference between best '*explain*' and '*compete*' modes' validation results, we can do a quick calculation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "479f6059-a81a-4eb5-9ae2-548c67fdab70",
   "metadata": {},
   "outputs": [],
   "source": [
    "lb_explain_df = pd.read_csv(os.path.join('AutoML_Ames_house_prices_explain', 'leaderboard.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7e13c5ca-7b39-4d82-899c-5fc20eb49527",
   "metadata": {},
   "outputs": [],
   "source": [
    "lb_explain_df.sort_values(by='metric_value', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "eb30cf2f-ae58-42e9-80c1-cb26dc8d58ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1568.98"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs_diff = lb_compete_df.iloc[0]['metric_value'] - lb_explain_df.iloc[0]['metric_value']\n",
    "round(abs_diff, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f8474ad-2115-472d-a474-c41fc7f8ea78",
   "metadata": {},
   "source": [
    "The difference is not huge when one is comparing it to absolute sale prices, but probably significant for '*compete*' mode's users taking part in Kaggle challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19fd42d1-65e9-45fb-abff-a72a31bfeda3",
   "metadata": {},
   "source": [
    "MLjar also provides useful plots for in-detail analysis the results got for each algoritm. We can view the learning curves (on the left - CatBoost, on the right - RandomForest):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886fdb83-2c89-4056-8dca-9f43acbc94d9",
   "metadata": {},
   "source": [
    "<img src=\"image/learning_curves_15_catboost.png\" width=\"500\"/> <img src=\"image/learning_curves_31_randomforest.png\" width=\"500\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a24e9d23-f6af-45a1-98a1-031d009c2303",
   "metadata": {},
   "source": [
    "Residuals (for Random Forest visibly more dispersed):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1729f0c0-dd09-4f17-b842-1a22b801478e",
   "metadata": {},
   "source": [
    "<img src=\"image/predicted_vs_residuals_15_catboost.png\" width=\"500\"/> <img src=\"image/predicted_vs_residuals_31_randomforest.png\" width=\"500\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78d42d0b-d6e6-4129-a5e0-2201e16bbdfe",
   "metadata": {},
   "source": [
    "Relationship between target and predicted values (for Random Forest, predicted values between 10.000 and 20.000 seem to be a lowered as compared to the true ones):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a640145e-4be8-4f9f-86f4-a79fc018175a",
   "metadata": {},
   "source": [
    "<img src=\"image/true_vs_predicted_15_catboost.png\" width=\"500\"/> <img src=\"image/true_vs_predicted_31_randomforest.png\" width=\"500\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eabec94-c049-4a40-8b2f-243d3d76bb28",
   "metadata": {},
   "source": [
    "The '*explain*' mode also offers feature importance plots (here permutation-based, for Xgboost algorithm). The plot shows the the '*Overallqual*' ('*Rates the overall material and finish of the house*'), '*GrLivArea*' ('*Above grade (ground) living area square feet*') and '*TotalBsmSF*' ('*Total square feet of basement area*') are the top three important predictors of the sale price:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3a3ae85-146f-4d4e-a05d-2f2e4d27a777",
   "metadata": {},
   "source": [
    "<img src=\"image/permutation_importance.png\" width=\"500\"/> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "101c50c2-8c0a-4204-afd0-e4f156250517",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "52fc34b3-2e2b-4df0-a16b-2afe410bf97c",
   "metadata": {},
   "source": [
    "### Step 5: Evaluating the final model on test data "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74375e3-004a-428e-a8bc-b238ed6631aa",
   "metadata": {},
   "source": [
    "At first, we need to prepare the test features and predict the sale price using the trained AutoML objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5e8afa03-85fd-4297-8860-1be06652ee0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.read_csv(os.path.join(competition_name, 'test.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3941ad4a-e245-4d6e-a0a9-6d77ecd69935",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1459, 80)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.shape # almost equal to the size of the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cc80d672-e063-40cc-be3a-a45cf61b1f9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_df = test_df.drop('Id', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c4dcc36f-1b76-4af2-aa7f-6c0b42b93ea8",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n"
     ]
    }
   ],
   "source": [
    "preds_explain = automl_explain.predict(X_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "52ec91f2-e1f5-4806-99b9-886ae257fd4f",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n",
      "/home/CENAGIS/aradecka/.local/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but StandardScaler was fitted without feature names\n"
     ]
    }
   ],
   "source": [
    "preds_compete = automl_compete.predict(X_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "636045bf-d33f-43c2-ae7f-e57b61a72198",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(numpy.ndarray, (1459,))"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(preds_explain), preds_explain.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "16f42539-d225-45b0-a418-91a07d051038",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([125835.13331867, 160868.93409775, 165978.22316025, ...,\n",
       "       166623.73097275, 123589.31567258, 231148.82614762])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_explain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3790b489-6e60-4ebb-96f5-6bdd80e35253",
   "metadata": {},
   "source": [
    "The resultant predictions have a form of Numpy Arrays. They need to be transformed into Pandas DataFrames with '*Id*' and '*SalePrice*' columns to fit the submission format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "78a7ba46-5dee-4f9c-af48-38b33b865455",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_explain_df = copy.deepcopy(test_df)\n",
    "preds_compete_df = copy.deepcopy(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2bf36fa8-8f71-495f-b80e-9984fda3995c",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_explain_df['SalePrice'] = list(preds_explain)\n",
    "preds_compete_df['SalePrice'] = list(preds_compete)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "8abd50a1-57b5-416d-ae98-ee69ab129eb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_explain_df = preds_explain_df[['Id', 'SalePrice']]\n",
    "preds_compete_df = preds_compete_df[['Id', 'SalePrice']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "795323fd-8291-406b-b5cb-944644def6e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_explain_df.to_csv('test_predictions_explain.csv', index=False)\n",
    "preds_compete_df.to_csv('test_predictions_compete.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fca0aa78-7d72-4285-b707-012e7abd2dc8",
   "metadata": {},
   "source": [
    "Finally, we can upload the '*explain*' and '*compete*' predictions to Kaggle for evaluation:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a03bc21f-69f3-437e-bd0c-a51b3ba9196f",
   "metadata": {},
   "source": [
    "<img src=\"image/kaggle_screen.png\" width=\"700\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7e385c1-df9e-4a06-a7d3-f7eb45b7649b",
   "metadata": {},
   "source": [
    "The obtained score's value does not look familiar, as it is the RMSE between log-scaled predicted and true values. The usage of AutoML compete mode allowed to get a result being in 16th percentile of the whole challenge's leaderboard 🎉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60542f32-1211-4cfe-a09e-6035822c9ab9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}