08_prep_data_for_machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing data for machine learning\n",
    "\n",
    "This notebook takes the output DataFrames the previous noebook, [07_get_many_historical_forecasts.ipynb](07_get_many_historical_forecasts.ipynb), and prepares them to be used in machine learning models later. \n",
    "\n",
    "This includes:\n",
    "- combining forecasts, actuals, and uncertainty indicators into a single DataFrame with useful column labels\n",
    "- adding solar position and month for each interval to be used as features\n",
    "- normalizing all power values to clear sky power (i.e., calculating clear sky indices)\n",
    "\n",
    "The last two steps are critical, as they will give our regression models much more robust features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pvlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "plants_df = pd.read_csv('pv_fleet_metadata_rev3.csv') # read in as a Pandas DataFrame\n",
    "plants_df = plants_df[plants_df['ac_capacity']>=10].copy().reset_index()\n",
    "num_plants = len(plants_df.index) # number of plants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_ac_all_act_2021 = pd.read_pickle('data_files/p_ac_all_act_2021.pkl')\n",
    "p_ac_all_cs_2021 = pd.read_pickle('data_files/p_ac_all_cs_2021.pkl')\n",
    "p_ac_all_fcast_2021 = pd.read_pickle('data_files/p_ac_all_fcast_2021.pkl')\n",
    "p_ac_all_fcast_max_2021 = pd.read_pickle('data_files/p_ac_all_fcast_max_2021.pkl')\n",
    "p_ac_all_fcast_min_2021 = pd.read_pickle('data_files/p_ac_all_fcast_min_2021.pkl')\n",
    "\n",
    "p_ac_all_act_2022 = pd.read_pickle('data_files/p_ac_all_act_2022.pkl')\n",
    "p_ac_all_cs_2022 = pd.read_pickle('data_files/p_ac_all_cs_2022.pkl')\n",
    "p_ac_all_fcast_2022 = pd.read_pickle('data_files/p_ac_all_fcast_2022.pkl')\n",
    "p_ac_all_fcast_max_2022 = pd.read_pickle('data_files/p_ac_all_fcast_max_2022.pkl')\n",
    "p_ac_all_fcast_min_2022 = pd.read_pickle('data_files/p_ac_all_fcast_min_2022.pkl')\n",
    "\n",
    "p_ac_all_act_2023 = pd.read_pickle('data_files/p_ac_all_act_2023.pkl')\n",
    "p_ac_all_cs_2023 = pd.read_pickle('data_files/p_ac_all_cs_2023.pkl')\n",
    "p_ac_all_fcast_2023 = pd.read_pickle('data_files/p_ac_all_fcast_2023.pkl')\n",
    "p_ac_all_fcast_max_2023 = pd.read_pickle('data_files/p_ac_all_fcast_max_2023.pkl')\n",
    "p_ac_all_fcast_min_2023 = pd.read_pickle('data_files/p_ac_all_fcast_min_2023.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_avg_tcc = pd.read_pickle('data_files/w_avg_tcc_2021-2023.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenate data across years:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_ac_all_act = pd.concat([p_ac_all_act_2021, p_ac_all_act_2022, p_ac_all_act_2023], axis=0)\n",
    "p_ac_all_cs = pd.concat([p_ac_all_cs_2021, p_ac_all_cs_2022, p_ac_all_cs_2023], axis=0)\n",
    "p_ac_all_fcast = pd.concat([p_ac_all_fcast_2021, p_ac_all_fcast_2022 ,p_ac_all_fcast_2023], axis=0)\n",
    "p_ac_all_fcast_max = pd.concat([p_ac_all_fcast_max_2021, p_ac_all_fcast_max_2022, p_ac_all_fcast_max_2023], axis=0)\n",
    "p_ac_all_fcast_min = pd.concat([p_ac_all_fcast_min_2021, p_ac_all_fcast_min_2022, p_ac_all_fcast_min_2023], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rename columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_ac_all_act.rename(columns={'power_ac': 'p_ac_all_act'}, inplace=True)\n",
    "p_ac_all_cs.rename(columns={'power_ac': 'p_ac_all_cs'}, inplace=True)\n",
    "p_ac_all_fcast.rename(columns={'power_ac': 'p_ac_all_fcast'}, inplace=True)\n",
    "p_ac_all_fcast_max.rename(columns={'power_ac': 'p_ac_all_fcast_max'}, inplace=True)\n",
    "p_ac_all_fcast_min.rename(columns={'power_ac': 'p_ac_all_fcast_min'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine into a single DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of dataframes to combine first\n",
    "dfs = [\n",
    "    p_ac_all_fcast,\n",
    "    p_ac_all_fcast_max,\n",
    "    p_ac_all_fcast_min,\n",
    "    p_ac_all_cs,\n",
    "    p_ac_all_act,\n",
    "    w_avg_tcc.tcc_std,\n",
    "    ]\n",
    "\n",
    "# combine all dataframes and drop na rows\n",
    "df = pd.concat(dfs, axis=1).dropna()\n",
    "\n",
    "df.index.name = 'time_center_labeled'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate normalized Max-Min:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['max_min_norm'] = (df['p_ac_all_fcast_max'] - df['p_ac_all_fcast_min'])/df['p_ac_all_cs']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate error statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "approx_nameplate = df.p_ac_all_cs.max() # use max of clear sky power as proxy for nameplate rating\n",
    "df['abs_err'] = (df.p_ac_all_fcast - df.p_ac_all_act).abs()\n",
    "df['norm_abs_err'] = df.abs_err/approx_nameplate\n",
    "df['csi_norm_abs_err'] = df.abs_err/df.p_ac_all_cs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize to clear sky by calculating clear sky index (CSI) of actual and forecasted power:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['csi_actual'] = df['p_ac_all_act']/df['p_ac_all_cs']\n",
    "df['csi_fcast'] = df['p_ac_all_fcast']/df['p_ac_all_cs']\n",
    "# df['csi_fcast_max'] = df['p_ac_all_fcast_max']/df['p_ac_all_cs']\n",
    "# df['csi_fcast_min'] = df['p_ac_all_fcast_min']/df['p_ac_all_cs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add month, hour, solar azimuth, and solar elevation\n",
    "latitude, longitude = 32.8 , -83.6\n",
    "solpos = pvlib.solarposition.get_solarposition(time=df.index,latitude=latitude, longitude=longitude)\n",
    "\n",
    "df['hour'] = df.index.hour\n",
    "df['month'] = df.index.month\n",
    "df['el'] = solpos.apparent_elevation\n",
    "df['zen'] = solpos.apparent_zenith\n",
    "df['az'] = solpos.azimuth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>p_ac_all_fcast</th>\n",
       "      <th>p_ac_all_fcast_max</th>\n",
       "      <th>p_ac_all_fcast_min</th>\n",
       "      <th>p_ac_all_cs</th>\n",
       "      <th>level_0</th>\n",
       "      <th>p_ac_all_act</th>\n",
       "      <th>tcc_std</th>\n",
       "      <th>max_min_norm</th>\n",
       "      <th>abs_err</th>\n",
       "      <th>norm_abs_err</th>\n",
       "      <th>csi_norm_abs_err</th>\n",
       "      <th>csi_actual</th>\n",
       "      <th>csi_fcast</th>\n",
       "      <th>hour</th>\n",
       "      <th>month</th>\n",
       "      <th>el</th>\n",
       "      <th>zen</th>\n",
       "      <th>az</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_center_labeled</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-01-01 09:00:00+00:00</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.657303</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>-44.725579</td>\n",
       "      <td>134.725579</td>\n",
       "      <td>90.865286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 10:00:00+00:00</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.346134</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>-32.169594</td>\n",
       "      <td>122.169594</td>\n",
       "      <td>98.221778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 11:00:00+00:00</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.346134</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>-19.832171</td>\n",
       "      <td>109.832171</td>\n",
       "      <td>105.126225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 12:00:00+00:00</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.346134</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.896291</td>\n",
       "      <td>97.896291</td>\n",
       "      <td>112.278339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 13:00:00+00:00</th>\n",
       "      <td>46.237118</td>\n",
       "      <td>146.468144</td>\n",
       "      <td>4.036621</td>\n",
       "      <td>827.123924</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32.438303</td>\n",
       "      <td>4.919961</td>\n",
       "      <td>0.172201</td>\n",
       "      <td>13.798815</td>\n",
       "      <td>0.00372</td>\n",
       "      <td>0.016683</td>\n",
       "      <td>0.039218</td>\n",
       "      <td>0.055901</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>3.619760</td>\n",
       "      <td>86.380240</td>\n",
       "      <td>120.212465</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           p_ac_all_fcast  p_ac_all_fcast_max  \\\n",
       "time_center_labeled                                             \n",
       "2021-01-01 09:00:00+00:00        0.000000            0.000000   \n",
       "2021-01-01 10:00:00+00:00        0.000000            0.000000   \n",
       "2021-01-01 11:00:00+00:00        0.000000            0.000000   \n",
       "2021-01-01 12:00:00+00:00        0.000000            0.000000   \n",
       "2021-01-01 13:00:00+00:00       46.237118          146.468144   \n",
       "\n",
       "                           p_ac_all_fcast_min  p_ac_all_cs  level_0  \\\n",
       "time_center_labeled                                                   \n",
       "2021-01-01 09:00:00+00:00            0.000000     0.000000      0.0   \n",
       "2021-01-01 10:00:00+00:00            0.000000     0.000000      0.0   \n",
       "2021-01-01 11:00:00+00:00            0.000000     0.000000      0.0   \n",
       "2021-01-01 12:00:00+00:00            0.000000     0.000000      0.0   \n",
       "2021-01-01 13:00:00+00:00            4.036621   827.123924      0.0   \n",
       "\n",
       "                           p_ac_all_act   tcc_std  max_min_norm    abs_err  \\\n",
       "time_center_labeled                                                          \n",
       "2021-01-01 09:00:00+00:00      0.000000  0.657303           NaN   0.000000   \n",
       "2021-01-01 10:00:00+00:00      0.000000  1.346134           NaN   0.000000   \n",
       "2021-01-01 11:00:00+00:00      0.000000  1.346134           NaN   0.000000   \n",
       "2021-01-01 12:00:00+00:00      0.000000  1.346134           NaN   0.000000   \n",
       "2021-01-01 13:00:00+00:00     32.438303  4.919961      0.172201  13.798815   \n",
       "\n",
       "                           norm_abs_err  csi_norm_abs_err  csi_actual  \\\n",
       "time_center_labeled                                                     \n",
       "2021-01-01 09:00:00+00:00       0.00000               NaN         NaN   \n",
       "2021-01-01 10:00:00+00:00       0.00000               NaN         NaN   \n",
       "2021-01-01 11:00:00+00:00       0.00000               NaN         NaN   \n",
       "2021-01-01 12:00:00+00:00       0.00000               NaN         NaN   \n",
       "2021-01-01 13:00:00+00:00       0.00372          0.016683    0.039218   \n",
       "\n",
       "                           csi_fcast  hour  month         el         zen  \\\n",
       "time_center_labeled                                                        \n",
       "2021-01-01 09:00:00+00:00        NaN     9      1 -44.725579  134.725579   \n",
       "2021-01-01 10:00:00+00:00        NaN    10      1 -32.169594  122.169594   \n",
       "2021-01-01 11:00:00+00:00        NaN    11      1 -19.832171  109.832171   \n",
       "2021-01-01 12:00:00+00:00        NaN    12      1  -7.896291   97.896291   \n",
       "2021-01-01 13:00:00+00:00   0.055901    13      1   3.619760   86.380240   \n",
       "\n",
       "                                   az  \n",
       "time_center_labeled                    \n",
       "2021-01-01 09:00:00+00:00   90.865286  \n",
       "2021-01-01 10:00:00+00:00   98.221778  \n",
       "2021-01-01 11:00:00+00:00  105.126225  \n",
       "2021-01-01 12:00:00+00:00  112.278339  \n",
       "2021-01-01 13:00:00+00:00  120.212465  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's export the full dataset as a pickle file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle('data_files/full_2021_2023_dataset.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's move on to working with our machine learning (regression) models in [09_machine_learning_models.ipynb](09_machine_learning_models.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}