tutorials/advanced/modelling/hyperparameter_optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter Optimization\n",
    "\n",
    "In this tutorial we will give examples of how to use the hyperparameter optimization functionality of QSPRpred.\n",
    "\n",
    "QSPRpred has a base class `HyperparameterOptimization` which defines the basic functionality of hyperparameter optimization. \n",
    "All subclasses of this class must implement the `optimize` method which takes a `QSPRModel` class as input and returns the best hyperparameters for the model. The optimization classes also require an `ModelAssessment` object as input which is used to assess the performance of the model, see the [model assessment tutorial](../../basics/modelling/model_assessment.ipynb) for more information.\n",
    "As well as an score aggregation function to aggregate scores returned by the assessment classes into a single score (i.e. from different folds).\n",
    "\n",
    "There are currently two hyperparameter optimization classes implemented in QSPRpred, `GridSearchOptimization` and `OptunaOptimization`.\n",
    "In this tutorial we will show you how to use both of these classes to optimize the hyperparameters of a `KNeighborsRegressor` model, but you can find their documentation [here](https://cddleiden.github.io/QSPRpred/docs/api/qsprpred.models.html#module-qsprpred.models.hyperparam_optimization).\n",
    "\n",
    "Let's first load the dataset and create the model to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.270505Z",
     "start_time": "2024-03-25T10:04:07.016567Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to find the pandas get_adjustment() function to patch\n",
      "Failed to patch pandas - PandasTools will have limited functionality\n"
     ]
    },
    {
     "data": {
      "text/plain": "                                                                                      SMILES  \\\nQSPRID                                                                                         \nHyperparamOptTutorialTutorialDataset_0000  Cc1nn(-c2cc(NC(=O)CCN(C)C)nc(-c3ccc(C)o3)n2)c(...   \nHyperparamOptTutorialTutorialDataset_0001  Nc1c(C(=O)Nc2ccc([N+](=O)[O-])cc2)sc2c1cc1CCCC...   \nHyperparamOptTutorialTutorialDataset_0002   O=C(Nc1nc2ncccc2n2c(=O)n(-c3ccccc3)nc12)c1ccccc1   \nHyperparamOptTutorialTutorialDataset_0003  CNC(=O)C12CC1C(n1cnc3c1nc(C#CCCCCC(=O)OC)nc3NC...   \nHyperparamOptTutorialTutorialDataset_0004  CCCn1c(=O)c2c(nc3cc(OC)ccn32)n(CCCNC(=O)c2ccc(...   \n...                                                                                      ...   \nHyperparamOptTutorialTutorialDataset_4077     CNc1ncc(C(=O)NCc2ccc(OC)cc2)c2nc(-c3ccco3)nn12   \nHyperparamOptTutorialTutorialDataset_4078         Nc1nc(-c2ccco2)c2ncn(C(=O)NCCc3ccccc3)c2n1   \nHyperparamOptTutorialTutorialDataset_4079              Nc1nc(Nc2ccc(F)cc2)nc(CSc2nnc(N)s2)n1   \nHyperparamOptTutorialTutorialDataset_4080      CCCOc1ccc(C=Cc2cc3c(c(=O)n(C)c(=O)n3C)n2C)cc1   \nHyperparamOptTutorialTutorialDataset_4081           CCOC(=O)c1cnc(NCC(C)C)n2nc(-c3ccco3)nc12   \n\n                                           pchembl_value_Mean    Year  \\\nQSPRID                                                                  \nHyperparamOptTutorialTutorialDataset_0000                8.68  2008.0   \nHyperparamOptTutorialTutorialDataset_0001                4.82  2010.0   \nHyperparamOptTutorialTutorialDataset_0002                5.65  2009.0   \nHyperparamOptTutorialTutorialDataset_0003                5.45  2009.0   \nHyperparamOptTutorialTutorialDataset_0004                5.20  2019.0   \n...                                                       ...     ...   \nHyperparamOptTutorialTutorialDataset_4077                7.09  2018.0   \nHyperparamOptTutorialTutorialDataset_4078                8.22  2008.0   \nHyperparamOptTutorialTutorialDataset_4079                4.89  2010.0   \nHyperparamOptTutorialTutorialDataset_4080                6.51  2013.0   \nHyperparamOptTutorialTutorialDataset_4081                7.35  2014.0   \n\n                                                                              QSPRID  \\\nQSPRID                                                                                 \nHyperparamOptTutorialTutorialDataset_0000  HyperparamOptTutorialTutorialDataset_0000   \nHyperparamOptTutorialTutorialDataset_0001  HyperparamOptTutorialTutorialDataset_0001   \nHyperparamOptTutorialTutorialDataset_0002  HyperparamOptTutorialTutorialDataset_0002   \nHyperparamOptTutorialTutorialDataset_0003  HyperparamOptTutorialTutorialDataset_0003   \nHyperparamOptTutorialTutorialDataset_0004  HyperparamOptTutorialTutorialDataset_0004   \n...                                                                              ...   \nHyperparamOptTutorialTutorialDataset_4077  HyperparamOptTutorialTutorialDataset_4077   \nHyperparamOptTutorialTutorialDataset_4078  HyperparamOptTutorialTutorialDataset_4078   \nHyperparamOptTutorialTutorialDataset_4079  HyperparamOptTutorialTutorialDataset_4079   \nHyperparamOptTutorialTutorialDataset_4080  HyperparamOptTutorialTutorialDataset_4080   \nHyperparamOptTutorialTutorialDataset_4081  HyperparamOptTutorialTutorialDataset_4081   \n\n                                           pchembl_value_Mean_original  \nQSPRID                                                                  \nHyperparamOptTutorialTutorialDataset_0000                         8.68  \nHyperparamOptTutorialTutorialDataset_0001                         4.82  \nHyperparamOptTutorialTutorialDataset_0002                         5.65  \nHyperparamOptTutorialTutorialDataset_0003                         5.45  \nHyperparamOptTutorialTutorialDataset_0004                         5.20  \n...                                                                ...  \nHyperparamOptTutorialTutorialDataset_4077                         7.09  \nHyperparamOptTutorialTutorialDataset_4078                         8.22  \nHyperparamOptTutorialTutorialDataset_4079                         4.89  \nHyperparamOptTutorialTutorialDataset_4080                         6.51  \nHyperparamOptTutorialTutorialDataset_4081                         7.35  \n\n[4082 rows x 5 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>SMILES</th>\n      <th>pchembl_value_Mean</th>\n      <th>Year</th>\n      <th>QSPRID</th>\n      <th>pchembl_value_Mean_original</th>\n    </tr>\n    <tr>\n      <th>QSPRID</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_0000</th>\n      <td>Cc1nn(-c2cc(NC(=O)CCN(C)C)nc(-c3ccc(C)o3)n2)c(...</td>\n      <td>8.68</td>\n      <td>2008.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_0000</td>\n      <td>8.68</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_0001</th>\n      <td>Nc1c(C(=O)Nc2ccc([N+](=O)[O-])cc2)sc2c1cc1CCCC...</td>\n      <td>4.82</td>\n      <td>2010.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_0001</td>\n      <td>4.82</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_0002</th>\n      <td>O=C(Nc1nc2ncccc2n2c(=O)n(-c3ccccc3)nc12)c1ccccc1</td>\n      <td>5.65</td>\n      <td>2009.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_0002</td>\n      <td>5.65</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_0003</th>\n      <td>CNC(=O)C12CC1C(n1cnc3c1nc(C#CCCCCC(=O)OC)nc3NC...</td>\n      <td>5.45</td>\n      <td>2009.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_0003</td>\n      <td>5.45</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_0004</th>\n      <td>CCCn1c(=O)c2c(nc3cc(OC)ccn32)n(CCCNC(=O)c2ccc(...</td>\n      <td>5.20</td>\n      <td>2019.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_0004</td>\n      <td>5.20</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_4077</th>\n      <td>CNc1ncc(C(=O)NCc2ccc(OC)cc2)c2nc(-c3ccco3)nn12</td>\n      <td>7.09</td>\n      <td>2018.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_4077</td>\n      <td>7.09</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_4078</th>\n      <td>Nc1nc(-c2ccco2)c2ncn(C(=O)NCCc3ccccc3)c2n1</td>\n      <td>8.22</td>\n      <td>2008.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_4078</td>\n      <td>8.22</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_4079</th>\n      <td>Nc1nc(Nc2ccc(F)cc2)nc(CSc2nnc(N)s2)n1</td>\n      <td>4.89</td>\n      <td>2010.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_4079</td>\n      <td>4.89</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_4080</th>\n      <td>CCCOc1ccc(C=Cc2cc3c(c(=O)n(C)c(=O)n3C)n2C)cc1</td>\n      <td>6.51</td>\n      <td>2013.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_4080</td>\n      <td>6.51</td>\n    </tr>\n    <tr>\n      <th>HyperparamOptTutorialTutorialDataset_4081</th>\n      <td>CCOC(=O)c1cnc(NCC(C)C)n2nc(-c3ccco3)nc12</td>\n      <td>7.35</td>\n      <td>2014.0</td>\n      <td>HyperparamOptTutorialTutorialDataset_4081</td>\n      <td>7.35</td>\n    </tr>\n  </tbody>\n</table>\n<p>4082 rows × 5 columns</p>\n</div>"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "from IPython.display import display\n",
    "\n",
    "from qsprpred.data import QSPRDataset, RandomSplit\n",
    "from qsprpred.data.descriptors.fingerprints import MorganFP\n",
    "\n",
    "os.makedirs(\"../../tutorial_output/data\", exist_ok=True)\n",
    "\n",
    "dataset = QSPRDataset.fromTableFile(\n",
    "    filename='../../tutorial_data/A2A_LIGANDS.tsv',\n",
    "    store_dir=\"../../tutorial_output/data\",\n",
    "    name=\"HyperparamOptTutorialTutorialDataset\",\n",
    "    target_props=[{\"name\": \"pchembl_value_Mean\", \"task\": \"REGRESSION\"}],\n",
    "    random_state=42\n",
    ")\n",
    "dataset.sample(500)\n",
    "\n",
    "display(dataset.getDF())\n",
    "\n",
    "# calculate compound features and split dataset into train and test\n",
    "dataset.prepareDataset(\n",
    "    split=RandomSplit(test_fraction=0.2, dataset=dataset),\n",
    "    feature_calculators=[MorganFP(radius=3, nBits=2048)],\n",
    "    recalculate_features=True,\n",
    ")\n",
    "\n",
    "dataset.getDF().head()\n",
    "\n",
    "from qsprpred.models import SklearnModel\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "os.makedirs(\"../../tutorial_output/models\", exist_ok=True)\n",
    "\n",
    "# This is an SKlearn model, so we will initialize it with the SklearnModel class\n",
    "model = SklearnModel(\n",
    "    base_dir='../../tutorial_output/models',\n",
    "    alg=KNeighborsRegressor,\n",
    "    name='HyperparamOptTutorialModel',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "This will initialize the model with default hyperparameters. In this case, the `parameters` property will be empty:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "model.parameters"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.273275Z",
     "start_time": "2024-03-25T10:04:23.271512Z"
    }
   },
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "source": [
    "However, we can check the full set of estimator's hyperparameters as well (note that this might change with the underlying model implementation, this is only true for scikit-learn models):"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "KNeighborsRegressor()",
      "text/html": "<style>#sk-container-id-1 {\n  /* Definition of color scheme common for light and dark mode */\n  --sklearn-color-text: black;\n  --sklearn-color-line: gray;\n  /* Definition of color scheme for unfitted estimators */\n  --sklearn-color-unfitted-level-0: #fff5e6;\n  --sklearn-color-unfitted-level-1: #f6e4d2;\n  --sklearn-color-unfitted-level-2: #ffe0b3;\n  --sklearn-color-unfitted-level-3: chocolate;\n  /* Definition of color scheme for fitted estimators */\n  --sklearn-color-fitted-level-0: #f0f8ff;\n  --sklearn-color-fitted-level-1: #d4ebff;\n  --sklearn-color-fitted-level-2: #b3dbfd;\n  --sklearn-color-fitted-level-3: cornflowerblue;\n\n  /* Specific color for light theme */\n  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n  --sklearn-color-icon: #696969;\n\n  @media (prefers-color-scheme: dark) {\n    /* Redefinition of color scheme for dark theme */\n    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n    --sklearn-color-icon: #878787;\n  }\n}\n\n#sk-container-id-1 {\n  color: var(--sklearn-color-text);\n}\n\n#sk-container-id-1 pre {\n  padding: 0;\n}\n\n#sk-container-id-1 input.sk-hidden--visually {\n  border: 0;\n  clip: rect(1px 1px 1px 1px);\n  clip: rect(1px, 1px, 1px, 1px);\n  height: 1px;\n  margin: -1px;\n  overflow: hidden;\n  padding: 0;\n  position: absolute;\n  width: 1px;\n}\n\n#sk-container-id-1 div.sk-dashed-wrapped {\n  border: 1px dashed var(--sklearn-color-line);\n  margin: 0 0.4em 0.5em 0.4em;\n  box-sizing: border-box;\n  padding-bottom: 0.4em;\n  background-color: var(--sklearn-color-background);\n}\n\n#sk-container-id-1 div.sk-container {\n  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n     but bootstrap.min.css set `[hidden] { display: none !important; }`\n     so we also need the `!important` here to be able to override the\n     default hidden behavior on the sphinx rendered scikit-learn.org.\n     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n  display: inline-block !important;\n  position: relative;\n}\n\n#sk-container-id-1 div.sk-text-repr-fallback {\n  display: none;\n}\n\ndiv.sk-parallel-item,\ndiv.sk-serial,\ndiv.sk-item {\n  /* draw centered vertical line to link estimators */\n  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n  background-size: 2px 100%;\n  background-repeat: no-repeat;\n  background-position: center center;\n}\n\n/* Parallel-specific style estimator block */\n\n#sk-container-id-1 div.sk-parallel-item::after {\n  content: \"\";\n  width: 100%;\n  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n  flex-grow: 1;\n}\n\n#sk-container-id-1 div.sk-parallel {\n  display: flex;\n  align-items: stretch;\n  justify-content: center;\n  background-color: var(--sklearn-color-background);\n  position: relative;\n}\n\n#sk-container-id-1 div.sk-parallel-item {\n  display: flex;\n  flex-direction: column;\n}\n\n#sk-container-id-1 div.sk-parallel-item:first-child::after {\n  align-self: flex-end;\n  width: 50%;\n}\n\n#sk-container-id-1 div.sk-parallel-item:last-child::after {\n  align-self: flex-start;\n  width: 50%;\n}\n\n#sk-container-id-1 div.sk-parallel-item:only-child::after {\n  width: 0;\n}\n\n/* Serial-specific style estimator block */\n\n#sk-container-id-1 div.sk-serial {\n  display: flex;\n  flex-direction: column;\n  align-items: center;\n  background-color: var(--sklearn-color-background);\n  padding-right: 1em;\n  padding-left: 1em;\n}\n\n\n/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\nclickable and can be expanded/collapsed.\n- Pipeline and ColumnTransformer use this feature and define the default style\n- Estimators will overwrite some part of the style using the `sk-estimator` class\n*/\n\n/* Pipeline and ColumnTransformer style (default) */\n\n#sk-container-id-1 div.sk-toggleable {\n  /* Default theme specific background. It is overwritten whether we have a\n  specific estimator or a Pipeline/ColumnTransformer */\n  background-color: var(--sklearn-color-background);\n}\n\n/* Toggleable label */\n#sk-container-id-1 label.sk-toggleable__label {\n  cursor: pointer;\n  display: block;\n  width: 100%;\n  margin-bottom: 0;\n  padding: 0.5em;\n  box-sizing: border-box;\n  text-align: center;\n}\n\n#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n  /* Arrow on the left of the label */\n  content: \"▸\";\n  float: left;\n  margin-right: 0.25em;\n  color: var(--sklearn-color-icon);\n}\n\n#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n  color: var(--sklearn-color-text);\n}\n\n/* Toggleable content - dropdown */\n\n#sk-container-id-1 div.sk-toggleable__content {\n  max-height: 0;\n  max-width: 0;\n  overflow: hidden;\n  text-align: left;\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-0);\n}\n\n#sk-container-id-1 div.sk-toggleable__content.fitted {\n  /* fitted */\n  background-color: var(--sklearn-color-fitted-level-0);\n}\n\n#sk-container-id-1 div.sk-toggleable__content pre {\n  margin: 0.2em;\n  border-radius: 0.25em;\n  color: var(--sklearn-color-text);\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-0);\n}\n\n#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n  /* unfitted */\n  background-color: var(--sklearn-color-fitted-level-0);\n}\n\n#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n  /* Expand drop-down */\n  max-height: 200px;\n  max-width: 100%;\n  overflow: auto;\n}\n\n#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n  content: \"▾\";\n}\n\n/* Pipeline/ColumnTransformer-specific style */\n\n#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n  color: var(--sklearn-color-text);\n  background-color: var(--sklearn-color-unfitted-level-2);\n}\n\n#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n  background-color: var(--sklearn-color-fitted-level-2);\n}\n\n/* Estimator-specific style */\n\n/* Colorize estimator box */\n#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-2);\n}\n\n#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n  /* fitted */\n  background-color: var(--sklearn-color-fitted-level-2);\n}\n\n#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n#sk-container-id-1 div.sk-label label {\n  /* The background is the default theme color */\n  color: var(--sklearn-color-text-on-default-background);\n}\n\n/* On hover, darken the color of the background */\n#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n  color: var(--sklearn-color-text);\n  background-color: var(--sklearn-color-unfitted-level-2);\n}\n\n/* Label box, darken color on hover, fitted */\n#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n  color: var(--sklearn-color-text);\n  background-color: var(--sklearn-color-fitted-level-2);\n}\n\n/* Estimator label */\n\n#sk-container-id-1 div.sk-label label {\n  font-family: monospace;\n  font-weight: bold;\n  display: inline-block;\n  line-height: 1.2em;\n}\n\n#sk-container-id-1 div.sk-label-container {\n  text-align: center;\n}\n\n/* Estimator-specific */\n#sk-container-id-1 div.sk-estimator {\n  font-family: monospace;\n  border: 1px dotted var(--sklearn-color-border-box);\n  border-radius: 0.25em;\n  box-sizing: border-box;\n  margin-bottom: 0.5em;\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-0);\n}\n\n#sk-container-id-1 div.sk-estimator.fitted {\n  /* fitted */\n  background-color: var(--sklearn-color-fitted-level-0);\n}\n\n/* on hover */\n#sk-container-id-1 div.sk-estimator:hover {\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-2);\n}\n\n#sk-container-id-1 div.sk-estimator.fitted:hover {\n  /* fitted */\n  background-color: var(--sklearn-color-fitted-level-2);\n}\n\n/* Specification for estimator info (e.g. \"i\" and \"?\") */\n\n/* Common style for \"i\" and \"?\" */\n\n.sk-estimator-doc-link,\na:link.sk-estimator-doc-link,\na:visited.sk-estimator-doc-link {\n  float: right;\n  font-size: smaller;\n  line-height: 1em;\n  font-family: monospace;\n  background-color: var(--sklearn-color-background);\n  border-radius: 1em;\n  height: 1em;\n  width: 1em;\n  text-decoration: none !important;\n  margin-left: 1ex;\n  /* unfitted */\n  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n  color: var(--sklearn-color-unfitted-level-1);\n}\n\n.sk-estimator-doc-link.fitted,\na:link.sk-estimator-doc-link.fitted,\na:visited.sk-estimator-doc-link.fitted {\n  /* fitted */\n  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n  color: var(--sklearn-color-fitted-level-1);\n}\n\n/* On hover */\ndiv.sk-estimator:hover .sk-estimator-doc-link:hover,\n.sk-estimator-doc-link:hover,\ndiv.sk-label-container:hover .sk-estimator-doc-link:hover,\n.sk-estimator-doc-link:hover {\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-3);\n  color: var(--sklearn-color-background);\n  text-decoration: none;\n}\n\ndiv.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n.sk-estimator-doc-link.fitted:hover,\ndiv.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n.sk-estimator-doc-link.fitted:hover {\n  /* fitted */\n  background-color: var(--sklearn-color-fitted-level-3);\n  color: var(--sklearn-color-background);\n  text-decoration: none;\n}\n\n/* Span, style for the box shown on hovering the info icon */\n.sk-estimator-doc-link span {\n  display: none;\n  z-index: 9999;\n  position: relative;\n  font-weight: normal;\n  right: .2ex;\n  padding: .5ex;\n  margin: .5ex;\n  width: min-content;\n  min-width: 20ex;\n  max-width: 50ex;\n  color: var(--sklearn-color-text);\n  box-shadow: 2pt 2pt 4pt #999;\n  /* unfitted */\n  background: var(--sklearn-color-unfitted-level-0);\n  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n}\n\n.sk-estimator-doc-link.fitted span {\n  /* fitted */\n  background: var(--sklearn-color-fitted-level-0);\n  border: var(--sklearn-color-fitted-level-3);\n}\n\n.sk-estimator-doc-link:hover span {\n  display: block;\n}\n\n/* \"?\"-specific style due to the `<a>` HTML tag */\n\n#sk-container-id-1 a.estimator_doc_link {\n  float: right;\n  font-size: 1rem;\n  line-height: 1em;\n  font-family: monospace;\n  background-color: var(--sklearn-color-background);\n  border-radius: 1rem;\n  height: 1rem;\n  width: 1rem;\n  text-decoration: none;\n  /* unfitted */\n  color: var(--sklearn-color-unfitted-level-1);\n  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n}\n\n#sk-container-id-1 a.estimator_doc_link.fitted {\n  /* fitted */\n  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n  color: var(--sklearn-color-fitted-level-1);\n}\n\n/* On hover */\n#sk-container-id-1 a.estimator_doc_link:hover {\n  /* unfitted */\n  background-color: var(--sklearn-color-unfitted-level-3);\n  color: var(--sklearn-color-background);\n  text-decoration: none;\n}\n\n#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n  /* fitted */\n  background-color: var(--sklearn-color-fitted-level-3);\n}\n</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KNeighborsRegressor()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;&nbsp;KNeighborsRegressor<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.4/modules/generated/sklearn.neighbors.KNeighborsRegressor.html\">?<span>Documentation for KNeighborsRegressor</span></a><span class=\"sk-estimator-doc-link \">i<span>Not fitted</span></span></label><div class=\"sk-toggleable__content \"><pre>KNeighborsRegressor()</pre></div> </div></div></div></div>"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.estimator"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.290942Z",
     "start_time": "2024-03-25T10:04:23.273920Z"
    }
   },
   "execution_count": 3
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "{'algorithm': 'auto',\n 'leaf_size': 30,\n 'metric': 'minkowski',\n 'metric_params': None,\n 'n_jobs': None,\n 'n_neighbors': 5,\n 'p': 2,\n 'weights': 'uniform'}"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.estimator.get_params()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.297491Z",
     "start_time": "2024-03-25T10:04:23.292098Z"
    }
   },
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can make sure which parameters were set after saving the model as well by checking the respective json files:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "('/home/sichom/projects/QSPRpred/tutorials/tutorial_output/models/HyperparamOptTutorialModel/HyperparamOptTutorialModel_meta.json',\n '/home/sichom/projects/QSPRpred/tutorials/tutorial_output/models/HyperparamOptTutorialModel/HyperparamOptTutorialModel.json')"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def load_json(file):\n",
    "    with open(file, 'r') as f:\n",
    "        return json.load(f)\n",
    "\n",
    "\n",
    "meta_file, estimator_file = model.save(save_estimator=True)\n",
    "meta_file, estimator_file"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.301602Z",
     "start_time": "2024-03-25T10:04:23.298070Z"
    }
   },
   "execution_count": 5
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "load_json(meta_file)[\"py/state\"][\"parameters\"]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.303855Z",
     "start_time": "2024-03-25T10:04:23.302248Z"
    }
   },
   "execution_count": 6
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "{'algorithm': 'auto',\n 'leaf_size': 30,\n 'metric': 'minkowski',\n 'metric_params': None,\n 'n_jobs': None,\n 'n_neighbors': 5,\n 'p': 2,\n 'weights': 'uniform'}"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_json(estimator_file)[\"params\"]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:23.306836Z",
     "start_time": "2024-03-25T10:04:23.304481Z"
    }
   },
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "source": [
    "Next, we will take a look at the hyperparameter optimization classes and how to use them."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Grid search\n",
    "Grid search is a simple hyperparameter optimization method which simply tries all combinations of hyperparameters in a grid and returns the best combination. We will first specify the grid of hyperparameters to search over."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:24.143278Z",
     "start_time": "2024-03-25T10:04:23.307368Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "qsprpred - WARNING - Random state supplied, but alg <class 'sklearn.neighbors._regression.KNeighborsRegressor'> does not support it. Ignoring this setting.\n"
     ]
    },
    {
     "data": {
      "text/plain": "{'n_neighbors': 5, 'weights': 'uniform'}"
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from qsprpred.models import GridSearchOptimization, CrossValAssessor\n",
    "\n",
    "# Define the search space\n",
    "search_space = {\"n_neighbors\": [3, 5], \"weights\": [\"uniform\"]}\n",
    "\n",
    "gridsearcher = GridSearchOptimization(\n",
    "    param_grid=search_space,\n",
    "    model_assessor=CrossValAssessor(scoring='r2'),\n",
    "    score_aggregation=np.median\n",
    ")\n",
    "gridsearcher.optimize(model, dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `optimize` method will return the best hyperparameters found for the model as determined by the used strategy. The best hyperparameters are set to the model and all subsequent fits will use these parameters unless specified otherwise. These parameters are also automatically saved to our model:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "{'n_neighbors': 5, 'weights': 'uniform'}"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_json(meta_file)[\"py/state\"][\"parameters\"]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:24.149829Z",
     "start_time": "2024-03-25T10:04:24.144241Z"
    }
   },
   "execution_count": 9
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "{'algorithm': 'auto',\n 'leaf_size': 30,\n 'metric': 'minkowski',\n 'metric_params': None,\n 'n_jobs': None,\n 'n_neighbors': 5,\n 'p': 2,\n 'weights': 'uniform'}"
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_json(estimator_file)[\"params\"]  # only the optimized hyperparameters were changed"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:24.159261Z",
     "start_time": "2024-03-25T10:04:24.151537Z"
    }
   },
   "execution_count": 10
  },
  {
   "cell_type": "markdown",
   "source": [
    "Hyperparmater optimizatin also resets the model, so we need to fit the model again before making predictions:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This KNeighborsRegressor instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.\n"
     ]
    }
   ],
   "source": [
    "from sklearn.exceptions import NotFittedError\n",
    "\n",
    "try:\n",
    "    # unable since the model is reset with the new hyperparameter setting\n",
    "    model.predict(dataset)\n",
    "except NotFittedError as e:\n",
    "    print(e)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:24.173727Z",
     "start_time": "2024-03-25T10:04:24.160091Z"
    }
   },
   "execution_count": 11
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "array([[7.154     ],\n       [8.356     ],\n       [4.712     ],\n       ...,\n       [5.87      ],\n       [4.834     ],\n       [6.92066667]])"
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# can make predictions after fitting\n",
    "model.fitDataset(dataset)\n",
    "model.predict(dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:26.096029Z",
     "start_time": "2024-03-25T10:04:24.174638Z"
    }
   },
   "execution_count": 12
  },
  {
   "cell_type": "markdown",
   "source": [
    "However, you can skip this step by setting the `refit_optimal` parameter to `True` in the `optimize` method. This will automatically fit the model with the best hyperparameters found:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "{'n_neighbors': 2, 'weights': 'uniform'}"
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_space = {\"n_neighbors\": [1, 2], \"weights\": [\"uniform\"]}\n",
    "\n",
    "gridsearcher = GridSearchOptimization(\n",
    "    param_grid=search_space,\n",
    "    model_assessor=CrossValAssessor(scoring='r2'),\n",
    "    score_aggregation=np.median\n",
    ")\n",
    "gridsearcher.optimize(model, dataset, refit_optimal=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:27.950807Z",
     "start_time": "2024-03-25T10:04:26.096971Z"
    }
   },
   "execution_count": 13
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "array([[7.57 ],\n       [8.335],\n       [4.51 ],\n       ...,\n       [5.87 ],\n       [4.98 ],\n       [6.185]])"
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:28.394743Z",
     "start_time": "2024-03-25T10:04:27.951710Z"
    }
   },
   "execution_count": 14
  },
  {
   "cell_type": "markdown",
   "source": [
    "Notice the change in output values with the new hyperparameters (the grid was changed on purpose to show the difference)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optuna\n",
    "In addition to the grid search, it is possible to use [Optuna](https://optuna.org/) to optimize the hyperparameters of a model. Optuna is a hyperparameter optimization framework which uses Bayesian optimization to find the best hyperparameters.\n",
    "\n",
    "Mainly, setting up the optimization is the same as for grid search, however, when specifiying the \n",
    "search space we need to also specify the type of each hyperparameter. This is because Optuna needs to know how to sample the hyperparameters.\n",
    "Aside from the search space, we also need to specify the number of trials to run, which is the number of different hyperparameter combinations to try. In this case, we will also use the `TestSetAssessor` to assess the performance of the model on the test set, to show the different ways of assessing the performance of the model in the optimization. \n",
    "\n",
    "We will now use Optuna to make a more thorough exploration of hyperparameters for our model:\n"
   ]
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2024-03-25 11:04:28,398] A new study created in memory with name: no-name-9759c32c-a7b9-4ac5-b024-4b1101e9941a\n",
      "[I 2024-03-25 11:04:28,512] Trial 0 finished with value: 0.6314076647146836 and parameters: {'n_neighbors': 4, 'weights': 'uniform'}. Best is trial 0 with value: 0.6314076647146836.\n",
      "[I 2024-03-25 11:04:28,629] Trial 1 finished with value: 0.6244271327283943 and parameters: {'n_neighbors': 6, 'weights': 'uniform'}. Best is trial 0 with value: 0.6314076647146836.\n",
      "[I 2024-03-25 11:04:28,734] Trial 2 finished with value: 0.486307378093249 and parameters: {'n_neighbors': 1, 'weights': 'uniform'}. Best is trial 0 with value: 0.6314076647146836.\n",
      "[I 2024-03-25 11:04:28,851] Trial 3 finished with value: 0.6297875073289022 and parameters: {'n_neighbors': 8, 'weights': 'distance'}. Best is trial 0 with value: 0.6314076647146836.\n",
      "[I 2024-03-25 11:04:28,973] Trial 4 finished with value: 0.6158780548550493 and parameters: {'n_neighbors': 9, 'weights': 'uniform'}. Best is trial 0 with value: 0.6314076647146836.\n",
      "[I 2024-03-25 11:04:29,119] Trial 5 finished with value: 0.5942176906581796 and parameters: {'n_neighbors': 2, 'weights': 'distance'}. Best is trial 0 with value: 0.6314076647146836.\n",
      "[I 2024-03-25 11:04:29,242] Trial 6 finished with value: 0.637380168234392 and parameters: {'n_neighbors': 5, 'weights': 'distance'}. Best is trial 6 with value: 0.637380168234392.\n",
      "[I 2024-03-25 11:04:29,357] Trial 7 finished with value: 0.5942176906581796 and parameters: {'n_neighbors': 2, 'weights': 'distance'}. Best is trial 6 with value: 0.637380168234392.\n",
      "[I 2024-03-25 11:04:29,484] Trial 8 finished with value: 0.6306764992899814 and parameters: {'n_neighbors': 5, 'weights': 'uniform'}. Best is trial 6 with value: 0.637380168234392.\n",
      "[I 2024-03-25 11:04:29,613] Trial 9 finished with value: 0.6244271327283943 and parameters: {'n_neighbors': 6, 'weights': 'uniform'}. Best is trial 6 with value: 0.637380168234392.\n"
     ]
    },
    {
     "data": {
      "text/plain": "{'n_neighbors': 5, 'weights': 'distance'}"
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from qsprpred.models import OptunaOptimization, TestSetAssessor\n",
    "\n",
    "# Note the specification of the hyperparameter types as first item in the list\n",
    "search_space = {\"n_neighbors\": [\"int\", 1, 10],\n",
    "                \"weights\": [\"categorical\", [\"uniform\", \"distance\"]]}\n",
    "\n",
    "# Optuna gridsearcher with the TestSetAssessor\n",
    "gridsearcher = OptunaOptimization(\n",
    "    n_trials=10,\n",
    "    param_grid=search_space,\n",
    "    model_assessor=TestSetAssessor(scoring='r2'),\n",
    ")\n",
    "gridsearcher.optimize(model, dataset, refit_optimal=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:30.768719Z",
     "start_time": "2024-03-25T10:04:28.395571Z"
    }
   },
   "execution_count": 15
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "array([[7.2       ],\n       [8.07      ],\n       [4.8       ],\n       ...,\n       [5.99714729],\n       [4.95978074],\n       [6.70326748]])"
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(dataset)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-25T10:04:31.223958Z",
     "start_time": "2024-03-25T10:04:30.769415Z"
    }
   },
   "execution_count": 16
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}