Assignments/3dasm_Homework8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03ce6c2e",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1a005b0c576ca69814e142351112a584",
     "grade": false,
     "grade_id": "cell-bce01a551d8dabe6",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# ENGN2350 Data-Driven Design and Analysis of Structures and Materials\n",
    "\n",
    "Coding exercises to explore the [`f3dasm`](https://f3dasm.readthedocs.io/en/latest/) package.\n",
    "\n",
    "**General instructions**:\n",
    "\n",
    "- Read the questions and answer in the cells under the \"PUT YOUR CODE IN THE CELL BELOW\" message.\n",
    "- Work through the notebook and make sure you fill in any place that says `YOUR CODE HERE` or `YOUR ANSWER HERE`. You can remove the `'raise NotImplementedError()'` code.\n",
    "- After \"END OF YOUR CODE\" , there is a cell that contains simple tests (with `assert` statements) to see if you did the exercises correctly. Not all exercises have tests! If you run the cell containing the tests and no error is given, you have succesfully solved the exercise!\n",
    "- Make sure you have the right version of `f3dasm` (1.5.4). The cell below checks this:\n",
    "\n",
    "- **ONLY WORK ON THE EXERCISE IN A JUPYTER NOTEBOOK ENVIRONMENT**\n",
    "\n",
    "> The homework assignments are generated and automatically graded by the `nbgrader` extension. If you open and save the notebook in Google Colab, metadata from Colab will be added, and the `nbgrader` metadata will be altered. As a result, `nbgrader` will be unable to automatically grade your homework. Therefore, we kindly ask students to only work on the notebook in Jupyter Notebook.\n",
    "\n",
    "- **DO NOT ADD OR REMOVE CELLS IN THE NOTEBOOK**\n",
    "\n",
    "> Most cells containing tests are set to read-only, but VS Code can bypass this restriction. Modifying or removing cells in the notebook may disrupt the `nbgrader` system, preventing automatic grading of your homework.\n",
    "\n",
    "**Instructions for handing in the homework**\n",
    "\n",
    "- Upload the Jupyter Notebook (`.ipynb file`) to canvas\n",
    "\n",
    "If there are any questions about the homework, send Martin an email (m.p.vanderschelling@tudelft.nl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28879bef",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "842be57f6f7a2576703ef499c8ecd970",
     "grade": false,
     "grade_id": "cell-4a9d48a9aa0178c3",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "We make sure you have the right version of `f3dasm` (1.5.4). The cell below checks this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ce61033",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "eb77065df8e10baff6decc95a6c1bcd3",
     "grade": false,
     "grade_id": "cell-2885211d7254cac3",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import f3dasm\n",
    "import IPython\n",
    "\n",
    "# Check if the f3dasm version is correct\n",
    "assert f3dasm.__version__ == '1.5.4', \"Your version of f3dasm is incorrect, please update it.\"\n",
    "\n",
    "# Check if the IPython version is at least 3\n",
    "assert IPython.version_info[0] >= 3, \"Your version of IPython is too old, please update it.\"\n",
    "\n",
    "# If no assert statements are triggered, print a succes message\n",
    "print(\"Your environment is set up correctly :)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "271d1df1",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "40ada260916a8fcb3dcb0a68252395fa",
     "grade": false,
     "grade_id": "cell-29cde56aff653320",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "You can put your name in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f8bba8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c6701c4",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "45c2bf59ec09d762fd2d607eb147c1a4",
     "grade": false,
     "grade_id": "cell-925719e94f410246",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39dac51d",
   "metadata": {},
   "source": [
    "## Homework 8\n",
    "\n",
    "In this homework you will explore advanced usage of the `f3dasm` package\n",
    "\n",
    "At the end of this homework you will know\n",
    "- how to use the built-in defaults of `f3dasm`\n",
    "- how to evaluate different sampling techniques in a structured way with `f3dasm`\n",
    "- how to use `f3dasm` for hyperparameter analysis for classification methods\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "662e4e94-5cce-4bb2-adcd-9aa1b90f89c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import some packages we might need later\n",
    "import numpy as np\n",
    "import f3dasm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ebffe34-6199-459f-a2a7-03eaa8d54b83",
   "metadata": {},
   "source": [
    "---\n",
    "## Exercise 1\n",
    "\n",
    "We will explore the default built-in implementation that are available in `f3dasm`.\n",
    "You can learn more about those in the [documentation page](https://f3dasm.readthedocs.io/en/latest/rst_doc_files/defaults.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fc8ab0f-44df-43dc-9dd3-e71bfdc1ff79",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "1.1 Create a 3D continuous Domain with bounds $-32.768$ and $32.768$ with the `make_continuous_domain` helper function and name the domain `domain_3d`:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8ee3b05-d62e-4277-8909-a557df356dd7",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4971256-e655-4aaf-aa28-dfb8a8392c0e",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e433bd7e220ddfd8ff9bd5584e68e31e",
     "grade": false,
     "grade_id": "cell-774c951f89dc9144",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b64eae4-dde6-458b-bc07-9fcdbdcb41cc",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a82595b8-7c53-4a7c-87b7-b8700de4cf06",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8d2eecb5b39d0b6ba2512b9c51628d1a",
     "grade": true,
     "grade_id": "cell-6a5f23068e8cd549",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly\n",
    "assert all(key in domain_3d.space for key in ['x0', 'x1', 'x2']), \"Your domain does not contain the right keys\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bb355f8-9e4e-48cf-b649-12c11e3f196b",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "1.2 Use the built-in latin hypercube sampler to create 20 design points and name the result `experimentdata_3d`. Set the seed to $123$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27c6fc45-fed8-4fcf-b0c3-d8288f70717b",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36666806-787e-4afe-a1f7-5af9193b1fd9",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0b8cd353ae23e8a8c6926ceca6fcb4b6",
     "grade": false,
     "grade_id": "cell-a3aa84fe820341bc",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04990284-ca00-4fa1-897c-0aa1bc59ac6c",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29eb3f47-a85b-4b6c-9d49-11006a9593bb",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "62b7640fffedd43ef89eb7ff7f1a1985",
     "grade": true,
     "grade_id": "cell-ee38064be7fb8704",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly\n",
    "assert np.allclose(experimentdata_3d.to_numpy()[0][0], [-30.48581 , -19.465248,  15.945922]), \"Your samples are not as expected, perhaps you didn't include the seed?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6e5992d-2618-4dc0-a4e8-3ed7816a40b0",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "1.3 Plot all the points in 2D scatter plots for all possible combinations of the input features, i.e. every pair of features as $x$ and $y$ of the scatter plots. Observe the sampling points and how they are distributed in the domain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e690f8-bc8f-424b-8e86-f60a24a039da",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bde2cff4-df60-4526-aee3-8763d4cf7033",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "213f4201ada0ac200630e56dfec5632c",
     "grade": false,
     "grade_id": "cell-cb0c4ac287b6d5c8",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe760f3-437e-4838-8380-24d1f65c4348",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accf6fec-ade2-4696-a699-7f15baaabc81",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "1.4 Evaluate all the designs with the built-in ['Ackley'](https://www.sfu.ca/~ssurjano/ackley.html) function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0f9adb3-533e-40b0-953d-16fb09f01634",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "359a7387-9883-48f9-af00-4869dfee88e0",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d2609c65acf83d0557ded41f8c17982e",
     "grade": false,
     "grade_id": "cell-ab743eef1e21a8de",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0debfb-4853-454c-9475-5b4bba468e28",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6a3e3ad-998a-4864-b2c5-87a8cb01b3cc",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3df5b7c7589ecbcbc4e6d9d66c53a77f",
     "grade": true,
     "grade_id": "cell-86be587e6a8eeb72",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly\n",
    "assert np.allclose(experimentdata_3d.to_numpy()[1][0], [21.800436]), \"Your first design does not contain the correct output value\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718b5d7d-d017-4989-a9fa-e1bd27c6f9b2",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "1.5 Optimize this experiment with the built-in Nelder-Mead optimizer for $50$ iterations:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf177976-1485-4f0a-802b-d2879626e264",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "120996b7-abde-42dd-bd46-a88cc290b67f",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "14ad9e5a399a02820478866fe2feec55",
     "grade": false,
     "grade_id": "cell-11fbcb8176d0e72f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4c0540b-0709-414e-ac92-1cfe5e3b3fb4",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e824291-5c9f-4fca-bfb4-33ff18660b36",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d119380eca6e0156aaf2abdab9b9231e",
     "grade": true,
     "grade_id": "cell-42ae8f34ec835be0",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly\n",
    "assert np.allclose(experimentdata_3d.to_numpy()[0][-1], [7.9844937, 4.9903407, 3.992201]), \"Your last iteration does not contain the correct output value\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "702cc428",
   "metadata": {},
   "source": [
    "---\n",
    "### Exercise 2\n",
    "\n",
    "In this next exercise, we are going to train a gaussian process regressor with data from built-in benchmark functions.\n",
    "We are going to vary the benchmark function used, the sampling technique and the number of training points\n",
    "\n",
    "Before we start exploring, we want to create a dataset with the built-in benchmark functions.\n",
    "\n",
    "We would like to:\n",
    "- .. create a 1D continuous domain with bounds $[-1, 1]$ ;\n",
    "- .. sample $N$ points with a given sampler, using $123$ as the seed for the random number generator;\n",
    "- .. evaluate the samples on a given benchmark function, with scaled bounds to our $[-1, 1]$ domain, no offset and `noise=0.1`\n",
    "\n",
    "2.1 Make a function `create_data` that inputs the type of built-in sampler, the number of sampling points and the built-in benchmark function be evaluated. The function returns a `f3dasm.ExperimentData` object with those samples, evaluated by the benchmark function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "808d8047-e3bd-4543-bc14-d9a571e97177",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b4e3ba8",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6b46ecf4339088d1c95adaa39201bba5",
     "grade": false,
     "grade_id": "cell-24039e8ef0107f0f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n",
    "def create_data(n_samples: int, sampler: str, benchmark_fn: str) -> f3dasm.ExperimentData:\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return experiment_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0471a139-f989-4f81-96fc-445571fca2b9",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f1533b3-d062-4b7f-93d2-ba3bb8a16b08",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a6296b9022256adb6a791366dfcb271f",
     "grade": true,
     "grade_id": "cell-d6394cee6b2e698d",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly \n",
    "import numpy as np\n",
    "data_rastrigin = create_data(n_samples=100, sampler='random', benchmark_fn='rastrigin')\n",
    "data_ackley = create_data(n_samples=30, sampler='latin', benchmark_fn='ackley')\n",
    "# data_ackley.to_numpy()[1][0][0], data_rastrigin.to_numpy()[0][0][0]\n",
    "assert np.isclose(data_rastrigin.to_numpy()[0][0][0], 0.3647037), \"Your create_data function does not output the correct values for the Rastrigin function!\"\n",
    "assert np.isclose(data_ackley.to_numpy()[1][0][0], 15.78891), \"Your create_data function does not output the correct values for the Ackley function!\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16c25fd4",
   "metadata": {},
   "source": [
    "---\n",
    "2.2 Next, we are going to create a helper function that trains the model and returns error metrics in order to evaluate the predictions\n",
    "\n",
    "\n",
    "Create a function `evaluate_regressor` that requires the training data, `X_train` and `y_train`, the testing data `X_test` and `y_test` and the scikit-learn model `model`\n",
    "\n",
    "The function should :\n",
    "\n",
    "- use the `model.fit` function to fit `X_train` and `y_train`\n",
    "- predict on `X_test` to create `y_pred`\n",
    "- calculate the $R^2$ and MSE on the testing data and return the two metrics as a tuple of two floats\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0c23c74-ef0c-4a1a-ab12-d16934e86248",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3581b5e1-eb87-44ae-a04e-19bf48ec6993",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c262a39fdd637e5e48e8d04134231d48",
     "grade": false,
     "grade_id": "cell-e13cbe76511613bd",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n",
    "\n",
    "def evaluate_regressor(X_train: np.ndarray, y_train: np.ndarray, X_test: np.ndarray, y_test: np.ndarray, model):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return r2, mse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aadbbc7-75f2-4a24-b47e-eeee700ef4da",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efa364e2-4c22-4aec-9639-fb8f299db4b3",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e3bb47d6edd24e27e3e5b77fb803588f",
     "grade": true,
     "grade_id": "cell-a2a603d59fda7c19",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly \n",
    "import numpy as np\n",
    "from sklearn.gaussian_process import GaussianProcessRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = data_rastrigin.to_numpy()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=123)\n",
    "model = GaussianProcessRegressor(random_state=123)\n",
    "r2, mse = evaluate_regressor(X_train, y_train, X_test, y_test, model)\n",
    "assert np.isclose(r2, 0.6298680), \"Your value of R2 is not as expected\"\n",
    "assert np.isclose(mse, 39.56785), \"Your value of MSE is not as expected\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4a4e7e2-bdc5-4aeb-ad1d-a34669851809",
   "metadata": {},
   "source": [
    "---\n",
    "2.3 Now we are putting everything together in a custom `DataGenerator` class:\n",
    "\n",
    "1. Create a new `Domain` object with 3 parameters:\n",
    "\n",
    "- `n_samples`, the number of training samples, as a categorical variable with possible values $6$, $11$ and $21$\n",
    "- `sampler`, the chosen built-in sampler, as a categorical variable with possible values `'random'`, `'sobol'` and `'latin'`\n",
    "- `benchmark_fn`, the chosen built-in benchmark function, as a categorical variable with possible values `'Styblinski Tang'`, `'Branin'` and `'Sphere'`\n",
    "\n",
    "2. Create an `ExperimentData` object from the domain and use the built-in `'grid'` sampler to create all possible combinations of `n_samples`, `sampler` and `benchmark_fn`\n",
    "> Hint: you should have an ExperimentData object with 27 different experiments\n",
    "\n",
    "3. Create a `EvaluateRegressor` class that inherits from `f3dasm.datageneration.DataGenerator`\n",
    "- Inside the `execute()` method:\n",
    "  - extract the values `n_samples` (type=`int`), `sampler` (type=`str`) and `benchmark_fn` (type=`str`) from the `self.experiment_sample` object\n",
    "  - Create `X_train` and `y_train` by calling the `create_data` function with the extracted values created earlier\n",
    "  - Create `X_test` and `y_test` by calling the `create_data` function, but set `n_samples=100` and `sampler='random'`\n",
    "  - Create a GaussianProcessRegressor from `sklearn`\n",
    "  - Call the `evaluate_regressor` function with the training data, testing data and GPR model\n",
    "  - Store the $R^2$ and the MSE values in the `self.experiment_sample` with names `r2` and `mse`\n",
    "\n",
    "4. Create an instance of the `EvaluateRegressor` class and `evaluate` the `ExperimentData` object.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "927ac6b8-1c0f-45c8-8b29-12ee78cc0b1d",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dcae2d6-c64b-482b-8c9e-a24aa0e88ea3",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "48cd2bb58b75697fe689443cb0e68634",
     "grade": false,
     "grade_id": "cell-e7bde3c97b6af6e5",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# You might want to import some classes/functions\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01cc110e-4880-450e-a88a-c3b9e9fd0790",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3a999d0",
   "metadata": {},
   "source": [
    "---\n",
    "1.3 Reflect on your findings? Can you explain why certain the performance differs when changing the number of training points and the sampling technique?\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddab0946-da85-4217-973f-48c5545ddf66",
   "metadata": {},
   "source": [
    "PUT YOUR WRITTEN ANSWER IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfa9b28f-a91b-493c-b6fc-6208437e784f",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "bc10f5e9317cbba3de54b0357f33c59c",
     "grade": true,
     "grade_id": "cell-512054b2cf22d10e",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf0fe3d-717d-4b2d-9da7-9ac84e13cc4d",
   "metadata": {},
   "source": [
    "END OF YOUR WRITTEN ANSWER"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23cbb289",
   "metadata": {},
   "source": [
    "---\n",
    "### Exercise 3\n",
    "\n",
    "We can also do hyperparameter investigation effectively with `f3dasm`.\n",
    "\n",
    "For the following exercise, we are going to investigate the influence of the regularization parameter $C$ of the [Suppor Vector Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) on the [iris dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html#sklearn.datasets.load_iris).\n",
    "\n",
    "3.1 Load the iris dataset from `scikit-learn`, select only the first two feature ('sepal length (cm)' and 'sepal width (cm)') and split the data in a 80/20 test and training set. Set the seed for the random number generator to $123$. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "799771ad-f668-4034-a66c-13745d65891c",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19cb96cc-e9c5-425d-98ef-f3ca2eb2adc9",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c6f57aa8eed14cc29e29e6efa7d1cdfd",
     "grade": false,
     "grade_id": "cell-45c087179d31db6c",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24974bcd-4d0c-4dfe-9792-7316e8c6e37d",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c60b3b3a",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c5aaf5a94f7d6e2040d7ec9c6df90541",
     "grade": true,
     "grade_id": "cell-b29f46c87533b824",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly\n",
    "assert (X_train[0] == [5.4, 3.9]).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9604508-5c09-4929-ba67-4a50a8a1106e",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "3.2 Create a `Domain` with parameter `C` and construct an `ExperimentData` object with `Data_c` (given below)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac682d39-89fc-4eec-ad6e-066e4c365cca",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea1a5abd-a427-47e0-ad30-7edcd56d6ff3",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8ed0e4d132b5468939f4ef40e1664196",
     "grade": false,
     "grade_id": "cell-8a1d06cfd64eadc6",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "Data_c = np.logspace(-2.3, 2.5, 40)\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8663e8d9-e39e-4b1e-9d39-83a5e5e4de7c",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4eac42-462a-4b85-9e93-9355c0e283e1",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "3.3 Fill in the `evaluate_classifier` method and `execute` function of the `EvaluateClassifier` class below:\n",
    "- The `evaluate_classifier` takes in the model and fits it to `self.X_train`, `self.y_train`, predicts the labels of `self.X_test` and returns the accuracy of the predictions\n",
    "- The `execute` function extract the `C` parameter from `self.experiment_sample`, creates the Support Vector Classifier with the extracted $C$ value and `random_state=123`, calls the `evaluate_classifier` method and stores the accuracy value back to the `self.experiment_sample`.\n",
    "  \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fcb03a9-d15e-4fed-8b76-575fafd71081",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95a20a36-6223-4153-9219-56a43849c385",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6902fd640dd597875f35ccdd79e19726",
     "grade": false,
     "grade_id": "cell-1eb4fea34663618d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n",
    "\n",
    "class EvaluateClassifier(f3dasm.datageneration.DataGenerator):\n",
    "    def __init__(self, X_train, y_train, X_test, y_test):\n",
    "        self.X_train = X_train\n",
    "        self.y_train = y_train\n",
    "        self.X_test = X_test\n",
    "        self.y_test = y_test\n",
    "        \n",
    "    def evaluate_classifier(self, model) -> float:\n",
    "        # YOUR CODE HERE\n",
    "        raise NotImplementedError()\n",
    "        return accuracy\n",
    "    \n",
    "    def execute(self) -> None:\n",
    "        # YOUR CODE HERE\n",
    "        raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc405024-76b6-44a6-8536-ab1280dd3c95",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf87de80-a37e-4a54-bf61-8627d57259c9",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "3.4 Evaluate all the experiments and plot the accuracy w.r.t. the regularization parameter $C$\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa590fd6-5ce3-4d35-aa94-f7ee3eb86999",
   "metadata": {},
   "source": [
    "PUT YOUR CODE IN THE CELL BELOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd06e0e6-2b12-411e-a1a8-675854a98e95",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "85067f36bc08258b00fa2f877feb6b0c",
     "grade": false,
     "grade_id": "cell-6761b8ab399ff28a",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fe1f3c0-a942-4748-8fa4-8b7fbc5c1b03",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "448ec6e94f7318340c957da265c41b02",
     "grade": true,
     "grade_id": "cell-8fe04d344559cb22",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# This cell checks if you did the exercise correctly\n",
    "_, y =edata.to_numpy()\n",
    "assert np.isclose(y, np.array([[0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.21052632],\n",
    "       [0.47368422],\n",
    "       [0.6315789 ],\n",
    "       [0.7368421 ],\n",
    "       [0.7894737 ],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.8684211 ],\n",
    "       [0.8684211 ],\n",
    "       [0.8684211 ],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.84210527],\n",
    "       [0.8684211 ],\n",
    "       [0.8684211 ],\n",
    "       [0.81578946],\n",
    "       [0.81578946],\n",
    "       [0.81578946],\n",
    "       [0.81578946],\n",
    "       [0.81578946],\n",
    "       [0.81578946],\n",
    "       [0.81578946]])).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea2bb6d-5b0f-4694-8939-78af9742977b",
   "metadata": {},
   "source": [
    "END OF YOUR CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36a22637-9a4b-4baf-83a9-e2b20dfede9b",
   "metadata": {},
   "source": [
    "End of the homework!\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:3dasm]",
   "language": "python",
   "name": "conda-env-3dasm-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}