Docs: Update hyperparameter plugin docs

docs/plugins/ablation_paths.rst

@@ -6,10 +6,31 @@ Starting from a default configuration, the default configuration is iteratively
 incumbent configuration by changing one hyperparameter at a time, choosing the
 hyperparameter that leads to the largest improvement in the objective function at each step.
 
+This plugin is capable of answering following questions:
+
+* How does each hyperparameter impact the objective function when transitioning from the default to the incumbent configuration?
+* Which hyperparameters lead to the most significant improvements in performance during the iterative optimization process?
+* Are there specific hyperparameters that are consistently crucial for performance improvements, regardless of the budget?
+
 To learn more about Ablation Paths, please see the paper
 `Efficient Parameter Importance Analysis via Ablation with Surrogates
 <https://doi.org/10.1609/aaai.v31i1.10657>`_.
 
+Options
+-------
+* **Objective**: Select the objective function you wish to analyze.
+
+* **Trees:** Specify the number of trees for the random forest surrogate model used in calculating importance scores.
+
+To refine your analysis, you can apply filters after calculation:
+
+* **Limit Hyperparameters**: Show only the top N most important hyperparameters.
+
+* **Show Confidence**: Selection of whether to display confidence estimates based on the surrogate model's uncertainty estimates.
+
+* **Budgets**: Filter the results to view importance scores specific to certain multi-fidelity budgets, allowing you to analyze how hyperparameter importance varies with budget changes.
+
+
 .. note::
     If a plot is not showing for a specific budget, you might have to evaluate more configurations
     from your configuration space in order to improve the performance of the surrogate model.

docs/plugins/configuration_cube.rst

@@ -1,38 +1,31 @@
 Configuration Cube
 ==================
 
-The configuration cube displays configurations and their score on the objective.
+The configuration cube displays configurations and their objective scores.
 Naturally, the space of configurations may be very high dimensional and so this plugins allows you
 to investigate the configurations through 2D or 3D *slices* of the configuration space,
 plotting the values of hyperparameters as points in this *slice* of the space.
 
-* **Objective**: Here you can select the objective you wish to analyze for.
-  The score for an objective will be communicated through the color of a point with an associated
-  heatmap.
+* **Objective**: Select the objective function you wish to analyze. The objective scores will be
+  represented using a color scale.
 
-* **Budget**: Here you can specify a filter for which budgets you wish to view.
-  The *Combined* budget will show all configurations trialed but only show their scores on the
-  highest budget if it was evaluated under multiple different budgets.
+* **Budget**: Select the multi-fidelity budget to be used. The plugin will only consider trials evaluated
+  on the selected budget. The *Combined* budget option displays all configurations but shows scores only
+  from the highest budget if a configuration was evaluated with multiple budgets.
 
 * **Number of Configurations**: This slider will allow you to select the number of configurations to
-  display, where the configurations are ordered by they when they were trialed.
-  You can move this along to see how the optimizer chose to evaluate certain hyperparamters over
-  time.
-
-* **Hyperparameters**: Here you can select the *slice* with which to look into the configuration
-  space. Selecting only one hyperparameter gives you a very basic 1D view, 2 Hyperparameters a 2d
-  grid and 3 hyperparamters a 3D cube.
-  If multiple configurations map to the same point then the one with the highest score for the
-  chosen objective is displayed. This is particular evident in the 1D case where you may have many
-  configurations use the same hyperparameter value.
+  display, where the configurations are ordered by they when they were trialed. Adjusting this slider
+  helps to visualize how the optimizer's focus on hyperparameters has evolved over time.
 
+* **Hyperparameters**: Choose the slice of the configuration space to visualize. Selecting:
+        * **One hyperparameter** provides a basic 1D view, where configurations are plotted along a single axis.
+        * **Two hyperparameters** generates a 2D grid view, showing configurations in a plane.
+        * **Three hyperparameters** creates a 3D cube view, plotting configurations in three dimensions. In cases where multiple configurations map to the same point, the one with the highest score for the selected objective is displayed. This is particularly noticeable in the 1D view, where many configurations may share the same hyperparameter value.
 
 The Configuration Cube is helpful to visually inspect and answer the following questions:
 
-* Does a certain hyperparameter have high influence over the final objectives score when compared to
-  another?
-* Does the optimizer tend to prefer one value of a hyperparameter over time?
-* How do hyperparamters correlate with respect to the given objective
+* How does the influence of one hyperparameter on the objective score compare to others?
+* Does the optimizer show a preference for specific values of hyperparameters over time?
 
 
 .. image:: ../images/plugins/configuration_cube.png

docs/plugins/importances.rst

@@ -1,20 +1,60 @@
 Importances
 ===========
 
-Hyperparameter importance tells you how large the influence of single hyperparameters are.
+The Importances plugin provides insights into how individual hyperparameters influence the
+objective performance of your models.
+It generates a bar plot displaying the importance scores for each hyperparameter.
+
 This plugin is capable of answering following questions:
 
-* Which hyperparameters are the most important?
-* How much differs the importance between the budgets?
+* Which individual hyperparameters have the greatest impact on the objective performance?
+* How does the importance of each hyperparameter vary across different budgets?
+* How does the local parameter importance compare to the global importance as determined by fANOVA?
 
+Importance Methods
+------------------
 
-.. warning::
-    This page is under construction.
+**Local Parameter Importance** (LPI) focuses on understanding the impact of individual
+hyperparameters in the context of a specific configuration (here, the incumbent). Leveraging a
+random forest surrogate model, it calculates the variance in performance caused by changes in each
+hyperparameter, and expresses this as a fraction of the total variance.
+This method is especially useful for pinpointing which parameters are crucial in the context of the
+incumbent their local effects. For more details on LPI, please see the paper
+`CAVE: Configuration Assessment, Visualization and Evaluation
+<https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/18-LION12-CAVE.pdf>`_.
+
+**fANOVA Importance** Functional ANOVA (fANOVA) evaluates hyperparameter importance by decomposing
+the performance variance into contributions from individual hyperparameters and their interactions.
+It uses a surrogate model, such as a random forest, to predict performance across different
+hyperparameter configurations, and then analyzes how much each hyperparameter and its interactions
+contribute to the overall variance in performance. This method provides a comprehensive view of
+hyperparameter importance across the entire configuration space, thus helping in understanding the
+global impact of hyperparameters. For more details on fANOVA, please refer to the paper
+`An Efficient Approach for Assessing Hyperparameter Importance
+<https://proceedings.mlr.press/v32/hutter14.html>`_.
+
+
+Further Options
+---------------
+
+Before calculation, besides the importance method there are two more options you can set:
+
+* **Objective:** Select the objective function you wish to analyze.
+
+* **Trees:** Specify the number of trees for the random forest surrogate model used in calculating importance scores.
+
+To refine your analysis, you can apply various filters after calculation:
+
+* **Hyperparameters:** Display only the importance scores for specific hyperparameters of interest.
+
+* **Limit Hyperparameters**: Show only the top N most important hyperparameters.
 
+* **Budgets**: Filter the results to view importance scores specific to certain multi-fidelity budgets, allowing you to analyze how hyperparameter importance varies with budget changes.
 
 .. image:: ../images/plugins/importances.png
 
 .. warning::
-    As the fANOVA implementation relies on pyrfr and pyrfr cannot be applied to constant hyperparameters,
-    as a workaround we remove constant hyperparameters before calculation.
-    This will break if there are conditions or forbiddens including constant hyperparameters.
+    As the fANOVA implementation relies on pyrfr, which cannot be applied to constant hyperparameters.
+    As a workaround, we remove constant hyperparameters before calculation.
+    This will break if the configuration space contains any conditions or forbiddens including
+    constant hyperparameters.

docs/plugins/parallel_coordinates.rst

@@ -1,62 +1,60 @@
 Parallel Coordinates
 ====================
 
-With parallel coordinates, you can see configurations plotted as a line through their hyperparameter
-values and to which final score they reach.
-You can use this to identify trends in hyperparamter value ranges that achieve certain scores.
-For example, you may find that high performing configurations may all share the same value for a
-certain categorical hyperparamter, as indicated by many high scoring lines passing through this
-value. These lines will be ordered according to their importance in determining the final score from
-left to right with the default to show the top 10 important ones.
-Lastly, you can place visual filters along these spines to really home-in on hyperparamters
-important for your analysis.
+The Parallel Coordinates plugin visualizes hyperparameter configurations as a line through their
+hyperparameter values and their final objective score.
+This visualization helps in identifying trends and patterns related to the final scores achieved
+by various configurations.
+For example, you may find that high performing configurations all share the same value for a
+certain categorical hyperparameter, as indicated by many high scoring lines passing through this
+value.
+The lines can be ordered according to their fANOVA importance from right to left
+(i.e. more important hyperparameters are closer to the objective score on the right).
 
 This can help provide insight and answers to various questions:
 
-* What hyperparamter values, or combinations of them, lead to a range of final scores?
-* Which hyperparamter values does the optimizer tend to prefer?
-* What hyperparamter configurations lead to crash configurations?
-
+* Which hyperparameter values or combinations are associated with high or low final scores?
+* What preferences does the optimizer exhibit for certain hyperparameters?
 
 Options
 -------
 
 * **Objective**: Here you can select the objective you wish to analyze for.
-  The score will be placed as the last bar along the x-axis along with incoming lines indication
-  which configuration achieved what score.
+  The final scores are depicted as the last item along the x-axis.
+
+* **Budget**: Select the multi-fidelity budget to be used. The plugin will only consider trials evaluated
+  on the selected budget. The *Combined* budget option displays all configurations but shows scores only
+  from the highest budget if a configuration was evaluated with multiple budgets.
+
+* **Show Important Hyperparameters**: Order hyperparameters according to their fANOVA importance.
+  The more right a hyperparameter stands, the more important it is. Activating this option might
+  take longer.
 
-* **Budget**: Here you can specify a filter for which budgets you wish to view.
-  The *Combined* budget will show all configurations trialed but only show their scores on the
-  highest budget if it was evaluated under multiple different budgets.
+To refine your analysis, you can apply filters after calculation:
 
-* **Show Important hyperparameters**: With this, you can limit the number of spines in the plot to
-  just the top 10 most important hyperparamters, calculated using fANOVA with 5 trees.
-  These will be arranged left to right according to their importance.
-  You can select *No* and instead show **all** hyperparamters, selecting those you consider most
-  important for you.
+* **Limit Hyperparameters**: Shows either the N most important hyperparameters (if show important
+  hyperparameters is true) or the first N selected hyperparameters.
+
+* **Hyperparameters**: Shows only the selected hyperparameters (only available if "Show Important
+  Hyperparameters" is false).
 
 * **Show Unsuccessful Configurations**: By default, we do not crowd the plot with configurations that
-  did not manage to complete but you may enable plotting them by changing this to *Yes*.
-  This could be useful to identify hyperparamters or combinations of them that caused them to not
-  complete.
+  did not manage to complete but you can enable plotting them by changing this to *Yes*.
+  This can be useful to identify hyperparameters or combinations of them that caused the trial to
+  crash.
 
 
 Using the plot
 --------------
-The plot can be overwhelming at first but it is quite dynamic and easy to filter.
-
-One of the most useful features to know is that you can draw a bar on a spine to add a visual filter
-to all of the configuration lines.
-Doing so on the objective, the right most spine, can help filter configurations that achieve a
-certain score range. You can place as many of these as you like and clicking on them again to remove
-them.
-
-You can further re-arrange the spines by dragging their label above the spine left or right.
-Doing so allows you to group together relevant hyperparamters and view configurations in a way that
-makes sense for the question or problem at hand.
+The plot offers some interactive features for in-depth analysis:
 
-Sometimes, well-performing configurations tend to use similar hyperparameters.
-This plugins let you analyze this behaviour in detail.
+* **Visual Filters:** Add visual filters by drawing lines on the hyperparameter or objective value
+  ranges. on the axes. For instance, placing a line on the objective value range allows you to
+  filter configurations within that specific range. Multiple filters can be applied, and clicking
+  on them again will remove them.
 
+* **Re-arranging:** You can drag and reposition the hyperparameters and the objective to better
+  group and compare them. This feature helps in customizing the view to focus on relevant hyperparameters
+  for your analysis.
 
 .. image:: ../images/plugins/parallel_coordinates.png

docs/plugins/partial_dependencies.rst

@@ -18,21 +18,29 @@ This plugin is capable of answering the following questions:
 
 * How does the objective change with respect to one or two hyperparameters? For example, does the
   accuracy increase if the learning rate decreases?
-* Do multiple trials show similar behavior?
+* What are the regions of the hyperparameter space that lead to better or worse performance?
 
 Options
 -------
-* **Objective**: Here you can select the objective you wish to analyze. In the one hyperparameter case the objective is plotted on the y-axis, in the two hyperparameter case it is represented by the color.
+* **Objective**: Select the objective function you wish to analyze. In the one hyperparameter case,
+  the objective is plotted on the y-axis, in the two hyperparameter case, it is represented by the color.
 
-* **Budget**: Here you can select the multi-fidelity budget to be used. The plugin will only consider trials evaluated on the selected budget.
+* **Budget**: Select the multi-fidelity budget to be used. The plugin will only consider trials evaluated
+  on the selected budget. The *Combined* budget option displays all configurations but shows scores only
+  from the highest budget if a configuration was evaluated with multiple budgets.
 
-* **Hyperparameter 1**: Here you can select the first hyperparameter. It is plotted along the x-axis.
+* **Hyperparameter #1**: Select the first hyperparameter. It is plotted along the x-axis.
 
-* **Hyperparameter 2**: Here you can optionally select the second hyperparameter.
+* **Hyperparameter #2**: Optionally select the second hyperparameter.
 
-* **Show Confidence**: Selection for whether or not to show confidence estimates (based on the surrogate model's uncertainty estimate).
+To refine your analysis, you can apply filters after calculation:
+
+* **Show Confidence**: Selection of whether to display confidence estimates based on the surrogate
+  model's uncertainty estimates.
+
+* **Show ICE Curves**: Selection for whether to to show the ICE curves used during computation
+  (only for the one hyperparameter case).
 
-* **Show ICE Curves**: Selection for whether or not to to show the ICE curves used during computation (only for the one hyperparameter case).
 
 
 .. image:: ../images/plugins/partial_dependencies.png

docs/plugins/symbolic_explanations.rst

@@ -9,7 +9,8 @@ The plugin is capable of answering similar questions as the Partial Dependencies
 
 * How does the objective change with respect to one or two hyperparameters? For example, does the
   accuracy increase if the learning rate decreases?
-* Do multiple trials show similar behavior?
+* What explicit formula describes the relationship between one or two hyperparameters and the
+  model's objective function?
 
 While the Partial Dependencies plugin provides a plot describing the effects of hyperparameters on
 the model performance, the Symbolic Explanations plugin additionally allows to obtain an explicit
@@ -19,4 +20,29 @@ To learn more about Symbolic Explanations, please see the paper
 `Symbolic Explanations for Hyperparameter Optimization
 <https://openreview.net/forum?id=JQwAc91sg_x>`_.
 
+Options
+-------
+* **Objective:** Select the objective function you wish to analyze. In the one hyperparameter case, the objective is plotted on the y-axis, in the two hyperparameter case, it is represented by the color.
+
+* **Budget**: Select the multi-fidelity budget to be used. The plugin will only consider trials evaluated
+  on the selected budget. The *Combined* budget option displays all configurations but shows scores only
+  from the highest budget if a configuration was evaluated with multiple budgets.
+
+* **Hyperparameter #1**: Select the first hyperparameter. It is plotted along the x-axis.
+
+* **Hyperparameter #2**: Optionally select the second hyperparameter.
+
+Furthermore, you can adjust the following parameters of the symbolic regression:
+
+* **Parsimony coefficient:** Penalizes the complexity of the resulting formulas. The higher the value, the higher the penalty on the complexity will be, resulting in simpler formulas.
+
+* **Generations:** The number of generations to evolve the symbolic regression.
+
+* **Population Size:** The number of formulas competing in each generation.
+
+* **Random seed:** The random seed to be used in the symbolic regression.
+
+* **Metric:** The metric to evaluate the fitness of the formulas.
+
+
 .. image:: ../images/plugins/symbolic_explanations.png