Skip to content

Commit

Permalink
docs: improve structure of plugin docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Sarah Krebs committed Aug 20, 2024
1 parent 636b407 commit 1a11f44
Show file tree
Hide file tree
Showing 12 changed files with 34 additions and 46 deletions.
4 changes: 2 additions & 2 deletions docs/plugins/ablation_paths.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ To learn more about Ablation Paths, please see the paper
`Efficient Parameter Importance Analysis via Ablation with Surrogates
<https://doi.org/10.1609/aaai.v31i1.10657>`_.

.. image:: ../images/plugins/ablation_paths.png

Options
-------
* **Objective**: Select the objective function you wish to analyze.
Expand All @@ -42,5 +44,3 @@ To refine your analysis, you can apply filters after calculation:
If the performance decreases no matter which hyperparameter is chosen next,
the hyperparameter with the smallest decrease in performance is chosen.
If two hyperparameters increase or decrease the performance equally, one is chosen randomly.

.. image:: ../images/plugins/ablation_paths.png
11 changes: 1 addition & 10 deletions docs/plugins/budget_correlation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,7 @@ This plugin is capable of answering following questions:
* Should I consider running further optimization at lower budgets to cover more of the
configuration space?


Options
-------

**Objective**: Here, you can select the objective you wish to analyze the correlation between
budgets for.

.. image:: ../images/plugins/budget_correlation.png

Correlation Interpretation
--------------------------
Expand All @@ -51,6 +45,3 @@ situation. For reference, here’s a guideline for interpreting correlation scor
* 0.30-0.39: Moderate relationship
* 0.20-0.29: Weak relationship
* 0.01-0.19: No or negligible relationship


.. image:: ../images/plugins/budget_correlation.png
27 changes: 17 additions & 10 deletions docs/plugins/configuration_cube.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,33 @@ Naturally, the space of configurations may be very high dimensional and so this
to investigate the configurations through 2D or 3D *slices* of the configuration space,
plotting the values of hyperparameters as points in this *slice* of the space.

The Configuration Cube is helpful to visually inspect and answer the following questions:

* How does the influence of one hyperparameter on the objective score compare to others?
* Does the optimizer show a preference for specific values of hyperparameters over time?

.. image:: ../images/plugins/configuration_cube.png

Options
-------
* **Objective**: Select the objective function you wish to analyze. The objective scores will be
represented using a color scale.

* **Budget**: Select the multi-fidelity budget to be used. The plugin will only consider trials evaluated
on the selected budget. The *Combined* budget option displays all configurations but shows scores only
from the highest budget if a configuration was evaluated with multiple budgets.

To refine your analysis, you can apply filters after calculation:

* **Number of Configurations**: This slider will allow you to select the number of configurations to
display, where the configurations are ordered by they when they were trialed. Adjusting this slider
helps to visualize how the optimizer's focus on hyperparameters has evolved over time.

* **Hyperparameters**: Choose the slice of the configuration space to visualize. Selecting:
* **One hyperparameter** provides a basic 1D view, where configurations are plotted along a single axis.
* **Two hyperparameters** generates a 2D grid view, showing configurations in a plane.
* **Three hyperparameters** creates a 3D cube view, plotting configurations in three dimensions. In cases where multiple configurations map to the same point, the one with the highest score for the selected objective is displayed. This is particularly noticeable in the 1D view, where many configurations may share the same hyperparameter value.

The Configuration Cube is helpful to visually inspect and answer the following questions:

* How does the influence of one hyperparameter on the objective score compare to others?
* Does the optimizer show a preference for specific values of hyperparameters over time?


.. image:: ../images/plugins/configuration_cube.png
* **One hyperparameter** provides a basic 1D view, where configurations are plotted along a single axis.
* **Two hyperparameters** generates a 2D grid view, showing configurations in a plane.
* **Three hyperparameters** creates a 3D cube view, plotting configurations in three dimensions.
In cases where multiple configurations map to the same point, the one with the highest score for the
selected objective is displayed. This is particularly noticeable in the 1D view, where many configurations
may share the same hyperparameter value.
5 changes: 2 additions & 3 deletions docs/plugins/configuration_footprint.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ algorithm, here `MDS <https://en.wikipedia.org/wiki/Multidimensional_scaling>`_.
preserve distances as accurately as possible when mapping from a high-dimensional space to a
lower-dimensional one. While not perfect, this technique provides valuable insights.

.. image:: ../images/plugins/configuration_footprint.png

There are two plots available. Both share the same axes, with points plotted in consistent
coordinates. Switching between them will provide a comprehensive understanding of your
configuration footprint:
Expand Down Expand Up @@ -65,6 +67,3 @@ representation of your configuration space.
There will likely be small clusters of **Evaluated Points**, where the optimizer focused on
finding optimal configurations, as well as scattered points across valid regions to provide a
broad understanding of scores for various objectives.


.. image:: ../images/plugins/configuration_footprint.png
4 changes: 1 addition & 3 deletions docs/plugins/configurations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ This plugin is capable of answering following questions:
* Which values have been used for a certain configuration?
* How can I access the configuration in Python?

.. image:: ../images/plugins/configurations.png

Overview
--------
Expand Down Expand Up @@ -48,6 +49,3 @@ Code
When a configuration is selected for deployment, it is essential to access it programmatically.
The code section provides you with the code necessary to retrieve the configuration, facilitating
its integration and use in your applications.


.. image:: ../images/plugins/configurations.png
5 changes: 2 additions & 3 deletions docs/plugins/cost_over_time.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ This plugin is capable of answering the following questions:
* How does optimizer A compare to optimizer B? Is optimizer A more effective than optimizer B?
* How long does it take for the optimizer to reach a specific objective value?

.. image:: ../images/plugins/cost_over_time.png


Inputs and Filters
------------------
Expand All @@ -46,6 +48,3 @@ assessing which optimizer converges more quickly and whether the optimizer conve

In contrast to individual runs, groups show the mean and standard deviation across all runs within
the group. This feature helps you understand the variability and spread among multiple runs.


.. image:: ../images/plugins/cost_over_time.png
4 changes: 2 additions & 2 deletions docs/plugins/importances.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ This plugin is capable of answering following questions:
* How does the importance of each hyperparameter vary across different budgets?
* How does the local parameter importance compare to the global importance as determined by fANOVA?

.. image:: ../images/plugins/importances.png

Importance Methods
------------------

Expand Down Expand Up @@ -51,8 +53,6 @@ To refine your analysis, you can apply various filters after calculation:

* **Budgets**: Filter the results to view importance scores specific to certain multi-fidelity budgets, allowing you to analyze how hyperparameter importance varies with budget changes.

.. image:: ../images/plugins/importances.png

.. warning::
As the fANOVA implementation relies on pyrfr, which cannot be applied to constant hyperparameters.
As a workaround, we remove constant hyperparameters before calculation.
Expand Down
4 changes: 1 addition & 3 deletions docs/plugins/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ This plugin is capable of answering the following questions:
* Why did certain trials crash?
* On which multi-fidelity budgets were the configurations evaluated?

.. image:: ../images/plugins/overview.png

Quick Information
-----------------
Expand Down Expand Up @@ -81,6 +82,3 @@ Configuration Space
Lastly, you can find details about the configuration space that was optimized. This section
includes information on the name of each parameter, the possible values that could be explored,
the default value, and whether the optimization was performed on a logarithmic or a linear scale.


.. image:: ../images/plugins/overview.png
4 changes: 2 additions & 2 deletions docs/plugins/parallel_coordinates.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ This can help provide insight and answers to various questions:
* Which hyperparameter values or combinations are associated with high or low final scores?
* What preferences does the optimizer exhibit for certain hyperparameters?

.. image:: ../images/plugins/parallel_coordinates.png

Options
-------

Expand Down Expand Up @@ -56,5 +58,3 @@ The plot offers some interactive features for in-depth analysis:
* **Re-arranging:** You can drag and reposition the hyperparameters and the objective to better
group and compare them. This feature helps in customizing the view to focus on relevant hyperparameters
for your analysis.

.. image:: ../images/plugins/parallel_coordinates.png
1 change: 0 additions & 1 deletion docs/plugins/pareto_front.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,4 @@ This plugin can help answer the following questions:
* Which configuration should I choose if I slightly favor objective A over objective B?
* How does optimizer A compare to optimizer B? Is optimizer A more effective than optimizer B?


.. image:: ../images/plugins/pareto_front.png
6 changes: 2 additions & 4 deletions docs/plugins/partial_dependencies.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ This plugin is capable of answering the following questions:
accuracy increase if the learning rate decreases?
* What are the regions of the hyperparameter space that lead to better or worse performance?

.. image:: ../images/plugins/partial_dependencies.png

Options
-------
* **Objective**: Select the objective function you wish to analyze. In the one hyperparameter case,
Expand All @@ -40,7 +42,3 @@ To refine your analysis, you can apply filters after calculation:

* **Show ICE Curves**: Selection for whether to to show the ICE curves used during computation
(only for the one hyperparameter case).



.. image:: ../images/plugins/partial_dependencies.png
5 changes: 2 additions & 3 deletions docs/plugins/symbolic_explanations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ To learn more about Symbolic Explanations, please see the paper
`Symbolic Explanations for Hyperparameter Optimization
<https://openreview.net/forum?id=JQwAc91sg_x>`_.

.. image:: ../images/plugins/symbolic_explanations.png

Options
-------
* **Objective:** Select the objective function you wish to analyze. In the one hyperparameter case, the objective is plotted on the y-axis, in the two hyperparameter case, it is represented by the color.
Expand All @@ -43,6 +45,3 @@ Furthermore, you can adjust the following parameters of the symbolic regression:
* **Random seed:** The random seed to be used in the symbolic regression.

* **Metric:** The metric to evaluate the fitness of the formulas.


.. image:: ../images/plugins/symbolic_explanations.png

0 comments on commit 1a11f44

Please sign in to comment.