Parameter constraints dependent on categorical variable identity?

[iandoxsee]

Hi all, I am trying to determine if my use case is supported in Botorch.

I have two parameters from a design space which also contains other categorical/range parameters:
x1 (categorical): A, B, C, D
x2 (range): 0 - 100

I would like to restrict the values of x2 which can be selected, depending on which value of x1 is chosen. For example, if x1 = A, then x2 is restricted to 0-80. If x1 = B, then x2 can take the full 0 -100, C can go 20-60, etc.

I looked at https://github.com/pytorch/botorch/blob/main/botorch/optim/optimize.py#L451-L456 and it's not clear whether this type of "mixed" parameter constraint is supported. Thanks for any help you can provide :)

[Balandat]

This is a "hierarchically constrained" parameter space. This is generally not super easy to handle, and we don't have great support for this in BoTorch today. The naïve approach would be to use fixed_features and enumerate the discrete chocies: https://github.com/pytorch/botorch/blob/main/botorch/optim/optimize.py#L429

Pseudocode:

best = -float("inf")
best_X = None

for x1 in {A, B, C, D}:
    acqfval, X = optimize_acqf(
        ...,
        bounds=get_bounds(x1),
        fixed_features=get_numeric_fixed_features(x1),
        ...,
    ) 
    if acqfval > best:
        best = acqfval
        best_X = X

where get_bounds(x1) returns the respective bounds on x2 (and x1, but those doesn't matter since you're fixing the feature) associated to this value, and get_numeric_fixed_features(x1) basically returns a dictionary with the corresponding numerical values for the one-hot encoding of x1 (assuming you use that approach). Of course this approach doesn't scale to much more complicated search spaces where you can't enumerate all discrete combinations

One question I have regarding this is whether in your setting x2 is still the same parameter, or whether you could also view this as a search space of the form where instead of just x2 you also have x3, x4, x5, and the problem is such that x2 is only "active" and meaningful when x1=A, x3 is only "active" and meaningful when x1=B, etc. This setting is a bit easier to handle than the hierarchically constrained setup from an optimization perspective. From from the description above though it does seem that x2 is indeed the same parameter.

cc @saitcakmak who has done some work on hierarchical search spaces in the past.

[iandoxsee]

Hi Max, thanks a lot for this reply.

I can confirm that the x2 parameter is indeed the same -- let's say it is temperature and x1 is a material identity where some materials have different temperature limits.

Since our real-world experiments have a feasible limit on resolution for each parameter, using a discrete space has been our preference and it sounds like that's beneficial for this approach :) Thanks for the great answer and I will let you know if we have any further issues implementing this. Much appreciated.

[hbai1]

Hi Max, thanks for your hint.

I am using the optimize_acqf_discrete function under these choices: https://github.com/pytorch/botorch/blob/main/botorch/optim/optimize.py#L429

I wonder why this one does not have the option of fixed_features. This is the function we need to use to loop through one fixed feature. Any hint I can do your pseudocode with optimize_acqf_discrete() ?

[Balandat]

@hbai1 I am not sure I fully understand - optimize_acqf_discrete() takes in a finite set of possible choices. So you should be able to drop the elements of choices that do have feature values different from the ones in fixed_features (or construct choices in that way in the first place). Does this make sense?

[hbai1]

A Ha! Make sense. Thank you! --Rachel