Random Restarts for Botorch GP models

[huylenguyen]

Hello, I am wondering if Botorch has any functionality built in for multiple random restarts when optimising the log-likelihood of the model? I have been told that this optimisation is not necessarily convex, and so it may be useful to perform multiple random runs of the optimizer.

[saitcakmak]

Hi @huylenguyen. We do no have this exposed out of the box. However, our default model fitting function (fit_gpytorch_mll) has a random restart approach that is used to handle any errors that may get raised during model fitting. The logic for this is here. If you wanted to do random restarts and pick the best, you could modify that to record the result of each fitting attempt -- rather than returning the first time model fitting succeeds -- and pick the best model afterwards.

[Balandat]

We should probably make it easier for folks to do this at some point. Created #1724 for tracking.