convergence failed repeatedly

[FrancescoCavarretta]

The optimization crashes with the following error:

CVode-- At t = 2748.42 and h = 2.58932e-05, the corrector
convergence failed repeatedly or with |h| = hmin.

How could I make the optimizer robust against this error?
Would it be possible to intercept this error and handle it without having any interruption in the optimization?

[AurelienJaquier]

Hi there! This is a problem that we also encountered. We have two ways of fixing this, although both have some caveats.

1/ When this error happens, the cell usually takes much more time to compute than usually, since it uses very small timesteps. To avoid the crash, we can tell BluePyOpt to stop the process running a cell after a given timeout. The trick is to give a timeout that will trigger before a 'bad' cell makes NEURON crash, but does not trigger before a 'good' cell finishes running.

You can set the timeout value (in seconds) at the evaluator level, here: https://github.com/BlueBrain/BluePyOpt/blob/master/bluepyopt/ephys/evaluators.py#L45

2/ Another way to prevent BluePyOpt to stop optimisation when NEURON crashes is to use a multiprocessing mapping function in the optimiser. That way, when NEURON crashes, it does it inside a process, and that does not make the optimisation crash. The caveat here is that we can not use ipyparallel and multiprocessing at the same time, and you can only use one node with the multiprocessing method on a cluster. It can thus lead to optimisations taking more time to finish.

Here is where you can set the mapping function:
https://github.com/BlueBrain/BluePyOpt/blob/master/bluepyopt/deapext/optimisationsCMA.py#L60 for CMA
https://github.com/BlueBrain/BluePyOpt/blob/master/bluepyopt/deapext/optimisations.py#L103 for IBEA
To create the multiprocessing mapping function, we use this function (not from BluePyOpt):
https://github.com/BlueBrain/BluePyEModel/blob/main/bluepyemodel/tools/multiprocessing.py#L84

I hope that helps.