Refactoring of in-house scheduling of computational jobs to allow scaling of simcore services and reduce load on system

[sanderegg]

The scheduling of tasks from the director-v2 has the following issues:

1. on each round all the computational jobs are checked (creating a lot of network calls to the dask-schedulers),
2. every time a new pipeline is added 1. is done again even if it was just done,
3. every time a pipeline is stopped 1. is done again,
4. it is not scalable on multiple director-v2 replicas,

--> A distributed lock shall be used to protect each pipeline separately (this way multiple replicas can take care of different pipelines),
after adding/stopping a pipeline only that one should be re-scheduled, not all of them

Tasks

Preview Give feedback

1. 🎨Computational backend: DV-2 computational scheduler becomes replicable (🗃️🚨) #6736
   a:director-v2 +1
   

[sanderegg]

closed by #6736