✨Autoscaling: scale down while in use 🚨

[sanderegg]

What do these changes do?

• resolves Autoscaling: scaling down unused machines while asking for other types of machines does not work #6367

Before this PR, the autoscaling would either scale UP or DOWN. Therefore if multiple tasks would be in the pipeline requiring one type of machine, and another set of tasks with a different type of machine came in the pipeline, the cluster scaling would deadlock as it would not be able to scale down the unused machines to make room for the machines required by the second set of tasks.

This PR aims to fix this:

• on each round of autoscaling task, any empty machine where no task could be assigned will proceed with the draining process and the subsequent termination process, thus making space for any other type of machine.

🚨: This PR requires some manual testing to validate the approach works as expected

Bonus:

• fixed autoscaling monitoring script with osparc.io
• some refactoring to ease a bit mocking of docker swarm + AWS together

Related issue/s

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 93.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.15%. Comparing base (9012c4d) to head (3a40819).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6898      +/-   ##
==========================================
+ Coverage   88.11%   88.15%   +0.03%     
==========================================
  Files        1587     1580       -7     
  Lines       62156    61978     -178     
  Branches     2008     2008              
==========================================
- Hits        54771    54636     -135     
+ Misses       7050     7006      -44     
- Partials      335      336       +1     

Flag	Coverage Δ
integrationtests	64.95% <ø> (-0.02%)	⬇️
unittests	86.35% <93.33%> (+0.01%)	⬆️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	93.49% <ø> (ø)
pkg_dask_task_models_library	97.09% <ø> (ø)
pkg_models_library	91.24% <ø> (ø)
pkg_notifications_library	84.57% <ø> (ø)
pkg_postgres_database	88.07% <ø> (ø)
pkg_service_integration	70.02% <ø> (ø)
pkg_service_library	75.04% <ø> (-0.06%)	⬇️
pkg_settings_library	90.60% <ø> (ø)
pkg_simcore_sdk	85.38% <ø> (ø)
agent	97.00% <ø> (ø)
api_server	90.04% <ø> (ø)
autoscaling	95.42% <93.33%> (+0.21%)	⬆️
catalog	90.57% <ø> (ø)
clusters_keeper	99.48% <ø> (ø)
dask_sidecar	91.26% <ø> (ø)
datcore_adapter	93.18% <ø> (ø)
director	76.40% <ø> (-0.09%)	⬇️
director_v2	91.37% <ø> (-0.02%)	⬇️
dynamic_scheduler	96.99% <ø> (ø)
dynamic_sidecar	89.75% <ø> (ø)
efs_guardian	90.12% <ø> (ø)
invitations	93.44% <ø> (ø)
osparc_gateway_server	∅ <ø> (∅)
payments	92.65% <ø> (ø)
resource_usage_tracker	89.58% <ø> (-0.07%)	⬇️
storage	89.60% <ø> (ø)
webclient	∅ <ø> (∅)
webserver	87.70% <ø> (+<0.01%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9012c4d...3a40819. Read the comment docs.

[bisgaard-itis]

Nice! Thanks a lot

[mrnicegyu11]

Didnt read the tests, rest looks very nice. one very minor comment

[matusdrobuliak66]

👍

[pcrespov]

thx!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud