⬆️Upgrade dask related libraries and services (⚠️🚨)

[sanderegg]

What do these changes do?

• Upgrade dask-models-library (full-upgrade)
• Upgrade dask-sidecar (full upgrade)
• Upgrade director-v2 (sync upgrade - dask)
• Upgrade autoscaling (sync upgrade - dask)
• Upgrade clusters-keeper (sync upgrade - dask)

Bonus

• fixed codecov upload of reports, also for integration tests

highlights of dask upgrades:

• new dask version: 2024.12.0
• since 2024.7.0: Pub/Sub mechanism is deprecated https://docs.dask.org/en/stable/changelog.html#publish-subscribe-apis-deprecated -> this is annoying and will need replacements
• Dask changelog

Important for release (⚠️ devops)

• whenever the dask library changes it is important that all dask-related services are in sync,
• client/scheduler/workers - python versions, dask versions, serialization library dependencies versions
  --> any private cluster must be stopped prior to deploying!!!!

Highlights on updated libraries (only updated libraries are included)

• #packages before ~ 129
• #packages after ~ 132

#	name	before	after	upgrade	count	packages
1	aio-pika	9.4.1	9.5.3	minor	1	dask-sidecar⬆️
2	aiobotocore	2.13.0	2.15.2	minor	1	dask-sidecar⬆️
3	aiocache	0.12.2	0.12.3		1	dask-sidecar⬆️
4	aiodocker	0.21.0	0.24.0	minor	1	dask-sidecar⬆️
5	aiofiles	23.2.1	24.1.0	MAJOR	1	dask-sidecar⬆️
6	aiohttp	3.9.5	3.11.10	minor	1	dask-sidecar⬆️
7	aioitertools	0.11.0	0.12.0	minor	1	dask-sidecar⬆️
8	aiormq	6.8.0	6.8.1		1	dask-sidecar⬆️
9	anyio	4.3.0	4.7.0	minor	1	dask-sidecar⬆️
10	appdirs	1.4.4	🗑️ removed		1	dask-task-models-library🧪
11	astroid	3.3.4	3.3.5		2	dask-sidecar🔧 dask-task-models-library🔧
12	attrs	23.2.0	24.2.0	MAJOR	2	dask-sidecar⬆️🧪
13	aws-sam-translator	1.89.0	1.94.0	minor	1	dask-sidecar🧪
14	black	24.8.0	24.10.0	minor	2	dask-sidecar🔧 dask-task-models-library🔧
15	blinker	1.8.2	1.9.0	minor	1	dask-sidecar🧪
16	blosc	1.11.1	1.11.2		3	dask-sidecar⬆️⬆️ director-v2⬆️
17	bokeh	3.4.1	3.6.2	minor	1	dask-sidecar⬆️
18	boto3	1.34.106	1.35.36	minor	1	dask-sidecar🧪
19	botocore	1.34.106	1.35.36	minor	2	dask-sidecar⬆️🧪
20	build	1.2.2	1.2.2.post1		2	dask-sidecar🔧 dask-task-models-library🔧
21	certifi	2024.7.4	2024.8.30	minor	2	dask-sidecar⬆️🧪
22	cfn-lint	1.10.3	1.20.2	minor	1	dask-sidecar🧪
23	charset-normalizer	3.3.2	3.4.0	minor	2	dask-sidecar⬆️🧪
24	cloudpickle	3.0.0	3.1.0	minor	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
25	contourpy	1.2.1	1.3.1	minor	1	dask-sidecar⬆️
26	coverage	7.6.1	7.6.8		2	dask-sidecar🧪 dask-task-models-library🧪
27	cryptography	43.0.1	44.0.0	MAJOR	1	dask-sidecar🧪
28	dask	2024.5.1, 2024.9.0	2024.12.0	minor	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
29	deprecated	1.2.14	1.2.15		1	dask-sidecar⬆️
30	dill	0.3.8	0.3.9		2	dask-sidecar🔧 dask-task-models-library🔧
31	distlib	0.3.8	0.3.9		2	dask-sidecar🔧 dask-task-models-library🔧
32	distributed	2024.5.1, 2024.9.0	2024.12.0	minor	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
33	dnspython	2.6.1	2.7.0	minor	2	dask-sidecar⬆️ dask-task-models-library🧪
34	email-validator	2.1.1	2.2.0	minor	1	dask-sidecar⬆️
35	faker	29.0.0	33.1.0	MAJOR	2	dask-sidecar🧪 dask-task-models-library🧪
36	faststream	0.5.31	0.5.33		1	dask-sidecar⬆️
37	flask	3.0.3	3.1.0	minor	1	dask-sidecar🧪
38	flexparser	0.3.1	0.4	minor	1	dask-task-models-library🧪
39	frozenlist	1.4.1	1.5.0	minor	1	dask-sidecar⬆️
40	fsspec	2024.5.0, 2024.9.0	2024.10.0	minor	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
41	googleapis-common-protos	1.65.0	1.66.0	minor	1	dask-sidecar⬆️
42	graphql-core	3.2.4	3.2.5		1	dask-sidecar🧪
43	grpcio	1.66.0	1.68.1	minor	1	dask-sidecar⬆️
44	identify	2.6.1	2.6.3		2	dask-sidecar🔧 dask-task-models-library🔧
45	idna	3.7	3.10	minor	2	dask-sidecar⬆️🧪
46	importlib-metadata	7.1.0	8.5.0	MAJOR	6	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ director-v2⬆️🧪
47	joserfc	1.0.0	1.0.1		1	dask-sidecar🧪
48	jsonpath-ng	1.6.1	1.7.0	minor	1	dask-sidecar🧪
49	jsonschema	4.22.0	4.23.0	minor	2	dask-sidecar⬆️🧪
50	jsonschema-path	0.3.3	🗑️ removed		1	dask-sidecar🧪
51	jsonschema-specifications	2023.12.1, 2023.7.1	2024.10.1	MAJOR	3	dask-sidecar⬆️🧪 dask-task-models-library🧪
52	markupsafe	2.1.5	3.0.2	MAJOR	10	autoscaling⬆️🧪 clusters-keeper⬆️🧪 dask-sidecar⬆️⬆️🧪 dask-task-models-library🧪 director-v2⬆️🧪
53	moto	5.0.15	5.0.22		1	dask-sidecar🧪
54	multidict	6.0.5	6.1.0	minor	1	dask-sidecar⬆️
55	mypy	1.12.0	1.13.0	minor	2	dask-sidecar🔧 dask-task-models-library🔧
56	networkx	3.3	3.4.2	minor	1	dask-sidecar🧪
57	numpy	1.26.4	2.1.3	MAJOR	4	dask-sidecar⬆️⬆️ director-v2⬆️🧪
58	openapi-schema-validator	0.6.2	0.4.3	🔥 downgrade	1	dask-sidecar🧪
59	openapi-spec-validator	0.7.1	0.5.5	🔥 downgrade	1	dask-sidecar🧪
60	opentelemetry-api	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
61	opentelemetry-exporter-otlp	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
62	opentelemetry-exporter-otlp-proto-common	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
63	opentelemetry-exporter-otlp-proto-grpc	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
64	opentelemetry-exporter-otlp-proto-http	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
65	opentelemetry-instrumentation	0.48, 0.47	0.49	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
66	opentelemetry-instrumentation-aiopg	0.48	0.49	minor	1	director-v2⬆️
67	opentelemetry-instrumentation-asgi	0.48, 0.47	0.49	minor	3	autoscaling⬆️ clusters-keeper⬆️ director-v2⬆️
68	opentelemetry-instrumentation-asyncpg	0.48	0.49	minor	1	director-v2⬆️
69	opentelemetry-instrumentation-botocore	0.47	0.49	minor	2	autoscaling⬆️ clusters-keeper⬆️
70	opentelemetry-instrumentation-dbapi	0.48	0.49	minor	1	director-v2⬆️
71	opentelemetry-instrumentation-fastapi	0.48, 0.47	0.49	minor	3	autoscaling⬆️ clusters-keeper⬆️ director-v2⬆️
72	opentelemetry-instrumentation-httpx	0.48, 0.47	0.49	minor	3	autoscaling⬆️ clusters-keeper⬆️ director-v2⬆️
73	opentelemetry-instrumentation-logging	0.48, 0.47	0.49	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
74	opentelemetry-instrumentation-redis	0.48, 0.47	0.49	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
75	opentelemetry-instrumentation-requests	0.48, 0.47	0.49	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
76	opentelemetry-proto	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
77	opentelemetry-sdk	1.27.0, 1.26.0	1.28.2	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
78	opentelemetry-semantic-conventions	0.48, 0.47	0.49	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
79	opentelemetry-util-http	0.48, 0.47	0.49	minor	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
80	orjson	3.10.7, 3.10.3	3.10.12		2	dask-sidecar⬆️ dask-task-models-library🧪
81	packaging	24.0, 24.1	24.2	minor	16	autoscaling⬆️🧪🔧 clusters-keeper⬆️🧪🔧 dask-sidecar⬆️⬆️🧪🔧 dask-task-models-library🧪🧪🔧 director-v2⬆️🧪🔧
82	pandas	2.2.2	2.2.3		1	dask-sidecar⬆️
83	pillow	10.3.0	11.0.0	MAJOR	1	dask-sidecar⬆️
84	pint	0.24.3	0.24.4		1	dask-task-models-library🧪
85	pip	24.2	24.3.1	minor	2	dask-sidecar🔧 dask-task-models-library🔧
86	pre-commit	3.8.0	4.0.1	MAJOR	2	dask-sidecar🔧 dask-task-models-library🔧
87	prometheus-client	0.20.0	0.21.1	minor	1	dask-sidecar⬆️
88	protobuf	4.25.4	5.29.0,5.29.1	MAJOR	4	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️ director-v2⬆️
89	psutil	6.0.0	6.1.0	minor	9	autoscaling⬆️🧪 clusters-keeper⬆️🧪 dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
90	pydantic	2.10.2	2.10.3		2	dask-sidecar⬆️🧪
91	pydantic-extra-types	2.9.0	2.10.0	minor	2	dask-sidecar⬆️ dask-task-models-library🧪
92	pyftpdlib	2.0.0	2.0.1		1	dask-sidecar🧪
93	pyinstrument	4.6.2	5.0.0	MAJOR	1	dask-sidecar⬆️
94	pylint	3.3.0	3.3.2		2	dask-sidecar🔧 dask-task-models-library🔧
95	pyopenssl	24.2.1	24.3.0	minor	1	dask-sidecar🧪
96	pyparsing	3.1.4	3.2.0	minor	1	dask-sidecar🧪
97	pyproject-hooks	1.1.0	1.2.0	minor	2	dask-sidecar🔧 dask-task-models-library🔧
98	pytest	8.3.3	8.3.4		2	dask-sidecar🧪 dask-task-models-library🧪
99	pytest-cov	5.0.0	6.0.0	MAJOR	2	dask-sidecar🧪 dask-task-models-library🧪
100	pytz	2024.1	2024.2	minor	1	dask-sidecar⬆️
101	pyyaml	6.0.1	6.0.2		13	autoscaling⬆️🧪🔧 clusters-keeper⬆️🧪🔧 dask-sidecar⬆️⬆️🧪🔧 director-v2⬆️🧪🔧
102	redis	5.0.4	5.2.0	minor	1	dask-sidecar⬆️
103	referencing	0.29.3	0.35.1	minor	2	dask-sidecar⬆️🧪
104	regex	2024.9.11	2024.11.6	minor	1	dask-sidecar🧪
105	rich	13.8.1, 13.7.1	13.9.4	minor	2	dask-sidecar⬆️ dask-task-models-library🧪
106	rpds-py	0.18.1, 0.20.0	0.22.3	minor	3	dask-sidecar⬆️🧪 dask-task-models-library🧪
107	ruff	0.6.7	0.8.2	minor	2	dask-sidecar🔧 dask-task-models-library🔧
108	s3fs	2024.5.0	2024.10.0	minor	1	dask-sidecar⬆️
109	s3transfer	0.10.2	0.10.4		1	dask-sidecar🧪
110	setuptools	75.1.0, 74.0.0	75.6.0	minor	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️🧪🔧 dask-task-models-library🔧 director-v2⬆️
111	six	1.16.0	1.17.0	minor	4	dask-sidecar⬆️🧪 dask-task-models-library🧪🧪
112	tenacity	8.5.0	9.0.0	MAJOR	1	dask-sidecar⬆️
113	termcolor	2.4.0	2.5.0	minor	2	dask-sidecar🧪 dask-task-models-library🧪
114	toolz	0.12.1	1.0.0	MAJOR	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
115	tornado	6.4, 6.4.1	6.4.2		7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪
116	tqdm	4.66.4	4.67.1	minor	1	dask-sidecar⬆️
117	typer	0.12.3, 0.12.5	0.15.1	minor	2	dask-sidecar⬆️ dask-task-models-library🧪
118	types-networkx	3.2.1.20240918	3.4.2.20241115	minor	1	director-v2🧪
119	types-python-dateutil	2.9.0.20240316, 2.9.0.20240906	2.9.0.20241206		2	dask-sidecar⬆️ dask-task-models-library🧪
120	tzdata	2024.1	2024.2	minor	1	dask-sidecar⬆️
121	virtualenv	20.26.5	20.28.0	minor	2	dask-sidecar🔧 dask-task-models-library🔧
122	watchdog	5.0.2	6.0.0	MAJOR	1	dask-sidecar🔧
123	werkzeug	3.0.4	3.1.3	minor	1	dask-sidecar🧪
124	wheel	0.44.0	0.45.1	minor	2	dask-sidecar🔧 dask-task-models-library🔧
125	wrapt	1.16.0	1.17.0	minor	2	dask-sidecar⬆️🧪
126	xmltodict	0.13.0	0.14.2	minor	1	dask-sidecar🧪
127	xyzservices	2024.4.0	2024.9.0	minor	1	dask-sidecar⬆️
128	yarl	1.9.4	1.18.3	minor	1	dask-sidecar⬆️
129	zipp	3.20.2, 3.18.2	3.21.0	minor	7	autoscaling⬆️ clusters-keeper⬆️ dask-sidecar⬆️⬆️ dask-task-models-library🧪 director-v2⬆️🧪

Legend:

• ⬆️ base dependency (only services because packages are floating)
• 🧪 test dependency
• 🔧 tool dependency

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.39%. Comparing base (35f11df) to head (25ab68a).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6873      +/-   ##
==========================================
+ Coverage   88.35%   88.39%   +0.04%     
==========================================
  Files        1569     1562       -7     
  Lines       61546    61365     -181     
  Branches     2001     1987      -14     
==========================================
- Hits        54378    54245     -133     
+ Misses       6833     6786      -47     
+ Partials      335      334       -1     

Flag	Coverage Δ
integrationtests	65.09% <70.58%> (+0.03%)	⬆️
unittests	86.58% <100.00%> (+0.03%)	⬆️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	93.49% <ø> (ø)
pkg_dask_task_models_library	97.09% <ø> (ø)
pkg_models_library	91.36% <ø> (ø)
pkg_notifications_library	84.57% <ø> (ø)
pkg_postgres_database	87.99% <ø> (ø)
pkg_service_integration	70.02% <ø> (ø)
pkg_service_library	76.00% <ø> (+0.06%)	⬆️
pkg_settings_library	90.60% <ø> (ø)
pkg_simcore_sdk	85.38% <ø> (ø)
agent	97.00% <ø> (ø)
api_server	90.03% <ø> (ø)
autoscaling	95.21% <ø> (ø)
catalog	90.57% <ø> (ø)
clusters_keeper	98.72% <ø> (ø)
dask_sidecar	91.26% <ø> (ø)
datcore_adapter	93.18% <ø> (ø)
director	76.40% <ø> (-0.09%)	⬇️
director_v2	91.38% <100.00%> (+0.01%)	⬆️
dynamic_scheduler	97.07% <ø> (ø)
dynamic_sidecar	89.78% <ø> (+0.03%)	⬆️
efs_guardian	90.12% <ø> (ø)
invitations	93.44% <ø> (ø)
osparc_gateway_server	∅ <ø> (∅)
payments	92.65% <ø> (ø)
resource_usage_tracker	89.07% <ø> (ø)
storage	89.66% <ø> (ø)
webclient	∅ <ø> (∅)
webserver	88.33% <ø> (+<0.01%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35f11df...25ab68a. Read the comment docs.

[pcrespov]

thx!

[bisgaard-itis]

Cool! Thanks a lot

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud