Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛🎨♻️Director-v0: improve registry caching #6799

Merged
merged 29 commits into from
Nov 22, 2024

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Nov 21, 2024

What do these changes do?

Since director-v0 was updated, the home-made caching mechanism was replaced by aiocache.
This lead to a difference in behavior, as the home-made caching was running independently of Rest API calls and would pre-build the cache before any other client would list services.
So the current situation is that the first call to the director /services takes a long time until it gets cached properly.

This PR brings:

  • migrated director-v0 client to registry from aiohttp to httpx,
  • added tenacity-based retrials on calls to the docker registry,
  • use of async generators,
  • modernized code,

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added the a:director issue related with the director service label Nov 21, 2024
@sanderegg sanderegg added this to the Event Horizon milestone Nov 21, 2024
@sanderegg sanderegg self-assigned this Nov 21, 2024
Copy link

codecov bot commented Nov 21, 2024

Codecov Report

Attention: Patch coverage is 68.58974% with 49 lines in your changes missing coverage. Please review.

Project coverage is 89.83%. Comparing base (ba881e9) to head (cb215ae).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6799      +/-   ##
==========================================
+ Coverage   88.40%   89.83%   +1.43%     
==========================================
  Files        1545     1102     -443     
  Lines       61634    47971   -13663     
  Branches     2155      410    -1745     
==========================================
- Hits        54488    43097   -11391     
+ Misses       6818     4804    -2014     
+ Partials      328       70     -258     
Flag Coverage Δ
integrationtests 64.76% <ø> (ø)
unittests 87.61% <68.58%> (+1.18%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library 90.62% <66.66%> (-0.38%) ⬇️
pkg_simcore_sdk 85.36% <ø> (ø)
agent 97.00% <ø> (ø)
api_server 89.72% <ø> (ø)
autoscaling 95.21% <ø> (ø)
catalog 90.57% <ø> (ø)
clusters_keeper 98.73% <ø> (ø)
dask_sidecar 91.26% <ø> (ø)
datcore_adapter 93.17% <ø> (ø)
director 76.41% <68.75%> (+0.35%) ⬆️
director_v2 91.06% <ø> (-0.02%) ⬇️
dynamic_scheduler 96.59% <ø> (ø)
dynamic_sidecar 89.75% <ø> (ø)
efs_guardian 90.12% <ø> (ø)
invitations 93.44% <ø> (ø)
osparc_gateway_server 85.49% <ø> (ø)
payments 92.76% <ø> (ø)
resource_usage_tracker 90.64% <ø> (ø)
storage 89.66% <ø> (ø)
webclient ∅ <ø> (∅)
webserver 88.68% <ø> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba881e9...cb215ae. Read the comment docs.

---- 🚨 Try these New Features:

@sanderegg sanderegg marked this pull request as ready for review November 21, 2024 19:29
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice!

timeout=timeout_settings,
json_serialize=json_dumps,
)
session = httpx.AsyncClient(transport=httpx.AsyncHTTPTransport(http2=True))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

registry accepts http/2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well it does not reject it so why not use it?

services/director/tests/unit/conftest.py Show resolved Hide resolved
Copy link

@sanderegg sanderegg merged commit 3655208 into ITISFoundation:master Nov 22, 2024
85 of 90 checks passed
@sanderegg sanderegg deleted the dv-0/list-service-caching branch November 22, 2024 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:director issue related with the director service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants