Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Fix aiohttp server autoinstrumentation #6391

Merged
merged 83 commits into from
Oct 15, 2024

Conversation

mrnicegyu11
Copy link
Member

@mrnicegyu11 mrnicegyu11 commented Sep 18, 2024

What do these changes do?

Although after carefully reading all the manuals and docs I am quite sure that my last, big opentelemetry PR has correctly added the opentelemetry-autoinstrumentation-pip-packages for both the aiohttp-client (when an aiohttp server sends RESR requests) and the aiohttp-server (when the aiohttp server recieves REST requests) correctly, the traces in jaeger were incomplete. Those spans relating to the aiohttp-server in the webserver, so e.g. the first REST call from the qooxdoo to the webserver, were missing or disconnected.

Some investigation with the debugger showed that the middleware of the aiohttp-server-autoinstrumentation's middleware was never successfully added to the aoihttp webserver (this: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/08def3e40a72a03afb16f3a6b492663de5252469/instrumentation/opentelemetry-instrumentation-aiohttp-server/src/opentelemetry/instrumentation/aiohttp_server/__init__.py#L194 ) . Now, that I add this manual step which the pip package should have done automatically, but dint, maybe simcore somewhere does low-level invasive stuff removing existing middlewares during initialization of the webserver?

This solves the issue, but it is not how the library is intended to be used....

Happy about comments

Update

I have investigated this in-depth and understood why this call is necessary and why it is likely even ok to do so in our case. Thanks for the help @GitHK much appreciated 🙏 ! Please read the comment I have added to the code, and see if this makes it clear / suffces. Thanks!

Related issue/s

How to test

Dev-ops checklist

@mrnicegyu11 mrnicegyu11 self-assigned this Sep 18, 2024
Copy link

codecov bot commented Sep 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.8%. Comparing base (cafbf96) to head (ebc9e8f).
Report is 633 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6391      +/-   ##
=========================================
+ Coverage    84.5%   87.8%    +3.2%     
=========================================
  Files          10    1212    +1202     
  Lines         214   52746   +52532     
  Branches       25     950     +925     
=========================================
+ Hits          181   46325   +46144     
- Misses         23    6243    +6220     
- Partials       10     178     +168     
Flag Coverage Δ
integrationtests 64.7% <ø> (?)
unittests 85.4% <100.0%> (+0.8%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../service-library/src/servicelib/aiohttp/tracing.py 77.1% <100.0%> (ø)

... and 1221 files with indirect coverage changes

@mrnicegyu11 mrnicegyu11 added this to the MartinKippenberger milestone Sep 18, 2024
@mrnicegyu11 mrnicegyu11 added the bug buggy, it does not work as expected label Sep 18, 2024
@mrnicegyu11 mrnicegyu11 marked this pull request as ready for review September 19, 2024 07:29
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please find out why this is happening. and without a hack.
Is this happening also in storage?

@mrnicegyu11
Copy link
Member Author

Please find out why this is happening. and without a hack. Is this happening also in storage?

I dont think I am able to do that on my own :(

@mrnicegyu11
Copy link
Member Author

Anyone feel free to pick this PR up I dont think I want to go down a rabbithole here on my own :--(

@sanderegg
Copy link
Member

Anyone feel free to pick this PR up I dont think I want to go down a rabbithole here on my own :--(

For info @mrnicegyu11 I am going through a rabbit hole myself in autoscaling tests that started failing randomly after open telemetry PR went in and finding issues that were unrelated to it. Sometimes one needs to take time, and the time increases with complexity.

@mrnicegyu11 mrnicegyu11 requested a review from ignapas as a code owner October 14, 2024 08:39
Copy link
Member

@odeimaiz odeimaiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there was a bad merge here. You might not want to touch those frontend files.

@mrnicegyu11
Copy link
Member Author

It looks like there was a bad merge here. You might not want to touch those frontend files.

thanks for catching this, I fixed it please re-review ;) @odeimaiz

@odeimaiz odeimaiz self-requested a review October 14, 2024 09:07
@mrnicegyu11
Copy link
Member Author

@sanderegg @pcrespov I added a test as requested, thanks :--) Please re-review

Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all good! thanks!

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@pcrespov
Copy link
Member

FYI: @mrnicegyu11 yet another deprecation warning within opentelemetry

python3.11/site-packages/opentelemetry/instrumentation/dependencies.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import (

@mrnicegyu11 mrnicegyu11 enabled auto-merge (squash) October 15, 2024 07:44
@mrnicegyu11 mrnicegyu11 disabled auto-merge October 15, 2024 08:18
@mrnicegyu11 mrnicegyu11 dismissed odeimaiz’s stale review October 15, 2024 08:48

doesnt apply anymore

@mrnicegyu11 mrnicegyu11 enabled auto-merge (squash) October 15, 2024 12:27
@mrnicegyu11 mrnicegyu11 disabled auto-merge October 15, 2024 12:27
Copy link

@mrnicegyu11 mrnicegyu11 merged commit 5dde4a4 into ITISFoundation:master Oct 15, 2024
68 checks passed
@mrnicegyu11 mrnicegyu11 deleted the 2024/fix/tracing branch October 15, 2024 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug buggy, it does not work as expected
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants