Move `_dd.apm.enabled` tag in tracing root span #4141

vpellan · 2024-11-21T15:30:19Z

What does this PR do?
It moves the addition of _dd.apm.enabled tag to spans in tracing's Rack middleware instead of Appsec middleware.

Motivation:
During the review of Standalone SCA system-tests, I found out that enabling 'Standalone Appsec Billing' (DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED=true) and disabling 'Appsec Threats' ( DD_APPSEC_ENABLED=false ) would make the tests fails as _dd.apm.enabled is set in appsec Rack middleware.

This is not correct as this tag is suppose to communicate to the Agent that we should not bill for APM traces, and Standalone Appsec Billing will still limit the number of traces to 1 per minute to keep services alive on the backend side even if Appsec Threats is not enabled, thus the traces must contain that tag.

Change log entry

None.

How to test the change?

Standalone SCA system-tests (./run.sh SCA_STANDALONE) should XPASS (or pass if force executed)

pr-commenter · 2024-11-21T15:56:46Z

Benchmarks

Benchmark execution time: 2024-12-03 14:24:02

Comparing candidate commit c28f69f in PR branch vpellan/move-dd-apm-enabled-tag with baseline commit ca7cc9d in branch master.

Found 1 performance improvements and 4 performance regressions! Performance is the same for 26 metrics, 2 unstable metrics.

scenario:tracing - 1 span trace - no writer

🟥 throughput [-754.516op/s; -710.115op/s] or [-5.727%; -5.390%]

scenario:tracing - 100 span trace - no writer

🟥 throughput [-19.685op/s; -18.754op/s] or [-5.626%; -5.360%]

scenario:tracing - Propagation - Datadog

🟥 throughput [-3245.506op/s; -3165.212op/s] or [-9.809%; -9.567%]

scenario:tracing - Propagation - Trace Context

🟩 throughput [+3675.128op/s; +3776.492op/s] or [+10.682%; +10.977%]

scenario:tracing - Tracing.log_correlation

🟥 throughput [-7663.952op/s; -7301.326op/s] or [-6.499%; -6.192%]

codecov-commenter · 2024-11-21T16:11:57Z

Codecov Report

Attention: Patch coverage is 98.93617% with 2 lines in your changes missing coverage. Please review.

Project coverage is 97.75%. Comparing base (ca7cc9d) to head (c28f69f).
Report is 24 commits behind head on master.

Files with missing lines	Patch %	Lines
lib/datadog/tracing/configuration/settings.rb	84.61%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4141      +/-   ##
==========================================
- Coverage   97.76%   97.75%   -0.01%     
==========================================
  Files        1357     1355       -2     
  Lines       81950    81992      +42     
  Branches     4168     4173       +5     
==========================================
+ Hits        80117    80150      +33     
- Misses       1833     1842       +9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

p-datadog · 2024-11-21T18:33:55Z

lib/datadog/tracing/contrib/rack/middlewares.rb

+            # We add this metric when ASM standalone is enabled to make sure we don't bill APM
+            if Datadog.configuration.appsec.standalone.enabled
+              request_span.set_metric(Tracing::Metadata::Ext::TAG_APM_ENABLED, 0)
+            end


In #3965 @anmarchenko mentioned tracing gaining a dependency on appsec and that pattern appears to continue here. I my opinion it would be better for tracing to provide hooks that appsec will hook into to avoid tracing referencing appsec in code, but I am not developing either part of the library.

I do also find it hard to ascertain whether the code behaves correctly - this PR sets APM enabled to 0 from tracing, but it is not clear from the diff where/under what conditions APM enabled is set to 1 (or if that happens at all).

Can you elaborate the reason about tagging a rack span specifically?

The env var name DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED is temporary, and what Standalone Appsec actually does is disabling tracing (or more precisely rate-limiting it at 1 trace per minute), and when 'Appsec' is activated along 'Standalone Appsec' (as Standalone Appsec does not in fact activate Appsec by itself), it force-keep traces that contains Appsec events.

There is a RFC currently being reviewed to find the final name for this env var, which will most likely be DD_TRACE_APM_ENABLED. When it will be approved, I will also change the configuration from Datadog.configuration.appsec.standalone.enabled to something like Datadog.configuration.tracing.apm.enabled, which will make more sense and show that there is no dependency from appsec in tracing (but the other way will be true, which already is anyway)

Can you elaborate the reason about tagging a rack span specifically?

For now, the only product other than tracing that uses traces is Appsec, and in Appsec we only instrument rack-based apps (Rails, Sinatra, Rack itself...)
But maybe we could tag somewhere else, like in http frameworks, now that we have moved this part in tracing. What do you think of adding that tag on the HTTP level ? Or should we stick to Rack, and maybe also add other frameworks like sidekiq ?

I understand this is experimental feature but similar concern with shotgun surgery and product coupling has been raised in the previous pull request code review, that signals the severity of design issue.

It is hard to make sense of the convention and logic using Datadog.configuration.tracing.apm.enabled, which APM includes multiple products including profiling.

I can see similar approach like rack, that AppSec contains the 3rd party patches that depends on TraceOperation.

Or maybe other kinds of dependency injection mechanism

Overall, the requirements (some kinds of hook systems, perhaps could be encompassed by @lloeki 's Instrumentation API ?) are not clear and I am afraid that if we are indifferent now could cast a long shadow in the future.

does it make sense to find the root span here and not the rack span?

Yes, this is changed in e0c503a

Note that "non-billing" naming is temporary.

y9v · 2024-11-27T08:21:43Z

.github/workflows/system-tests.yml

@@ -11,7 +11,7 @@ on:
 env:
  REGISTRY: ghcr.io
  REPO: ghcr.io/datadog/dd-trace-rb
-  SYSTEM_TESTS_REF: main # This must always be set to `main` on dd-trace-rb's master branch
+  SYSTEM_TESTS_REF: igor/standalone-sca # This must always be set to `main` on dd-trace-rb's master branch


reminder: please change this branch to master before merging

y9v · 2024-11-27T08:23:48Z

lib/datadog/tracing/contrib/rack/middlewares.rb

+            # We add this metric when ASM standalone is enabled to make sure we don't bill APM
+            if Datadog.configuration.appsec.standalone.enabled
+              request_span.set_metric(Tracing::Metadata::Ext::TAG_APM_ENABLED, 0)
+            end


does it make sense to find the root span here and not the rack span?

vpellan · 2024-11-27T18:28:22Z

As of 27th. of November, the scope of this PR has been changed to moving all "non-billing" mode code to Tracing part.
("Non billing" mode is what was previously referred as "ASM Standalone" mode, that is now referred as "APM Tracing disabled" mode, but since we have already have a different way to completely disable tracing, I refer to it as non-billing mode in this PR to differentiate both, but please note that this is not the official naming for this feature)
This is still a WIP (specs not updated) so I changed this PR back to Draft

…ec middleware

…LONE_ENABLED

… non-billing mode

…ive trace

…lity

datadog-datadog-prod-us1 · 2024-12-02T14:15:21Z

Datadog Report

Branch report: vpellan/move-dd-apm-enabled-tag
Commit report: c28f69f
Test service: dd-trace-rb

✅ 0 Failed, 22066 Passed, 1461 Skipped, 6m 26.73s Total Time

vpellan requested review from a team as code owners November 21, 2024 15:30

github-actions bot added integrations Involves tracing integrations appsec Application Security monitoring product tracing labels Nov 21, 2024

vpellan requested a review from a team as a code owner November 21, 2024 15:47

p-datadog approved these changes Nov 21, 2024

View reviewed changes

y9v reviewed Nov 27, 2024

View reviewed changes

vpellan marked this pull request as draft November 27, 2024 17:46

github-actions bot added the core Involves Datadog core libraries label Nov 27, 2024

vpellan changed the title ~~Move _dd.apm.enabled tag in tracing Rack middleware~~ Move _dd.apm.enabled tag in tracing root span Nov 28, 2024

vpellan added 14 commits December 2, 2024 14:51

Move _dd.apm.enabled tag in Tracing Rack Middleware instead of AppS…

51e38a0

…ec middleware

Add SCA_STANDALONE scenario and force-execute SCA_STANDALONE tests

69a9d56

Changed c.appsec.standalone.enabled to c.tracing.apm.enabled

40d3afe

Reverse config while we are still using DD_EXPERIMENTAL_APPSEC_STANDA…

0b1b2f7

…LONE_ENABLED

Use TraceOperation method in should_skip_distributed_tracing

c88c86b

Add _dd.apm.enabled: 0 to root spans instead of rack spans when using…

0502899

… non-billing mode

Renamed appsec_standalone_reject? to non_billing_reject?

809a0d6

Lint

a06edd1

Fix should_skip_distributed_tracing with non-billing mode when no act…

6888b3d

…ive trace

Un-factorized shoukd_skip_distributed_tracing for better code readabi…

8e80471

…lity

fix should_skip_distributed_tracing? methods to use trace

d92e76a

Fix "should_skip_distributed_tracing?" in contrib

8ec7c4d

non-billing mode spec

5508805

switch tests values for apm enabled

e6639a4

vpellan force-pushed the vpellan/move-dd-apm-enabled-tag branch from 7ee5c9f to e6639a4 Compare December 2, 2024 14:02

vpellan force-pushed the vpellan/move-dd-apm-enabled-tag branch from b98fbcb to 2061cce Compare December 2, 2024 14:30

Fix error on should_skip_distributed_tracing? on ethon instrumentation

e8acbf8

vpellan force-pushed the vpellan/move-dd-apm-enabled-tag branch from 2061cce to e8acbf8 Compare December 2, 2024 14:34

vpellan added 3 commits December 2, 2024 16:54

Fix gRPC non-billing mode test

cbfa586

Fix gRPC specs

a82d3f2

Remove older should_skip_distributed_tracing?

c28f69f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move `_dd.apm.enabled` tag in tracing root span #4141

Move `_dd.apm.enabled` tag in tracing root span #4141

vpellan commented Nov 21, 2024 •

edited

Loading

pr-commenter bot commented Nov 21, 2024 •

edited

Loading

codecov-commenter commented Nov 21, 2024 •

edited

Loading

p-datadog Nov 21, 2024

TonyCTHsu Nov 22, 2024

vpellan Nov 22, 2024

vpellan Nov 22, 2024 •

edited

Loading

TonyCTHsu Nov 22, 2024 •

edited

Loading

y9v Nov 27, 2024

vpellan Nov 27, 2024

y9v Nov 27, 2024

y9v Nov 27, 2024

vpellan commented Nov 27, 2024 •

edited

Loading

datadog-datadog-prod-us1 bot commented Dec 2, 2024 •

edited

Loading

Move _dd.apm.enabled tag in tracing root span #4141

Are you sure you want to change the base?

Move _dd.apm.enabled tag in tracing root span #4141

Conversation

vpellan commented Nov 21, 2024 • edited Loading

pr-commenter bot commented Nov 21, 2024 • edited Loading

Benchmarks

scenario:tracing - 1 span trace - no writer

scenario:tracing - 100 span trace - no writer

scenario:tracing - Propagation - Datadog

scenario:tracing - Propagation - Trace Context

scenario:tracing - Tracing.log_correlation

codecov-commenter commented Nov 21, 2024 • edited Loading

Codecov Report

p-datadog Nov 21, 2024

Choose a reason for hiding this comment

TonyCTHsu Nov 22, 2024

Choose a reason for hiding this comment

vpellan Nov 22, 2024

Choose a reason for hiding this comment

vpellan Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

TonyCTHsu Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

y9v Nov 27, 2024

Choose a reason for hiding this comment

vpellan Nov 27, 2024

Choose a reason for hiding this comment

y9v Nov 27, 2024

Choose a reason for hiding this comment

y9v Nov 27, 2024

Choose a reason for hiding this comment

vpellan commented Nov 27, 2024 • edited Loading

datadog-datadog-prod-us1 bot commented Dec 2, 2024 • edited Loading

Datadog Report

Move `_dd.apm.enabled` tag in tracing root span #4141

Move `_dd.apm.enabled` tag in tracing root span #4141

vpellan commented Nov 21, 2024 •

edited

Loading

pr-commenter bot commented Nov 21, 2024 •

edited

Loading

codecov-commenter commented Nov 21, 2024 •

edited

Loading

vpellan Nov 22, 2024 •

edited

Loading

TonyCTHsu Nov 22, 2024 •

edited

Loading

vpellan commented Nov 27, 2024 •

edited

Loading

datadog-datadog-prod-us1 bot commented Dec 2, 2024 •

edited

Loading