[DOC] Add new documentation for distributed tracing #4964

vagimeli · 2023-09-05T18:53:22Z

Description

Add new documentation for distributed tracing

Issues Resolved

opensearch-project/OpenSearch#6750
Fixes #3951

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli · 2023-09-05T22:03:46Z

@Gaganjuneja @reta The draft documentation is ready for technical review. Please make your suggestions/updates by end of week. Thank you :)

Signed-off-by: Melissa Vagi <[email protected]>

_observing-your-data/trace/distributed-tracing.md

reta · 2023-09-08T15:32:52Z

_observing-your-data/trace/distributed-tracing.md

+2. **Exporters:** Exporters are responsible for persisting the data. OpenTelemetry provides several out-of-the-box exporters, and OpenSearch currently supports the following:
+    - `LoggingSpanExporter`: Exports spans to a log file, generating a separate file in the logs directory `_otel_traces.log`. Default is `telemetry.otel.tracer.span.exporter.class=io.opentelemetry.exporter.logging.LoggingSpanExporter`.
+    - `OtlpGrpcSpanExporter`: Exports spans by using [gRPC](https://grpc.io/). To use this exporter, you need to install the `otel-collector` on the node and specify the endpoint using the setting `telemetry.otel.tracer.exporter.endpoint`. By default, it writes to the endpoint `http://localhost:4317/`. If you want to configure it for HTTPS, follow the guidance in the [OpenTelemetry Configuration](https://opentelemetry.io/docs/collector/configuration/) documentation. The setting is as follows:
+        - `telemetry.otel.tracer.span.exporter.class=org.opensearch.telemetry.tracing.exporter.OtlpGrpcSpanExporterProvider`


Also not there yet, the pull request is not ready

Yes, Only support for LoggingSpanExporter is there as of 2.10.

We can update these parameters when they're available.

_observing-your-data/trace/distributed-tracing.md

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Signed-off-by: Melissa Vagi <[email protected]>

natebower

@vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_observing-your-data/trace/distributed-tracing.md

natebower · 2023-10-12T11:39:14Z

_observing-your-data/trace/distributed-tracing.md

+
+## Get started
+
+The distributed tracing feature is experimental as of OpenSearch 2.10. To begin using the distributed tracing feature, you need to first enable it using `opensearch.experimental.feature.telemetry.enabled` feature flag, and subsequently activate the tracer, using the dynamic setting `telemetry.tracer.enabled`. It's important to exercise caution when enabling this feature, as it can consume system resources. Detailed information on enabling and configuring distributed tracing, including on-demand troubleshooting and request sampling, is described in the following sections.


Second sentence: "...and subsequently activate the tracer by using..."?

_observing-your-data/trace/distributed-tracing.md

natebower · 2023-10-12T11:51:25Z

_observing-your-data/trace/distributed-tracing.md

+
+### Sampling
+
+Distributed tracing can generate numerous spans, consuming system resources unnecessarily. To reduce the number of traces, also called samples, you can configure different sampling thresholds. By default, sampling is configured to include only 1% of all HTTP requests. Sampling has the following types:


"Sampling includes" instead of "Sampling has"?

natebower · 2023-10-12T11:53:20Z

_observing-your-data/trace/distributed-tracing.md

+1. **Head sampling:** Sampling decisions are made before initiating the root span of a request. OpenSearch supports two head sampling methods:
+    - **Probabilistic:** A blanket limit on incoming requests, dynamically adjustable with the `telemetry.tracer.sampler.probability` setting. This setting ranges between 0 and 1. Default is 0.01, which indicates that 1% of incoming requests are sampled.
+    - **On-Demand:** For debugging specific requests, you can send the `trace=true` attribute as part of the HTTP headers, causing those requests to be sampled regardless of the probabilistic sampling setting.
+2. **Tail sampling:** To configure tail sampling, follow the [OpenTelemetry tail sampling documentation](https://opentelemetry.io/docs/concepts/sampling/#tail-sampling). Configuration depends on the type of collector you choose. Updates on ongoing work for OpenSearch are in the [RFC](https://github.com/opensearch-project/OpenSearch/issues/8918) on GitHub.


"follow the instructions in"? Can we delete the last sentence as unnecessary?

Revised. Yes, deleted.

_observing-your-data/trace/distributed-tracing.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Signed-off-by: Melissa Vagi <[email protected]>

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Signed-off-by: Melissa Vagi <[email protected]>

* Write new documentation about distributed tracing Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

…ct#4964) * Write new documentation about distributed tracing Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

* Write new documentation about distributed tracing Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

Create PR

3b46742

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli added 2 - In progress Issue/PR: The issue or PR is in progress. v2.10.0 labels Sep 5, 2023

vagimeli self-assigned this Sep 5, 2023

vagimeli requested review from cwillum, hdhalter, kolchfa-aws, Naarcha-AWS, ananzh, seanneumann, AMoo-Miki and natebower as code owners September 5, 2023 18:53

This was referenced Sep 5, 2023

[RFC] Distributed Tracing opensearch-project/OpenSearch#6750

Open

[DOC] Distributed Tracing #3951

Closed

vagimeli added 5 commits September 5, 2023 15:19

First draft for technical review

31bc59e

Signed-off-by: Melissa Vagi <[email protected]>

Fix copy labels

888319b

Signed-off-by: Melissa Vagi <[email protected]>

Fix copy labels

083d6fc

Signed-off-by: Melissa Vagi <[email protected]>

Fix copy labels

d07bac9

Signed-off-by: Melissa Vagi <[email protected]>

Fix copy labels

a6f8969

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli added 2 commits September 5, 2023 16:03

Merge branch 'main' into distributed-tracing

82b79b3

Fix copy labels

0c688e8

Signed-off-by: Melissa Vagi <[email protected]>

hdhalter added 3 - Tech review PR: Tech review in progress and removed 2 - In progress Issue/PR: The issue or PR is in progress. labels Sep 7, 2023

reta reviewed Sep 8, 2023

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed Sep 8, 2023

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed Sep 8, 2023

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed Sep 8, 2023

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed Sep 8, 2023

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

vagimeli and others added 6 commits October 11, 2023 10:47

Update _observing-your-data/trace/distributed-tracing.md

a310dd3

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

9c7312c

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

1b6de17

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

2af4a53

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

b9aceb9

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Address doc review feedback

e00a470

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Oct 11, 2023

natebower reviewed Oct 12, 2023

View reviewed changes

vagimeli and others added 14 commits October 12, 2023 13:12

Update _observing-your-data/trace/distributed-tracing.md

1786927

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

138a8bc

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update distributed-tracing.md

30512e2

Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

66fe27e

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

ef33ea1

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

0080688

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

b12a52f

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

8ff95df

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

e1f3bd3

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

ebebef1

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

517552b

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _observing-your-data/trace/distributed-tracing.md

a5ecf52

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

address editorial comments

c91e137

Signed-off-by: Melissa Vagi <[email protected]>

Merge branch 'main' into distributed-tracing

c5172f2

vagimeli merged commit c4ff0aa into main Oct 12, 2023
4 checks passed

vagimeli deleted the distributed-tracing branch October 12, 2023 20:12

hdhalter added release-notes PR: Include this PR in the automated release notes and removed 5 - Editorial review PR: Editorial review in progress labels Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOC] Add new documentation for distributed tracing #4964

[DOC] Add new documentation for distributed tracing #4964

vagimeli commented Sep 5, 2023 •

edited by hdhalter

Loading

vagimeli commented Sep 5, 2023 •

edited

Loading

reta Sep 8, 2023

Gaganjuneja Sep 8, 2023

vagimeli Sep 11, 2023

natebower left a comment

natebower Oct 12, 2023

natebower Oct 12, 2023

natebower Oct 12, 2023

vagimeli Oct 12, 2023


		## Get started

		The distributed tracing feature is experimental as of OpenSearch 2.10. To begin using the distributed tracing feature, you need to first enable it using `opensearch.experimental.feature.telemetry.enabled` feature flag, and subsequently activate the tracer, using the dynamic setting `telemetry.tracer.enabled`. It's important to exercise caution when enabling this feature, as it can consume system resources. Detailed information on enabling and configuring distributed tracing, including on-demand troubleshooting and request sampling, is described in the following sections.


		### Sampling

		Distributed tracing can generate numerous spans, consuming system resources unnecessarily. To reduce the number of traces, also called samples, you can configure different sampling thresholds. By default, sampling is configured to include only 1% of all HTTP requests. Sampling has the following types:

[DOC] Add new documentation for distributed tracing #4964

[DOC] Add new documentation for distributed tracing #4964

Conversation

vagimeli commented Sep 5, 2023 • edited by hdhalter Loading

Description

Issues Resolved

Checklist

vagimeli commented Sep 5, 2023 • edited Loading

reta Sep 8, 2023

Choose a reason for hiding this comment

Gaganjuneja Sep 8, 2023

Choose a reason for hiding this comment

vagimeli Sep 11, 2023

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

natebower Oct 12, 2023

Choose a reason for hiding this comment

natebower Oct 12, 2023

Choose a reason for hiding this comment

natebower Oct 12, 2023

Choose a reason for hiding this comment

vagimeli Oct 12, 2023

Choose a reason for hiding this comment

vagimeli commented Sep 5, 2023 •

edited by hdhalter

Loading

vagimeli commented Sep 5, 2023 •

edited

Loading