feat: tracing, improved metrics, go-libp2p-kad-dht and boxo upgrade, …

…request timeouts (#87)

* fix: larger duration buckets for better visibility

* feat: log accept header

* fix: move instrumentation to boxo

* feat: add tracing with auth token

* feat: add 30 second request timeout

* chore: remove replace directive

* chore: add missing funcSampler

* chore: remove request timeout

this isn't working too well. We need to look more deeply into this

* chore: update changelog

* chore: go mod tidy

* chore: go-libp2p-kad-dht v0.28.1

* chore: latest boxo#720

* chore: mod tidy

* chore: boxo main

with ipfs/boxo#720 and ipfs/boxo#718

* Apply suggestions from code review

Co-authored-by: Marcin Rataj <[email protected]>

* fix: typo

---------

Co-authored-by: Daniel N <[email protected]>
Co-authored-by: Marcin Rataj <[email protected]>

CHANGELOG.md

@@ -15,8 +15,13 @@ The following emojis are used to highlight certain changes:
 
 ### Added
 
+- Add request tracing with sampling or require token for requests with Traceparent header. See [tracing.md](./docs/tracing.md) for more details.
+
 ### Changed
 
+- go-libp2p-kad-dht updated to [v0.28.1](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.28.1)
+- Metrics `someguy_http_request_duration_seconds` and `someguy_http_response_size_bytes` were replaced with  `delegated_routing_server_http_request_duration_seconds` and `delegated_routing_server_http_response_size_bytes` from upstream `boxo/routing/http/server`.
+
 ### Removed
 
 ### Fixed

docs/environment-variables.md

@@ -19,6 +19,9 @@
   - [`GOLOG_LOG_FMT`](#golog_log_fmt)
   - [`GOLOG_FILE`](#golog_file)
   - [`GOLOG_TRACING_FILE`](#golog_tracing_file)
+- [Tracing](#tracing)
+  - [`SOMEGUY_SAMPLING_FRACTION`](#someguy_sampling_fraction)
+  - [`SOMEGUY_TRACING_AUTH`](#someguy_tracing_auth)
 
 ## Configuration
 
@@ -140,3 +143,22 @@ Sets the file to which the logs are saved. By default, they are printed to the s
 Sets the file to which the tracing events are sent. By default, tracing is disabled.
 
 Warning: Enabling tracing will likely affect performance.
+
+## Tracing
+
+See [tracing.md](tracing.md).
+
+### `SOMEGUY_TRACING_AUTH`
+
+Optional, setting to non-empty value enables on-demand tracing per-request.
+
+The ability to pass `Traceparent` or `Tracestate` headers is guarded by an
+`Authorization` header. The value of the `Authorization` header should match
+the value in the `SOMEGUY_TRACING_AUTH` environment variable.
+
+### `SOMEGUY_SAMPLING_FRACTION`
+
+Optional, set to 0 by default.
+
+The fraction (between 0 and 1) of requests that should be sampled.
+This is calculated independently of any Traceparent based sampling.

docs/tracing.md

@@ -0,0 +1,39 @@
+## Tracing
+
+Tracing across the stack follows, as much as possible, the [Open Telemetry]
+specifications. Configuration environment variables are specified in the
+[OpenTelemetry Environment Variable Specification] where possible. The
+[Boxo Tracing] documentation is the basis for tracing here.
+
+> [!NOTE]
+> A major distinction from the more [general tracing enabled in boxo][Boxo Tracing] is that when
+> tracing is enabled it is restricted to flows through HTTP Gateway requests, rather
+> than also included background processes.
+
+### Fractional Sampling
+
+To sample a % of requests set [`SOMEGUY_SAMPLING_FRACTION`](environment-variables.md#someguy_sampling_fraction) to a value between `0` and `1`.
+
+### Per Request
+
+Per-request tracing is possible when a non-empty [`SOMEGUY_TRACING_AUTH`](environment-variables.md#someguy_tracing_auth) is set in Someguy and when there are both valid
+[Authorization](headers.md#authorization) and [`Traceparent`](headers.md#traceparent) HTTP headers passed in the request.
+
+### Per-request tracing example:
+
+```console
+$ export SOMEGUY_TRACING_AUTH=CHANGEME-tracing-auth-secret # use value from Someguy config
+$ export CID=bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
+$ curl -H "Authorization: $SOMEGUY_TRACING_AUTH" -H "Traceparent: 00-$(openssl rand -hex 16)-00$(openssl rand -hex 7)-01" http://127.0.0.1:8090/routing/v1/providers/$CID -v -o /dev/null
+...
+> Authorization: CHANGEME-tracing-auth-secret
+> Traceparent: 00-b617dc6b6e302ccbabe0115eac80320b-00033792c7de8fc6-01
+...
+````
+
+Now you can search for `trace_id = b617dc6b6e302ccbabe0115eac80320b` to find the trace.
+
+[Boxo Tracing]: https://github.com/ipfs/boxo/blob/main/docs/tracing.md
+[Open Telemetry]: https://opentelemetry.io/
+[OpenTelemetry Environment Variable Specification]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md
+[Trace Context]: https://www.w3.org/TR/trace-context

go.mod

@@ -7,29 +7,34 @@ require (
 	github.com/coreos/go-systemd/v22 v22.5.0
 	github.com/dustin/go-humanize v1.0.1
 	github.com/felixge/httpsnoop v1.0.4
-	github.com/ipfs/boxo v0.24.1
+	github.com/ipfs/boxo v0.24.4-0.20241119003055-e38f236348d6
 	github.com/ipfs/go-cid v0.4.1
 	github.com/ipfs/go-log/v2 v2.5.1
 	github.com/libp2p/go-libp2p v0.37.0
-	github.com/libp2p/go-libp2p-kad-dht v0.27.0
+	github.com/libp2p/go-libp2p-kad-dht v0.28.1
 	github.com/libp2p/go-libp2p-record v0.2.0
 	github.com/multiformats/go-multiaddr v0.13.0
 	github.com/multiformats/go-multibase v0.2.0
 	github.com/multiformats/go-multihash v0.2.3
 	github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58
 	github.com/prometheus/client_golang v1.20.5
 	github.com/rs/cors v1.11.0
-	github.com/slok/go-http-metrics v0.12.0
 	github.com/stretchr/testify v1.9.0
 	github.com/urfave/cli/v2 v2.27.3
-	golang.org/x/sys v0.26.0
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.56.0
+	go.opentelemetry.io/contrib/propagators/autoprop v0.57.0
+	go.opentelemetry.io/otel v1.32.0
+	go.opentelemetry.io/otel/sdk v1.32.0
+	go.opentelemetry.io/otel/trace v1.32.0
+	golang.org/x/sys v0.27.0
 )
 
 require (
 	github.com/Jorropo/jsync v1.0.1 // indirect
 	github.com/andybalholm/brotli v1.1.0 // indirect
 	github.com/benbjohnson/clock v1.3.5 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/containerd/cgroups v1.1.0 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
@@ -50,6 +55,7 @@ require (
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/gorilla/mux v1.8.1 // indirect
 	github.com/gorilla/websocket v1.5.3 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
 	github.com/hashicorp/golang-lru v1.0.2 // indirect
@@ -93,6 +99,7 @@ require (
 	github.com/onsi/ginkgo/v2 v2.20.2 // indirect
 	github.com/opencontainers/runtime-spec v1.2.0 // indirect
 	github.com/opentracing/opentracing-go v1.2.0 // indirect
+	github.com/openzipkin/zipkin-go v0.4.3 // indirect
 	github.com/pion/datachannel v1.5.9 // indirect
 	github.com/pion/dtls/v2 v2.2.12 // indirect
 	github.com/pion/ice/v2 v2.3.36 // indirect
@@ -120,16 +127,25 @@ require (
 	github.com/quic-go/webtransport-go v0.8.1-0.20241018022711-4ac2c9250e66 // indirect
 	github.com/raulk/go-watchdog v1.3.0 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
-	github.com/samber/lo v1.46.0 // indirect
+	github.com/samber/lo v1.47.0 // indirect
+	github.com/slok/go-http-metrics v0.12.0 // indirect
 	github.com/spaolacci/murmur3 v1.1.0 // indirect
 	github.com/stretchr/objx v0.5.2 // indirect
 	github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect
 	github.com/wlynxg/anet v0.0.5 // indirect
 	github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
 	go.opencensus.io v0.24.0 // indirect
-	go.opentelemetry.io/otel v1.28.0 // indirect
-	go.opentelemetry.io/otel/metric v1.28.0 // indirect
-	go.opentelemetry.io/otel/trace v1.28.0 // indirect
+	go.opentelemetry.io/contrib/propagators/aws v1.32.0 // indirect
+	go.opentelemetry.io/contrib/propagators/b3 v1.32.0 // indirect
+	go.opentelemetry.io/contrib/propagators/jaeger v1.32.0 // indirect
+	go.opentelemetry.io/contrib/propagators/ot v1.32.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.31.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.31.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.31.0 // indirect
+	go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.31.0 // indirect
+	go.opentelemetry.io/otel/exporters/zipkin v1.31.0 // indirect
+	go.opentelemetry.io/otel/metric v1.32.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.3.1 // indirect
 	go.uber.org/dig v1.18.0 // indirect
 	go.uber.org/fx v1.23.0 // indirect
 	go.uber.org/mock v0.5.0 // indirect
@@ -143,6 +159,9 @@ require (
 	golang.org/x/text v0.19.0 // indirect
 	golang.org/x/tools v0.26.0 // indirect
 	gonum.org/v1/gonum v0.15.1 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20241104194629-dd2ea8efbc28 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20241104194629-dd2ea8efbc28 // indirect
+	google.golang.org/grpc v1.67.1 // indirect
 	google.golang.org/protobuf v1.35.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	lukechampine.com/blake3 v1.3.0 // indirect