feat(routing/http/server): expose prometheus metrics #718

2color · 2024-11-13T09:49:04Z

What's in this PR

feat: add instrumentation with handler label to delegated routing endpoints
feat: allow passing in custom prometheus registry

Why

Until now, we relied on instrumentation in consumers of the delegated routing server, e.g. in someguy. The problem is that you cannot get endpoint/handler specific metrics.

The duration buckets were chosen based on probelab data and production someguy metrics. In many cases, requests take over 10 seconds.

codecov · 2024-11-13T10:27:39Z

Codecov Report

Attention: Patch coverage is 82.60870% with 4 lines in your changes missing coverage. Please review.

Project coverage is 60.40%. Comparing base (625aadd) to head (786e657).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
routing/http/server/server.go	82.60%	4 Missing ⚠️

@@            Coverage Diff             @@
##             main     #718      +/-   ##
==========================================
+ Coverage   60.38%   60.40%   +0.02%     
==========================================
  Files         243      243              
  Lines       31021    31039      +18     
==========================================
+ Hits        18731    18749      +18     
+ Misses      10626    10625       -1     
- Partials     1664     1665       +1

Files with missing lines	Coverage Δ
routing/http/server/server.go	`74.22% <82.60%> (+0.14%)`	⬆️

... and 10 files with indirect coverage changes

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

lidel

Tested with #679 in ipfs/someguy#87 and lgtm.

Preview of metrics produced by this PR:

# HELP delegated_routing_server_http_request_duration_seconds The latency of the HTTP requests.
# TYPE delegated_routing_server_http_request_duration_seconds histogram
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="0.1"} 0
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="0.5"} 0
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="1"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="2"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="5"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="8"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="10"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="20"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="30"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="+Inf"} 2
delegated_routing_server_http_request_duration_seconds_sum{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service=""} 1.6826577409999999
delegated_routing_server_http_request_duration_seconds_count{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service=""} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="0.1"} 1
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="0.5"} 1
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="1"} 1
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="2"} 1
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="5"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="8"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="10"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="20"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="30"} 2
delegated_routing_server_http_request_duration_seconds_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="+Inf"} 2
delegated_routing_server_http_request_duration_seconds_sum{code="200",handler="/routing/v1/providers/{cid}",method="GET",service=""} 3.079128329
delegated_routing_server_http_request_duration_seconds_count{code="200",handler="/routing/v1/providers/{cid}",method="GET",service=""} 2
# HELP delegated_routing_server_http_requests_inflight The number of inflight requests being handled at the same time.
# TYPE delegated_routing_server_http_requests_inflight gauge
delegated_routing_server_http_requests_inflight{handler="/routing/v1/peers/{peer-id}",service=""} 0
delegated_routing_server_http_requests_inflight{handler="/routing/v1/providers/{cid}",service=""} 0
# HELP delegated_routing_server_http_response_size_bytes The size of the HTTP responses.
# TYPE delegated_routing_server_http_response_size_bytes histogram
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="100"} 0
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="1000"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="10000"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="100000"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="1e+06"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="1e+07"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="1e+08"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="1e+09"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service="",le="+Inf"} 2
delegated_routing_server_http_response_size_bytes_sum{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service=""} 398
delegated_routing_server_http_response_size_bytes_count{code="200",handler="/routing/v1/peers/{peer-id}",method="GET",service=""} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="100"} 0
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="1000"} 0
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="10000"} 0
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="100000"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="1e+06"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="1e+07"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="1e+08"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="1e+09"} 2
delegated_routing_server_http_response_size_bytes_bucket{code="200",handler="/routing/v1/providers/{cid}",method="GET",service="",le="+Inf"} 2
delegated_routing_server_http_response_size_bytes_sum{code="200",handler="/routing/v1/providers/{cid}",method="GET",service=""} 42133
delegated_routing_server_http_response_size_bytes_count{code="200",handler="/routing/v1/providers/{cid}",method="GET",service=""} 2

I'm going to merge this to allow us to bubble this up to someguy and delegated-ipfs.dev

routing/http/server/server.go

lidel · 2024-11-18T23:00:42Z

routing/http/server/server.go

@@ -133,12 +143,28 @@ func Handler(svc ContentRouter, opts ...Option) http.Handler {
 		opt(server)
 	}

+	if server.promRegistry == nil {
+		server.promRegistry = prometheus.NewRegistry()


iiuc this will disable metric for users by default, requiring opt-in via WithPrometheusRegistry.

I noticed we already call NewRegistry() in other places, and changing this requires a slight refactor of tests, so let's keep this as-is and cleanup in #722 without blocking this PR.

Yeah. Creating a new registry without access to the registry is an anti-pattern because you can't expose those metrics.

We should definitely default to the default registry.

with ipfs/boxo#720 and ipfs/boxo#718

…request timeouts (#87) * fix: larger duration buckets for better visibility * feat: log accept header * fix: move instrumentation to boxo * feat: add tracing with auth token * feat: add 30 second request timeout * chore: remove replace directive * chore: add missing funcSampler * chore: remove request timeout this isn't working too well. We need to look more deeply into this * chore: update changelog * chore: go mod tidy * chore: go-libp2p-kad-dht v0.28.1 * chore: latest boxo#720 * chore: mod tidy * chore: boxo main with ipfs/boxo#720 and ipfs/boxo#718 * Apply suggestions from code review Co-authored-by: Marcin Rataj <[email protected]> * fix: typo --------- Co-authored-by: Daniel N <[email protected]> Co-authored-by: Marcin Rataj <[email protected]>

2color added 2 commits November 13, 2024 10:20

feat: add instrumentation to delegated routing endpoint

8ad9bba

feat: allow passing in custom registry

0fbf207

2color requested a review from a team as a code owner November 13, 2024 09:49

2color added 3 commits November 13, 2024 10:55

docs: update changelog

a9aeafe

chore: go mod tidy

abb2fc2

fix: create new registry unless set

a5f2aa7

2color mentioned this pull request Nov 13, 2024

feat: add tracing, request timeouts and move instrumentation to boxo ipfs/someguy#87

Merged

chore: adjust histogram buckets

2d8efc3

2color requested a review from lidel November 13, 2024 14:18

2color mentioned this pull request Nov 14, 2024

Improve Someguy routing api performance ipfs/someguy#88

Open

lidel self-assigned this Nov 18, 2024

lidel mentioned this pull request Nov 18, 2024

refactor: default to prometheus.DefaultRegisterer #722

Draft

2 tasks

lidel approved these changes Nov 18, 2024

View reviewed changes

refactor: rename histogram

786e657

lidel changed the title ~~feat: instrument delegated routing server~~ feat(routing/http/server): expose prometheus metrics (#718) Nov 19, 2024

lidel enabled auto-merge (squash) November 19, 2024 00:06

lidel disabled auto-merge November 19, 2024 00:06

lidel changed the title ~~feat(routing/http/server): expose prometheus metrics (#718)~~ feat(routing/http/server): expose prometheus metrics Nov 19, 2024

lidel merged commit 1bf1488 into main Nov 19, 2024
17 checks passed

lidel deleted the add-instrumentation branch November 19, 2024 00:31

lidel added a commit to ipfs/someguy that referenced this pull request Nov 19, 2024

chore: boxo main

a2bfa98

with ipfs/boxo#720 and ipfs/boxo#718

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(routing/http/server): expose prometheus metrics #718

feat(routing/http/server): expose prometheus metrics #718

2color commented Nov 13, 2024 •

edited

Loading

codecov bot commented Nov 13, 2024 •

edited

Loading

lidel left a comment •

edited

Loading

lidel Nov 18, 2024

2color Nov 19, 2024

feat(routing/http/server): expose prometheus metrics #718

feat(routing/http/server): expose prometheus metrics #718

Conversation

2color commented Nov 13, 2024 • edited Loading

What's in this PR

Why

codecov bot commented Nov 13, 2024 • edited Loading

Codecov Report

lidel left a comment • edited Loading

Choose a reason for hiding this comment

lidel Nov 18, 2024

Choose a reason for hiding this comment

2color Nov 19, 2024

Choose a reason for hiding this comment

2color commented Nov 13, 2024 •

edited

Loading

codecov bot commented Nov 13, 2024 •

edited

Loading

lidel left a comment •

edited

Loading