Add connectors HTTP metrics

[pubmodmatt]

Add connectors HTTP metrics:

• apollo_router_http_connector_requests_total - counts requests by subgraph/connector/status
• apollo_router_http_connector_request_duration_seconds - a histogram of request times by subgraph/connector

Example metric data:

# HELP apollo_router_http_connector_request_duration_seconds Duration of connectors HTTP requests
# TYPE apollo_router_http_connector_request_duration_seconds histogram
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.001"} 0
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.005"} 0
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.015"} 0
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.05"} 2
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.1"} 2
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.2"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.3"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.4"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="0.5"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="1"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="5"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="10"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router",le="+Inf"} 3
apollo_router_http_connector_request_duration_seconds_sum{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router"} 0.175789542
apollo_router_http_connector_request_duration_seconds_count{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",subgraph="posts",otel_scope_name="apollo/router"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.001"} 0
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.005"} 0
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.015"} 0
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.05"} 2
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.1"} 2
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.2"} 2
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.3"} 2
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.4"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="0.5"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="1"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="5"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="10"} 3
apollo_router_http_connector_request_duration_seconds_bucket{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router",le="+Inf"} 3
apollo_router_http_connector_request_duration_seconds_sum{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router"} 0.433746667
apollo_router_http_connector_request_duration_seconds_count{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",subgraph="posts",otel_scope_name="apollo/router"} 3
# HELP apollo_router_http_connector_requests_total Total number of HTTP requests made through connectors
# TYPE apollo_router_http_connector_requests_total counter
apollo_router_http_connector_requests_total{connector="posts.jsonPlaceholder http: GET /users/{$args.id!}",status="200",subgraph="posts",otel_scope_name="apollo/router"} 3
apollo_router_http_connector_requests_total{connector="posts.jsonPlaceholder http: GET /users/{$this.id!}/posts",status="200",subgraph="posts",otel_scope_name="apollo/router"} 3

---

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

• Changes are compatible¹
• Documentation² completed
• Performance impact assessed and acceptable
• Tests added and passing³
  • Unit Tests
  • Integration Tests
  • Manual Tests

Exceptions

Note any exceptions here

Notes

Footnotes

1. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩

2. Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩

3. Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

[lennyburdette]

@bnjjj can you take a look at these new metrics? 🙏🏻

[bnjjj]

I'm wondering if we really need this as we already have metrics for subgraphs https://www.apollographql.com/docs/router/configuration/telemetry/instrumentation/instruments#opentelemetry-standard-instruments What's the difference, how could we try to include this metrics in custom telemetry ? Is it related to a specific service in plugin trait ?

[bnjjj]

cc @BrynCooke

[pubmodmatt]

I'm wondering if we really need this as we already have metrics for subgraphs https://www.apollographql.com/docs/router/configuration/telemetry/instrumentation/instruments#opentelemetry-standard-instruments What's the difference, how could we try to include this metrics in custom telemetry ? Is it related to a specific service in plugin trait ?

@bnjjj - these metrics are modeled after these router metrics, only instead of for subgraph calls, these are for REST API calls made through the connector service. They are scoped to a connector, so we can see, for example, that for a given connector, how many requests received a status 200 vs 404, or a histogram of how long the HTTP requests take in the REST API. If there's a better way of doing this please let me know.

[BrynCooke]

Agree that metrics should integrate with the existing standard instruments rather than making new ones. In particular adding a selector for the connector. This way users can configure the metrics to include whatever information they want.

The problem with fixed metrics is that users almost always need to add extra attributes to make them useful. In general the rule is, if the metric change was a result of a request then it should be integrated with the telemetry config, lifecycle things such as router reloads can be static.

[pubmodmatt]

Going to rework this based on conversations with the router team.