Adds prometheus export point to the agent pod.

[meowjesty]

• Issue: Add Prometheus export point to the agent pod #2975

(wip, missing docs)

Note for reviewer

I've tried keeping the changes in each system to each commit, hopefully that helps when reviewing this, since it touches a bunch of files.

[aviramha]

Note - this should be behind a flag.

[Razz4780]

I didn't go over the code in agent submodules yet, because I have an important request that might require changes there - please no new actor framework ;_;
You're already using axum router, so why not go with State? 👀
I'd just put all metrics in there (AtomicU64), wrap them together in Arc and share between the modules and the metrics endpoint

[Razz4780]

I think we're missing:

1. Stolen HTTP requests in progress
2. Connected clients
3. DNS lookups in progress

I see you went for global gauges, that's fine as well

[Razz4780]

Something should watch over the metrics task. Maybe the CancellationToken we have in start_agent?

[meowjesty]

Help, how do I make this happen? Just clone the cancellation token before tokio::spawn?

[Razz4780]

If you don't like statics, you can go with sth like this:

#[derive(Clone)]
struct Metrics {
    open_file_descriptors: IntCounter,
    // other counters
}

// pass registry to the metrics http server
// pass metrics clones to agent tasks
fn setup_metrics_registry() -> (Registry, Metrics) {
    let open_file_descriptors = IntCounter::new("name", "help").unwrap();

    let registry = Registry::new();
    registry.register(Box::new(open_file_descriptors.clone())).unwrap();

    let metrics = Metrics {
        open_file_descriptors,
    };

    (registry, metrics)
}

// metrics endpoint handler
async fn get_metrics(State(registry): State<Registry>) -> (StatusCode, String) {
    let mut buffer = String::new();
    let encoder = TextEncoder::new();
    let metric_families = registry.gather();
    
    match encoder.encode_utf8(&metric_families, &mut buffer) {
        Ok(()) => (StatusCode::OK, buffer),
        Err(error) => {
            (StatusCode::INTERNAL_SERVER_ERROR, error.to_string())
        }
    }
}

// metrics http server router
async fn metrics_router(registry: Registry) -> Router {
    Router::new().route("/metrics", get(get_metrics)).with_state(registry)
}

Imho looks nicer and will allow for unit testing (statics would be shared across tests)