Multi-Cloud Metrics Server

[Iceber]

What would you like to be added?

The clusterpedia apiserver will have a built-in multi-cluster metrics server that will provide at least the following features:

• Use kubectl --cluster <cluster> top node/pod to view the resource consumption of a node or pod under a cluster
• The user can choose to append resource consumption to annotation when List/Get resources
• Not only does it provide the same cpu and memory metrics as the metrics server, but it should also provide other metrics such as network-related information via external metrics, custom metrics
• Also supports watch metrics, so that users can listen to resource consumption changes without the need for training rounds

We will refer to the following project implementation

• metrics server
• prometheus adapter

The multi-cloud metrics server will be implemented in two ways:

1. the same as the metrics server, with direct access to the cluster's /api/v1/nodes//proxy/metrics/resources and /api/v1/nodes//proxy/metrics/cadvisor interfaces on a regular rotation
2. get metrics information through OpenTelemetry, which can be used for future Agent mode

We plan to implement the first way first in Q3

Why is this needed?

Users could easily get the resource consumption through clusterpedia

[grzesuav]

will it focus on cpu/memory (resources metrics) + network ones (as external/custom) or will allow exposing others via for example prometheus ?

[Iceber]

We should be exposing cpu and memory by default, and possibly network and file io

or will allow exposing others via for example prometheus ?

This looks like a custom metrics resource based on promql or some other syntax, which looks good and is a good idea, and we can consider supporting it after we implement the basic features

[grzesuav]

@Iceber what API you would like to implement first?

• metrics.k8s.io
• custommetrics
• externalmetrics

in first implementation ? As metrics.k8s.io is just for cpu/memory. so for network/io, right ?

[Iceber]

@grzesuav We will first implement a basic usable minimal functionality, and the first stage will not involve prometheus or opentelemery.

And we will first implement the resource metrics(metrics.k8s.io) —— cpu and memory, which can be obtained directly from /api/v1/nodes/< node >/proxy/metrics/resource. （similar to the metrics server's logic)

Then we'll use /api/v1/nodes/< node >/proxy/metrics/cadvisor to get more information, including network and io, etc

In the third phase, we will start to implement more features based on opentelemetry or prometheus.

More suggestions are welcome.

[Iceber]

/assign

[Iceber]

Clusterpedia & Multi Metrics Server

[grzesuav]

@Iceber how you distinguish from which cluster metrics come from ?

[Iceber]

@grzesuav In the early stage, the two implementations of the multicluster metrics server would be two separate modules, and users would have to choose between them to deploy.

Then we would consider combining the two implementations to decide via crd whether to use thanos/otel or node proxy for metric api information for a cluster