diff --git a/.github/actions/spelling/expect.txt b/.github/actions/spelling/expect.txt index c080ebe566..19153ad8a7 100644 --- a/.github/actions/spelling/expect.txt +++ b/.github/actions/spelling/expect.txt @@ -517,6 +517,7 @@ printcolumn privs proj promapi +prome promhttp providerref providerresult diff --git a/docs/docs/getting-started/metrics.md b/docs/docs/getting-started/metrics.md index 976fec7b54..0cfe589db5 100644 --- a/docs/docs/getting-started/metrics.md +++ b/docs/docs/getting-started/metrics.md @@ -21,71 +21,202 @@ The Keptn Metrics Server unifies and standardizes access to all this data. Minimal configuration is required because Keptn hooks directly into Kubernetes primitives. -The -[Kubernetes metric server](https://github.com/kubernetes-sigs/metrics-server) -requires that you maintain point-to-point integrations -from Argo Rollouts, Flux, KEDA, and HPA. +The Keptn metrics feature +integrates metrics from any number of metrics providers into a single set of metrics. +This makes it easier to use than the +[Kubernetes metric server](https://github.com/kubernetes-sigs/metrics-server), +which requires that you maintain point-to-point integrations +from each source -- Argo Rollouts, Flux, KEDA, HPA, etc. Each has plugins but it is difficult to maintain them, especially if you are using multiple tools, -and multiple observability platforms, +multiple observability platforms, and multiple instances of some tools or observability platforms. -The Keptn metrics feature unites metrics from all these sources into a single set. ## Using this exercise -This exercise is based on the -[simplenode-dev](https://github.com/keptn-sandbox/klt-on-k3s-with-argocd) -example. -You can clone that repo to access it locally -or just look at it for examples -as you implement the functionality "from scratch" -on your local Kubernetes deployment cluster. +This exercise runs on a +[kind](https://kind.sigs.k8s.io/) +cluster that you set up locally +but could be run on any Kubernetes cluster. -The steps to implement metrics in an existing cluster are: - -1. [Install Keptn](../installation/index.md) -1. Configure the metrics you want to use: - - [Define metrics providers](#define-metrics-providers) - - [Define KeptnMetric information](#define-keptnmetric-information) +1. [Set up the cluster](#set-up-the-cluster) + by creating a local Kubernetes cluster, + installing an instance of Prometheus to use as a data source, + and installing Keptn on the cluster. +1. [Configure Keptn for your metrics](#configure-keptn-for-your-metrics) + by defining custom resources + for each data provider and each piece of data you use. +1. Run the metrics If you want to create your own cluster to run this exercise, follow the instructions in [Installation](../installation/index.md). -## Define metrics to use +## Set up the cluster -You need to define the external observability platforms -from which you want to pull data -and then the specific data you want to pull. -This data is pulled and fetched continuously -at an interval you specify for each specific bit of data. -Data is available through the resource and through the data provider itself, -as well as the Kubernetes CLI. +1. Create the kind cluster: -### Define metrics providers + ```shell + kind create cluster + ``` -Populate a -[KeptnMetricsProvider](../reference/crd-reference/metricsprovider.md) -resource for each external observability platform you want to use. +1. Install and configure Prometheus as your data source -For our example, we define two observability platforms: + For this simple exercise, we use on instance of + [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) + as our data source. + In actual practice, you can use multiple instances + of multiple types of data sources to gather your metrics. -- `dev-prometheus` -- `dev-dynatrace` + Use the following command sequence to install an instance of Prometheus: -You can specify a virtually unlimited number of providers, -including multiple instances of each observability platform. -Each one must be assigned a unique name, -identified by the type of platform it is -and the URL of the target server. -If the target server is protected by a Secret, -provide information about the token and key. + ```shell + helm repo add prometheus-community https://prometheus-community.github.io/helm-charts + helm repo update + helm install kube-prom-stack prometheus-community/kube-prometheus-stack -The [keptn-metrics-provider.yaml](../reference/crd-reference/metricsprovider.md#examples) -file for our example looks like: + ``` -```yaml -{% include "./assets/metric-providers.yaml" %} -``` +1. Install Keptn on your cluster + + Use the following command sequence to install Keptn on your cluster: + + ```shell + helm repo add keptn https://charts.lifecycle.keptn.sh/ + helm repo update + helm upgrade --install keptn keptn/keptn -n keptn-system --create-namespace --wait + ``` + + For more details about how to install Keptn, see the + [Installation Guide](../installation/index.md). + +### Expose Prometheus and get an existing metric + +Now we need to expose Prometheus +and chose an existing metric to use for this exercise. + +1. Use port forwarding to expose prometheus and find a suitable metric: + + ```shell + kubectl get svc + ``` + + This command shows multiple services. + Look for the Prometheus UI (the one on port `9090`) + which should be called `prometheus-operated` (no typo). + +1. Port forward to access the UI: + + ```shell + kubectl -n default port-forward svc/prometheus-operated 9090 + ``` + +1. Go to `http://localhost:9090/` and, in the search box, + type `prometheus_` and select `pr`. + This produces a list of metrics that are defined + +1. For this exercise, we are going to use a single metric. + In real practice, you would be defining many metrics. + Choose a metric that has a non-zero value. + For example: + + ```shell + prometheus_http_requests_total{code="200", container="prometheus", \ + endpoint="http-web", handler="/-/healthy", instance="10.244.0.15:9090", \ + job="kube-prom-stack-kube-prome-prometheus", namespace="default",\ + pod="prometheus-kube-prom-stack-kube-prome-prometheus-0", \ + service="kube-prom-stack-kube-prome-prometheus"} + + ``` + + For a cleaner display, remove some attributes: + + ```shell + prometheus_http_requests_total{code="200", container="prometheus", \ + endpoint="http-web", handler="/-/healthy", \ + job="kube-prom-stack-kube-prome-prometheus", namespace="default", \ + service="kube-prom-stack-kube-prome-prometheus"} + ``` + + This is your metric and value as it is stored in Prometheus. + +## Configure Keptn for your metrics + +Now you must tell Keptn about the metrics you are using.j +You do this by defining: + +- A + [KeptnMetricsProvider](../reference/crd-reference/metricsprovider.md) + resource to define the external observability platform you are using + as a data source. + For this exercise, this is the Prometheus server + you installed and exposed above. +- A + [KeptnMetric](../reference/crd-reference/metric.md) + resource to define each metric query you want to pull. + +The steps are: + +1. Create a + [KeptnMetricsProvider](../reference/crd-reference/metricsprovider.md/) + resource for the observability platform you are using as a data source. + To do this, create a new `.yml` file with content like the following: + + ```yaml + apiVersion: metrics.keptn.sh/v1 + kind: KeptnMetricsProvider + metadata: + name: local-prometheus + namespace: default + spec: + type: prometheus + targetServer: "http://prometheus-operated.default.svc.cluster.local:9090/" + ``` + + You can specify a virtually unlimited number of providers, + including multiple instances of each observability platform. + Each one must be assigned a unique `name`, + identified by the `type` of platform it is + and the URL of the target server. + If the target server is protected by a Secret, + provide information about the token and key. + + Keptn uses the `name` you assign to reference this data source. + + The `targetServer` field tells Keptn where to find this data source; + in this case, it points to a prometheus API endpoint on port 9090. + + Apply this file with the following command: + + ```shell + kubectl apply -f YOUR-KEPTN-METRIC-PROVIDER.yml + ``` + +1. Define your + [KeptnMetric](../reference/crd-reference/metric.md/) + custom resource for each piece of data you want to pull + (in this case, the Prometheus query for the metric you selected). + + This is where you use the Prometheus query from before. + Note: The namespaces of `KeptnMetricsProvider` and `KeptnMetric` must match. + + ```yaml + apiVersion: metrics.keptn.sh/v1 + kind: KeptnMetric + metadata: + name: prometheus-http-requests-total + namespace: default + spec: + provider: + name: local-prometheus + query: 'prometheus_http_requests_total{code="200", handler="/-/healthy", job="kube-prom-stack-kube-prome-prometheus", namespace="default", service="kube-prom-stack-kube-prome-prometheus"}' + fetchIntervalSeconds: 10 + ``` + + This data is pulled and fetched continuously + at an interval you specify for each specific bit of data. + Data is available through the resource + and through the data provider itself, + as well as the Kubernetes CLI. ### Define KeptnMetric information @@ -104,40 +235,45 @@ Each of these are configured to fetch data every 10 seconds but you could configure a different `fetchIntervalSeconds` value for each metric. -The -[keptn-metric.yaml](https://github.com/keptn-sandbox/klt-on-k3s-with-argocd/blob/main/simplenode-dev/keptn-metric.yaml) -file for our example looks like: - -```yaml -{% include "./assets/metric.yaml" %} -``` - Note the following: - Populate one YAML file per metric then apply all of them. - Each metric is assigned a unique `name`. -- The value of the `spec.provider.name` field +- The value of the `spec.provider.name` field in the `KeptnMetric` resource must correspond to the name assigned in the `metadata.name` field of a `KeptnMetricsProvider` resource. - Information is fetched in on a continuous basis at a rate specified by the value of the `spec.fetchIntervalSeconds` field. -### View available metrics +## View available metrics -Use the following command to view -the metrics that are configured in your cluster. -This example displays the two metrics we configured above: +Keptn automatically starts pulling the metrics +(every `fetchIntervalSeconds` seconds): ```shell -kubectl get KeptnMetrics -A +kubectl -n default get keptnmetrics ``` +The above command should show: + ```shell -NAMESPACE NAME PROVIDER QUERY -simplenode-dev availability-slo dev-dynatrace func:slo.availability_simplenodeservice -simplenode-dev available-cpus dev-prometheus sum(kube_node_status_capacity{resource=`cpu`}) +NAME PROVIDER QUERY INTERVAL VALUE +prometheus-http-requests-total local-prometheus prometheus_http_requests_total{code="200", handler="/-/healthy", job="kube-prom-stack-kube-prome-prometheus", namespace="default", service="kube-prom-stack-kube-prome-prometheus"} 756 +``` + +or to retrieve a specific KeptnMetric value: + +```shell +kubectl -n default get keptnmetric/prometheus-http-requests-total +``` + +Use the following command to view +the metrics that are configured in your cluster: + +```shell +kubectl get KeptnMetrics -A ``` ## Run the metrics @@ -157,60 +293,14 @@ The syntax to retrieve metrics from the CR is: kubectl get keptnmetrics.metrics.keptn.sh -n ``` -For example, the output for the `available-cpus` metric looks like: - -```shell -$ kubectl get keptnmetrics.metrics.keptn.sh -n simplenode-dev available-cpus - -NAME PROVIDER QUERY VALUE -cpu-throttling my-provider sum(kube_node_status_capacity{resource=`cpu`}) 6.000 -``` - The syntax to retrieve metrics through the Kubernetes API is: ```yaml kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces//keptnmetrics.metrics.sh//" ``` -For example, the output for the `available-cpus` looks like: - -```yaml -$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/simplenode-dev/keptnmetrics.metrics.sh/available-cpus/available-cpus" - -{ - "kind": "MetricValueList", - "apiVersion": "custom.metrics.k8s.io/v1beta2", - "metadata": {}, - "items": [ - { - "describedObject": { - "kind": "KeptnMetric", - "namespace": "simplenode-dev", - "name": "available-cpus", - "apiVersion": "metrics.keptn.sh/v1" - }, - "metric": { - "name": "available-cpus", - "selector": {} - }, - "timestamp": "2023-05-11T08:05:36Z", - "value": "6" - } - ] -} -``` - You can also display the metrics graphically using a dashboard such as Grafana. -## Implementing autoscaling with HPA - -The Kubernetes HorizontalPodAutoscaler (HPA) -uses metrics to provide autoscaling for the cluster. -HPA can retrieve KeptnMetrics and use those metrics to implement HPA. -See -Using the [HorizontalPodAutoscaler](../use-cases/hpa.md) -for detailed information. - ## Learn more To learn more about the Keptn Metrics Server, see: @@ -219,3 +309,7 @@ To learn more about the Keptn Metrics Server, see: [Keptn Metrics Operator](../components/metrics-operator.md) - More information about implementing Keptn Metrics: [Keptn Metrics](../guides/evaluatemetrics.md) +- How to integrate Keptn metrics with + the Kubernetes HorizontalPodAutoscaler (HPA) + to provide autoscaling for the cluster: + [Using the HorizontalPodAutoscaler](../use-cases/hpa.md)