Skip to content

Commit

Permalink
feat: add Thanos metrics provider (#3358)
Browse files Browse the repository at this point in the history
Signed-off-by: Moritz Wiesinger <[email protected]>
  • Loading branch information
mowies authored Mar 27, 2024
1 parent 25d5c06 commit 7cc1622
Show file tree
Hide file tree
Showing 27 changed files with 63 additions and 40 deletions.
3 changes: 2 additions & 1 deletion .github/actions/spelling/expect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ annadreal
apicommon
apierr
apierrors
apilifecycle
apiservice
APIURL
apk
Expand Down Expand Up @@ -621,6 +622,7 @@ testcommon
testmetrics
testreg
tgz
thanos
thisthatdc
thschue
timeblock
Expand Down Expand Up @@ -694,4 +696,3 @@ Yash
yml
YOURNAME
yourregistry
apilifecycle
4 changes: 2 additions & 2 deletions .github/scripts/.helm-tests/default/result.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13612,8 +13612,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13748,8 +13748,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2589,8 +2589,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
4 changes: 2 additions & 2 deletions .github/scripts/.helm-tests/metrics-only/result.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2589,8 +2589,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
4 changes: 2 additions & 2 deletions .github/scripts/.helm-tests/metrics-with-certs/result.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2604,8 +2604,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/assets/crd/examples/yaml-synopsis.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ metadata:
name: <data-source-instance-name>
namespace: <namespace>
spec:
type: prometheus | dynatrace | dql | datadog
type: prometheus | thanos | dynatrace | dql | datadog
targetServer: "<data-source-url>"
secretKeyRef:
name: <secret-name>
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/components/metrics-operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ of the application and infrastructure.

While Kubernetes has ways to extend its metrics APIs, there are limitations,
especially that they allow you to use only a single observability platform
such as Prometheus, Dynatrace or Datadog.
such as Prometheus, Thanos, Dynatrace or Datadog.
The Keptn Metrics Operator solves this problem
by providing a single entry point for
all your metrics data, regardless of its source.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/contribute/software/add-new-metric-provider.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ The steps to create your own metrics provider are:
[line](https://github.com/keptn/lifecycle-toolkit/blob/main/metrics-operator/api/v1/keptnmetricsprovider_types.go#L29)
to look like this

`// +kubebuilder:validation:Pattern:=prometheus|dynatrace|datadog|dql|placeholder`.
`// +kubebuilder:validation:Pattern:=prometheus|thanos|dynatrace|datadog|dql|placeholder`.

In the metric-operator directory run `make generate manifests` to update the metrics-operator crd config
Then modify the helm chart and the helm chart crd validation to match the update in the metrics-operator crd config
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/core-concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The Keptn metrics feature extends the functionality of

* Handles observability data from multiple instances
of multiple observability solutions
– Prometheus, Dynatrace, Datadog and others –
– Prometheus, Thanos, Dynatrace, Datadog and others –
as well as data that comes directly from your cloud provider
such as AWS, Google, or Azure.

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/getting-started/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ such as whether a rollout is good, or whether to scale up or down.

Your observability data may come
from multiple observability solutions --
Prometheus, Dynatrace, Datadog and others --
Prometheus, Thanos, Dynatrace, Datadog and others --
or may be data that comes directly
from your cloud provider such as AWS, Google, or Azure.
The Keptn Metrics Server unifies and standardizes access to all this data.
Expand Down
1 change: 1 addition & 0 deletions docs/docs/installation/k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ Your cluster should include the following:

* At least one observability data provider such as
[Prometheus](https://prometheus.io/),
[Thanos](https://thanos.io/),
[Dynatrace](https://www.dynatrace.com/),
or [Datadog](https://www.datadoghq.com/);
you can use multiple instances of different data providers.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/migrate/keptn/strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ Keptn v1
[SLIs](https://v1.keptn.sh/docs/1.0.x/reference/files/sli/)
(Service Level Indicators)
represent queries from the data provider
such as Prometheus, Dynatrace, or Datadog,
such as Prometheus, Thanos, Dynatrace, or Datadog,
which is configured as a Keptn integration.

When migrating to Keptn, you need to define a
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/api-reference/metrics/v1/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ _Appears in:_

| Field | Description | Default | Optional |Validation |
| --- | --- | --- | --- | --- |
| `type` _string_ | Type represents the provider type. This can be one of prometheus, dynatrace, datadog, dql. || x | Optional: {} <br />Pattern: `prometheus|dynatrace|datadog|dql` <br /> |
| `type` _string_ | Type represents the provider type. This can be one of prometheus, thanos, dynatrace, datadog, dql. || x | Optional: {} <br />Pattern: `prometheus|thanos|dynatrace|datadog|dql` <br /> |
| `targetServer` _string_ | TargetServer defines URL (including port and protocol) at which the metrics provider is reachable. || x | |
| `secretKeyRef` _[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#secretkeyselector-v1-core)_ | SecretKeyRef defines an optional secret for access credentials to the metrics provider. ||| Optional: {} <br /> |

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/crd-reference/analysisvaluetemplate.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ metadata:
namespace: <namespace-where-this-resource-resides>
spec:
provider:
name: prometheus | dynatrace | dql | datadog
name: prometheus | thanos | dynatrace | dql | datadog
query: <query>
```
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/crd-reference/evaluationdefinition.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ kind: KeptnEvaluationDefinition
metadata:
name: <evaluation-name>
spec:
source: prometheus | dynatrace | datadog
source: prometheus | dynatrace | datadog | thanos
objectives:
- name: query-1
query: "xxxx"
Expand Down
6 changes: 3 additions & 3 deletions docs/docs/reference/crd-reference/metricsprovider.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ comments: true
# KeptnMetricsProvider

A `KeptnMetricsProvider` resource defines an instance of a data provider
(such as Prometheus, Dynatrace, or Datadog)
(such as Prometheus, Thanos, Dynatrace, or Datadog)
that is used by one or more [KeptnMetric](metric.md) resources.

One Keptn application can perform
Expand Down Expand Up @@ -90,7 +90,7 @@ For detailed information please look at the [Examples section](#examples).

<!-- markdownlint-disable MD046 -->

=== "Prometheus"
=== "Prometheus and Thanos"

An example of Prometheus as a metrics provider with a Secret holding
the authentication data looks like the following:
Expand Down Expand Up @@ -129,7 +129,7 @@ For detailed information please look at the [Examples section](#examples).
> **Note**
When using Dynatrace as metrics provider you can
define the key name of your DT token stored in a secret,
which is not possible for Datadog or Prometheus.
which is not possible for Datadog, Prometheus or Thanos.
For this example `myCustomTokenKey` was used.

<!-- markdownlint-enable MD046 -->
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/use-cases/non-k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,13 +179,13 @@ similar to what the metrics evaluations of the
Keptn v1 quality gates feature provided.
The data used can come from multiple instances
of multiple data providers
(such as Prometheus, Dynatrace, and DataDog).
(such as Prometheus, Thanos, Dynatrace, and DataDog).

A Keptn analysis can be run for any application running anywhere
as long Keptn can access a monitoring provider endpoint
that serves metrics for the application.
You can point to multiple instances of the supported monitoring providers
(Prometheus, Dynatrace, Datadog, and dql)
(Prometheus, Thanos, Dynatrace, Datadog, and dql)
so the application itself can run anywhere.

To implement a Keptn analysis for your deployment:
Expand Down
8 changes: 4 additions & 4 deletions metrics-operator/api/v1/keptnmetricsprovider_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ import (
// KeptnMetricsProviderSpec defines the desired state of KeptnMetricsProvider
type KeptnMetricsProviderSpec struct {
// +kubebuilder:validation:Optional
// +kubebuilder:validation:Pattern:=prometheus|dynatrace|datadog|dql
// Type represents the provider type. This can be one of prometheus, dynatrace, datadog, dql.
// +kubebuilder:validation:Pattern:=prometheus|thanos|dynatrace|datadog|dql
// Type represents the provider type. This can be one of prometheus, thanos, dynatrace, datadog, dql.
Type string `json:"type"`
// TargetServer defines URL (including port and protocol) at which the metrics provider is reachable.
TargetServer string `json:"targetServer"`
Expand Down Expand Up @@ -73,7 +73,7 @@ func (p *KeptnMetricsProvider) HasSecretDefined() bool {
if p.Spec.SecretKeyRef == (corev1.SecretKeySelector{}) {
return false
}
//if the secret name exists the secret is defined
// if the secret name exists the secret is defined
if strings.TrimSpace(p.Spec.SecretKeyRef.Name) == "" {
return false
}
Expand All @@ -84,7 +84,7 @@ func (p *KeptnMetricsProvider) HasSecretKeyDefined() bool {
if p.Spec.SecretKeyRef == (corev1.SecretKeySelector{}) {
return false
}
//if the secret name exists the secret is defined
// if the secret name exists the secret is defined
if strings.TrimSpace(p.Spec.SecretKeyRef.Key) == "" {
return false
}
Expand Down
15 changes: 14 additions & 1 deletion metrics-operator/api/v1/keptnmetricsprovider_types_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ func TestKeptnMetricsProvider_GetType(t *testing.T) {
want string
}{
{
name: "provider type set",
name: "prometheus provider type set",
fields: fields{
ObjectMeta: metav1.ObjectMeta{
Name: "provider1",
Expand All @@ -31,6 +31,19 @@ func TestKeptnMetricsProvider_GetType(t *testing.T) {
},
want: "prometheus",
},
{
name: "thanos provider type set",
fields: fields{
ObjectMeta: metav1.ObjectMeta{
Name: "provider1",
},
Spec: KeptnMetricsProviderSpec{
Type: "thanos",
TargetServer: "",
},
},
want: "thanos",
},
{
name: "provider type not set, should return name",
fields: fields{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ spec:
type: string
type:
description: Type represents the provider type. This can be one of
prometheus, dynatrace, datadog, dql.
pattern: prometheus|dynatrace|datadog|dql
prometheus, thanos, dynatrace, datadog, dql.
pattern: prometheus|thanos|dynatrace|datadog|dql
type: string
required:
- targetServer
Expand Down
1 change: 1 addition & 0 deletions metrics-operator/controllers/common/providers/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package providers
const DynatraceProviderType = "dynatrace"
const DynatraceDQLProviderType = "dql"
const PrometheusProviderType = "prometheus"
const ThanosProviderType = "thanos"
const DataDogProviderType = "datadog"

var SupportedProviders = []string{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ var errCouldNotCast = fmt.Errorf("could not cast result")
var errNoValues = fmt.Errorf("no values in query result")
var errTooManyValues = fmt.Errorf("too many values in query result")

const warningLogString = "%s API returned warnings: %s"

type KeptnPrometheusProvider struct {
Log logr.Logger
K8sClient client.Client
Expand Down Expand Up @@ -61,7 +63,7 @@ func (r *KeptnPrometheusProvider) FetchAnalysisValue(ctx context.Context, query
return "", err
}
if len(warnings) != 0 {
r.Log.Info("Prometheus API returned warnings: " + warnings[0])
r.Log.Info(fmt.Sprintf(warningLogString, provider.GetType(), warnings[0]))
}
res, _, err := getResultForMatrix(result)
return res, err
Expand All @@ -84,7 +86,7 @@ func (r *KeptnPrometheusProvider) EvaluateQuery(ctx context.Context, metric metr
return "", nil, err
}
if len(warnings) != 0 {
r.Log.Info("Prometheus API returned warnings: " + warnings[0])
r.Log.Info(fmt.Sprintf(warningLogString, provider.GetType(), warnings[0]))
}
return getResultForMatrix(result)
} else {
Expand All @@ -93,7 +95,7 @@ func (r *KeptnPrometheusProvider) EvaluateQuery(ctx context.Context, metric metr
return "", nil, err
}
if len(warnings) != 0 {
r.Log.Info("Prometheus API returned warnings: " + warnings[0])
r.Log.Info(fmt.Sprintf(warningLogString, provider.GetType(), warnings[0]))
}
return getResultForVector(result)
}
Expand All @@ -114,7 +116,7 @@ func (r *KeptnPrometheusProvider) EvaluateQueryForStep(ctx context.Context, metr
return nil, nil, err
}
if len(warnings) != 0 {
r.Log.Info("Prometheus API returned warnings: " + warnings[0])
r.Log.Info(fmt.Sprintf(warningLogString, provider.GetType(), warnings[0]))
}
return getResultForStepMatrix(result)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ func TestFetchAnalysisValueWithAuth(t *testing.T) {

svr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
header := r.Header.Get("Authorization")
//prometheus encodes basic user password in header
// prometheus encodes basic user password in header
t.Log(header)
encoded := "Basic " + base64.StdEncoding.EncodeToString([]byte("user:password"))
if strings.Contains(header, encoded) {
Expand Down
2 changes: 1 addition & 1 deletion metrics-operator/controllers/common/providers/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ type ProviderFactory func(providerType string, log logr.Logger, k8sClient client
func NewProvider(providerType string, log logr.Logger, k8sClient client.Client) (KeptnSLIProvider, error) {

switch strings.ToLower(providerType) {
case PrometheusProviderType:
case PrometheusProviderType, ThanosProviderType:
return prometheus.NewPrometheusProvider(log, k8sClient), nil
case DynatraceProviderType:
return &dynatrace.KeptnDynatraceProvider{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ func TestFactory(t *testing.T) {
provider: &prometheus.KeptnPrometheusProvider{},
err: false,
},
{
providerType: ThanosProviderType,
provider: &prometheus.KeptnPrometheusProvider{},
err: false,
},
{
providerType: DynatraceProviderType,
provider: &dynatrace.KeptnDynatraceProvider{},
Expand Down

0 comments on commit 7cc1622

Please sign in to comment.