Merge branch 'main' into tpatterson/compactor-resource-limtis

CHANGELOG.md

@@ -6,6 +6,8 @@
 
 ##### Enhancements
 
+* [11851](https://github.com/grafana/loki/pull/11851) **elcomtik**: Helm: Allow the definition of resources for GrafanaAgent pods.
+* [11819](https://github.com/grafana/loki/pull/11819) **jburnham**: Ruler: Add the ability to disable the `X-Scope-OrgId` tenant identification header in remote write requests.
 * [11633](https://github.com/grafana/loki/pull/11633) **cyriltovena**: Add profiling integrations to tracing instrumentation.
 * [11571](https://github.com/grafana/loki/pull/11571) **MichelHollands**: Add a metrics.go log line for requests from querier to ingester
 * [11477](https://github.com/grafana/loki/pull/11477) **MichelHollands**: support GET for /ingester/shutdown
@@ -54,6 +56,7 @@
 * [11143](https://github.com/grafana/loki/pull/11143) **sandeepsukhani** otel: Add support for per tenant configuration for mapping otlp data to loki format
 * [11499](https://github.com/grafana/loki/pull/11284) **jmichalek132** Config: Adds `frontend.log-query-request-headers` to enable logging of request headers in query logs.
 * [11817](https://github.com/grafana/loki/pull/11817) **ashwanthgoli** Ruler: Add support for filtering results of `/prometheus/api/v1/rules` endpoint by rule_name, rule_group, file and type.
+* [11897](https://github.com/grafana/loki/pull/11897) **ashwanthgoli** Metadata: Introduces a separate split interval of `split_recent_metadata_queries_by_interval` for `recent_metadata_query_window` to help with caching recent metadata query results.
 
 ##### Fixes
 * [11074](https://github.com/grafana/loki/pull/11074) **hainenber** Fix panic in lambda-promtail due to mishandling of empty DROP_LABELS env var.

docs/sources/configure/_index.md

@@ -1274,6 +1274,10 @@ remote_write:
   # CLI flag: -ruler.remote-write.config-refresh-period
   [config_refresh_period: <duration> | default = 10s]
 
+  # Add X-Scope-OrgID header in remote write requests.
+  # CLI flag: -ruler.remote-write.add-org-id-header
+  [add_org_id_header: <boolean> | default = true]
+
 # Configuration for rule evaluation.
 evaluation:
   # The evaluation mode for the ruler. Can be either 'local' or 'remote'. If set
@@ -2907,6 +2911,30 @@ The `limits_config` block configures global and per-tenant limits in Loki.
 # CLI flag: -querier.split-metadata-queries-by-interval
 [split_metadata_queries_by_interval: <duration> | default = 1d]
 
+# Experimental. Split interval to use for the portion of metadata request that
+# falls within `recent_metadata_query_window`. Rest of the request which is
+# outside the window still uses `split_metadata_queries_by_interval`. If set to
+# 0, the entire request defaults to using a split interval of
+# `split_metadata_queries_by_interval.`.
+# CLI flag: -experimental.querier.split-recent-metadata-queries-by-interval
+[split_recent_metadata_queries_by_interval: <duration> | default = 1h]
+
+# Experimental. Metadata query window inside which
+# `split_recent_metadata_queries_by_interval` gets applied, portion of the
+# metadata request that falls in this window is split using
+# `split_recent_metadata_queries_by_interval`. The value 0 disables using a
+# different split interval for recent metadata queries.
+# 
+# This is added to improve cacheability of recent metadata queries. Query split
+# interval also determines the interval used in cache key. The default split
+# interval of 24h is useful for caching long queries, each cache key holding 1
+# day's results. But metadata queries are often shorter than 24h, to cache them
+# effectively we need a smaller split interval. `recent_metadata_query_window`
+# along with `split_recent_metadata_queries_by_interval` help configure a
+# shorter split interval for recent metadata queries.
+# CLI flag: -experimental.querier.recent-metadata-query-window
+[recent_metadata_query_window: <duration> | default = 0s]
+
 # Interval to use for time-based splitting when a request is within the
 # `query_ingesters_within` window; defaults to `split-queries-by-interval` by
 # setting to 0.

docs/sources/release-notes/cadence.md

@@ -8,7 +8,7 @@ weight: 1
 
 ## Stable Releases
 
-Loki releases (this includes [Promtail](/clients/promtail), [Loki Canary](/operations/loki-canary/), etc) use the following
+Loki releases (this includes [Promtail](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/promtail/), [Loki Canary](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/loki-canary/), etc.) use the following
 naming scheme: `MAJOR`.`MINOR`.`PATCH`.
 
 - `MAJOR` (roughly once a year): these releases include large new features and possible backwards-compatibility breaks.
@@ -18,14 +18,14 @@ naming scheme: `MAJOR`.`MINOR`.`PATCH`.
 {{% admonition type="note" %}}
 While our naming scheme resembles [Semantic Versioning](https://semver.org/), at this time we do not strictly follow its
 guidelines to the letter. Our goal is to provide regular releases that are as stable as possible, and we take backwards-compatibility
-seriously. As with any software, always read the [release notes](/release-notes) and the [upgrade guide](/upgrading) whenever
+seriously. As with any software, always read the [release notes](https://grafana.com/docs/loki/<LOKI_VERSION>/release-notes/) and the [upgrade guide](https://grafana.com/docs/loki/<LOKI_VERSION>/setup/upgrade/) whenever
 choosing a new version of Loki to install.
 {{% /admonition %}}
 
 New releases are based of a [weekly release](#weekly-releases) which we have vetted for stability over a number of weeks.
 
 We strongly recommend keeping up-to-date with patch releases as they are released. We post updates of new releases in the `#loki` channel
-of our [Slack community](/community/getting-in-touch).
+of our [Slack community](https://grafana.com/docs/loki/<LOKI_VERSION>/community/getting-in-touch/).
 
 You can find all of our releases [on GitHub](https://github.com/grafana/loki/releases) and on [Docker Hub](https://hub.docker.com/r/grafana/loki).
 

docs/sources/setup/install/helm/reference.md

@@ -2806,6 +2806,15 @@ true
 			<td><pre lang="json">
 null
 </pre>
+</td>
+		</tr>
+		<tr>
+			<td>monitoring.selfMonitoring.grafanaAgent.resources</td>
+			<td>object</td>
+			<td>Resource requests and limits for the grafanaAgent pods</td>
+			<td><pre lang="json">
+{}
+</pre>
 </td>
 		</tr>
 		<tr>

docs/variables.mk

@@ -1,8 +1,5 @@
 # List of projects to provide to the make-docs script.
 PROJECTS := loki
 
-# Use alternative image until make-docs 3.0.0 is rolled out.
-export DOCS_IMAGE := grafana/docs-base:dbd975af06
-
 # Set the DOC_VALIDATOR_IMAGE to match the one defined in CI.
 export DOC_VALIDATOR_IMAGE := $(shell sed -En 's, *image: "(grafana/doc-validator.*)",\1,p' "$(shell git rev-parse --show-toplevel)/.github/workflows/doc-validator.yml")

operator/CHANGELOG.md

@@ -1,5 +1,6 @@
 ## Main
 
+- [11964](https://github.com/grafana/loki/pull/11964) **xperimental**: Provide Azure region for managed credentials using environment variable
 - [11920](https://github.com/grafana/loki/pull/11920) **xperimental**: Refactor handling of credentials in managed-auth mode
 - [11869](https://github.com/grafana/loki/pull/11869) **periklis**: Add support for running with Google Workload Identity
 - [11868](https://github.com/grafana/loki/pull/11868) **xperimental**: Integrate support for OpenShift-managed credentials in Azure

operator/internal/config/managed_auth.go

@@ -26,6 +26,7 @@ func discoverManagedAuthConfig() *ManagedAuthConfig {
 	clientID := os.Getenv("CLIENTID")
 	tenantID := os.Getenv("TENANTID")
 	subscriptionID := os.Getenv("SUBSCRIPTIONID")
+	region := os.Getenv("REGION")
 
 	switch {
 	case roleARN != "":
@@ -40,6 +41,7 @@ func discoverManagedAuthConfig() *ManagedAuthConfig {
 				ClientID:       clientID,
 				SubscriptionID: subscriptionID,
 				TenantID:       tenantID,
+				Region:         region,
 			},
 		}
 	}

operator/internal/handlers/credentialsrequest_create.go → operator/internal/handlers/credentialsrequest.go

@@ -2,12 +2,10 @@ package handlers
 
 import (
 	"context"
-	"errors"
 	"fmt"
 
 	"github.com/ViaQ/logerr/v2/kverrors"
 	"github.com/go-logr/logr"
-	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/runtime"
 	ctrl "sigs.k8s.io/controller-runtime"
@@ -19,11 +17,8 @@ import (
 	"github.com/grafana/loki/operator/internal/external/k8s"
 	"github.com/grafana/loki/operator/internal/manifests"
 	"github.com/grafana/loki/operator/internal/manifests/openshift"
-	"github.com/grafana/loki/operator/internal/manifests/storage"
 )
 
-var errAzureNoRegion = errors.New("can not create CredentialsRequest: missing secret field: region")
-
 // CreateCredentialsRequest creates a new CredentialsRequest resource for a Lokistack
 // to request a cloud credentials Secret resource from the OpenShift cloud-credentials-operator.
 func CreateCredentialsRequest(ctx context.Context, log logr.Logger, scheme *runtime.Scheme, managedAuth *config.ManagedAuthConfig, k k8s.Client, req ctrl.Request) error {
@@ -39,32 +34,6 @@ func CreateCredentialsRequest(ctx context.Context, log logr.Logger, scheme *runt
 		return kverrors.Wrap(err, "failed to lookup LokiStack", "name", req.String())
 	}
 
-	if managedAuth.Azure != nil && managedAuth.Azure.Region == "" {
-		// Managed environment for Azure does not provide Region, but we need this for the CredentialsRequest.
-		// This looks like an oversight when creating the UI in OpenShift, but for now we need to pull this data
-		// from somewhere else -> the Azure Storage Secret
-		storageSecretName := client.ObjectKey{
-			Namespace: stack.Namespace,
-			Name:      stack.Spec.Storage.Secret.Name,
-		}
-		storageSecret := &corev1.Secret{}
-		if err := k.Get(ctx, storageSecretName, storageSecret); err != nil {
-			if apierrors.IsNotFound(err) {
-				// Skip this error here as it will be picked up by the LokiStack handler instead
-				ll.Error(err, "could not find secret for LokiStack", "name", req.String())
-				return nil
-			}
-			return err
-		}
-
-		region := storageSecret.Data[storage.KeyAzureRegion]
-		if len(region) == 0 {
-			return errAzureNoRegion
-		}
-
-		managedAuth.Azure.Region = string(region)
-	}
-
 	opts := openshift.Options{
 		BuildOpts: openshift.BuildOptions{
 			LokiStackName:      stack.Name,

operator/internal/handlers/credentialsrequest_create_test.go → operator/internal/handlers/credentialsrequest_test.go

@@ -6,7 +6,6 @@ import (
 
 	cloudcredentialv1 "github.com/openshift/cloud-credential-operator/pkg/apis/cloudcredential/v1"
 	"github.com/stretchr/testify/require"
-	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime/schema"
@@ -19,7 +18,7 @@ import (
 	"github.com/grafana/loki/operator/internal/external/k8s/k8sfakes"
 )
 
-func credentialsRequestFakeClient(cr *cloudcredentialv1.CredentialsRequest, lokistack *lokiv1.LokiStack, secret *corev1.Secret) *k8sfakes.FakeClient {
+func credentialsRequestFakeClient(cr *cloudcredentialv1.CredentialsRequest, lokistack *lokiv1.LokiStack) *k8sfakes.FakeClient {
 	k := &k8sfakes.FakeClient{}
 	k.GetStub = func(_ context.Context, name types.NamespacedName, object client.Object, _ ...client.GetOption) error {
 		switch object.(type) {
@@ -33,11 +32,6 @@ func credentialsRequestFakeClient(cr *cloudcredentialv1.CredentialsRequest, loki
 				return errors.NewNotFound(schema.GroupResource{}, name.Name)
 			}
 			k.SetClientObject(object, lokistack)
-		case *corev1.Secret:
-			if secret == nil {
-				return errors.NewNotFound(schema.GroupResource{}, name.Name)
-			}
-			k.SetClientObject(object, secret)
 		}
 		return nil
 	}
@@ -58,7 +52,7 @@ func TestCreateCredentialsRequest_CreateNewResource(t *testing.T) {
 		},
 	}
 
-	k := credentialsRequestFakeClient(nil, lokistack, nil)
+	k := credentialsRequestFakeClient(nil, lokistack)
 	req := ctrl.Request{
 		NamespacedName: client.ObjectKey{Name: "my-stack", Namespace: "ns"},
 	}
@@ -89,13 +83,8 @@ func TestCreateCredentialsRequest_CreateNewResourceAzure(t *testing.T) {
 			Namespace: "ns",
 		},
 	}
-	secret := &corev1.Secret{
-		Data: map[string][]byte{
-			"region": []byte(wantRegion),
-		},
-	}
 
-	k := credentialsRequestFakeClient(nil, lokistack, secret)
+	k := credentialsRequestFakeClient(nil, lokistack)
 	req := ctrl.Request{
 		NamespacedName: client.ObjectKey{Name: "my-stack", Namespace: "ns"},
 	}
@@ -105,6 +94,7 @@ func TestCreateCredentialsRequest_CreateNewResourceAzure(t *testing.T) {
 			ClientID:       "test-client-id",
 			SubscriptionID: "test-tenant-id",
 			TenantID:       "test-subscription-id",
+			Region:         "test-region",
 		},
 	}
 
@@ -122,47 +112,6 @@ func TestCreateCredentialsRequest_CreateNewResourceAzure(t *testing.T) {
 	require.Equal(t, wantRegion, providerSpec.AzureRegion)
 }
 
-func TestCreateCredentialsRequest_CreateNewResourceAzure_Errors(t *testing.T) {
-	lokistack := &lokiv1.LokiStack{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      "my-stack",
-			Namespace: "ns",
-		},
-	}
-	req := ctrl.Request{
-		NamespacedName: client.ObjectKey{Name: "my-stack", Namespace: "ns"},
-	}
-
-	tt := []struct {
-		secret    *corev1.Secret
-		wantError string
-	}{
-		{
-			secret:    &corev1.Secret{},
-			wantError: errAzureNoRegion.Error(),
-		},
-	}
-
-	for _, tc := range tt {
-		tc := tc
-		t.Run(tc.wantError, func(t *testing.T) {
-			t.Parallel()
-
-			managedAuth := &config.ManagedAuthConfig{
-				Azure: &config.AzureEnvironment{
-					ClientID:       "test-client-id",
-					SubscriptionID: "test-tenant-id",
-					TenantID:       "test-subscription-id",
-				},
-			}
-			k := credentialsRequestFakeClient(nil, lokistack, tc.secret)
-
-			err := CreateCredentialsRequest(context.Background(), logger, scheme, managedAuth, k, req)
-			require.EqualError(t, err, tc.wantError)
-		})
-	}
-}
-
 func TestCreateCredentialsRequest_DoNothing_WhenCredentialsRequestExist(t *testing.T) {
 	req := ctrl.Request{
 		NamespacedName: client.ObjectKey{Name: "my-stack", Namespace: "ns"},
@@ -187,7 +136,7 @@ func TestCreateCredentialsRequest_DoNothing_WhenCredentialsRequestExist(t *testi
 		},
 	}
 
-	k := credentialsRequestFakeClient(cr, lokistack, nil)
+	k := credentialsRequestFakeClient(cr, lokistack)
 
 	err := CreateCredentialsRequest(context.Background(), logger, scheme, managedAuth, k, req)
 	require.NoError(t, err)

operator/internal/handlers/internal/storage/secrets.go

@@ -182,18 +182,11 @@ func extractAzureConfigSecret(s *corev1.Secret, fg configv1.FeatureGates) (*stor
 	// Extract and validate optional fields
 	endpointSuffix := s.Data[storage.KeyAzureStorageEndpointSuffix]
 	audience := s.Data[storage.KeyAzureAudience]
-	region := s.Data[storage.KeyAzureRegion]
 
 	if !workloadIdentity && len(audience) > 0 {
 		return nil, fmt.Errorf("%w: %s", errSecretFieldNotAllowed, storage.KeyAzureAudience)
 	}
 
-	if fg.OpenShift.ManagedAuthEnv {
-		if len(region) == 0 {
-			return nil, fmt.Errorf("%w: %s", errSecretMissingField, storage.KeyAzureRegion)
-		}
-	}
-
 	return &storage.AzureStorageConfig{
 		Env:              string(env),
 		Container:        string(container),

operator/internal/handlers/internal/storage/secrets_test.go

@@ -156,27 +156,6 @@ func TestAzureExtract(t *testing.T) {
 			},
 			wantError: "missing secret field: subscription_id",
 		},
-		{
-			name: "managed auth - no region",
-			secret: &corev1.Secret{
-				ObjectMeta: metav1.ObjectMeta{Name: "test"},
-				Data: map[string][]byte{
-					"environment":  []byte("here"),
-					"account_name": []byte("test-account-name"),
-					"container":    []byte("this,that"),
-				},
-			},
-			managedSecret: &corev1.Secret{
-				Data: map[string][]byte{},
-			},
-			featureGates: configv1.FeatureGates{
-				OpenShift: configv1.OpenShiftFeatureGates{
-					Enabled:        true,
-					ManagedAuthEnv: true,
-				},
-			},
-			wantError: "missing secret field: region",
-		},
 		{
 			name: "managed auth - no auth override",
 			secret: &corev1.Secret{

operator/internal/manifests/openshift/credentialsrequest.go

@@ -12,6 +12,8 @@ import (
 	"github.com/grafana/loki/operator/internal/manifests/storage"
 )
 
+const azureFallbackRegion = "centralus"
+
 func BuildCredentialsRequest(opts Options) (*cloudcredentialv1.CredentialsRequest, error) {
 	stack := client.ObjectKey{Name: opts.BuildOpts.LokiStackName, Namespace: opts.BuildOpts.LokiStackNamespace}
 
@@ -62,6 +64,15 @@ func encodeProviderSpec(env *config.ManagedAuthConfig) (*runtime.RawExtension, e
 		}
 	case env.Azure != nil:
 		azure := env.Azure
+		if azure.Region == "" {
+			// The OpenShift Console currently does not provide a UI to configure the Azure Region
+			// for an operator using managed credentials. Because the CredentialsRequest is currently
+			// not used to create a Managed Identity, the region is actually never used.
+			// We default to the US region if nothing is set, so that the CredentialsRequest can be
+			// created. This should have no effect on the generated credential secret.
+			// The region can be configured by setting an environment variable on the operator Subscription.
+			azure.Region = azureFallbackRegion
+		}
 
 		spec = &cloudcredentialv1.AzureProviderSpec{
 			Permissions: []string{

operator/internal/manifests/storage/options.go

@@ -63,7 +63,6 @@ type AzureStorageConfig struct {
 	Container        string
 	EndpointSuffix   string
 	Audience         string
-	Region           string
 	WorkloadIdentity bool
 }
 

operator/internal/manifests/storage/var.go

@@ -88,8 +88,6 @@ const (
 	KeyAzureStorageEndpointSuffix = "endpoint_suffix"
 	// KeyAzureEnvironmentName is the secret data key for the Azure cloud environment name.
 	KeyAzureEnvironmentName = "environment"
-	// KeyAzureRegion is the secret data key for storing the Azure cloud region.
-	KeyAzureRegion = "region"
 	// KeyAzureAudience is the secret data key for customizing the audience used for the ServiceAccount token.
 	KeyAzureAudience = "audience"