Resolved conflicts

Signed-off-by: Sheikh-Abubaker <[email protected]>

.github/workflows/checks.yml

@@ -1,5 +1,8 @@
 name: Checks
-on: [push]
+on:
+  pull_request:
+    branches:    
+      - main
 jobs:
   checks:
     runs-on: ubuntu-latest
@@ -10,6 +13,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@08e2f20817b15149a52b5b3ebe7de50aff2ba8c5
+        with:
+          version: v1.55.1
+          only-new-issues: true
       - run: make lint
       - run: make check-doc
       - run: make check-mod

CHANGELOG.md

@@ -53,6 +53,7 @@
 * [11679](https://github.com/grafana/loki/pull/11679) **dannykopping** Cache: extending #11535 to align custom ingester query split with cache keys for correct caching of results.
 * [11143](https://github.com/grafana/loki/pull/11143) **sandeepsukhani** otel: Add support for per tenant configuration for mapping otlp data to loki format
 * [11499](https://github.com/grafana/loki/pull/11284) **jmichalek132** Config: Adds `frontend.log-query-request-headers` to enable logging of request headers in query logs.
+* [11817](https://github.com/grafana/loki/pull/11817) **ashwanthgoli** Ruler: Add support for filtering results of `/prometheus/api/v1/rules` endpoint by rule_name, rule_group, file and type.
 
 ##### Fixes
 * [11074](https://github.com/grafana/loki/pull/11074) **hainenber** Fix panic in lambda-promtail due to mishandling of empty DROP_LABELS env var.

docs/sources/configure/_index.md

@@ -3143,14 +3143,22 @@ shard_streams:
 
 # OTLP log ingestion configurations
 otlp_config:
+  # Configuration for resource attributes to store them as index labels or
+  # Structured Metadata or drop them altogether
   resource_attributes:
-    [ignore_defaults: <boolean>]
+    # Configure whether to ignore the default list of resource attributes to be
+    # stored as index labels and only use the given resource attributes config
+    [ignore_defaults: <boolean> | default = false]
 
-    [attributes: <list of AttributesConfigs>]
+    [attributes_config: <list of attributes_configs>]
 
-  [scope_attributes: <list of AttributesConfigs>]
+  # Configuration for scope attributes to store them as Structured Metadata or
+  # drop them altogether
+  [scope_attributes: <list of attributes_configs>]
 
-  [log_attributes: <list of AttributesConfigs>]
+  # Configuration for log attributes to store them as Structured Metadata or
+  # drop them altogether
+  [log_attributes: <list of attributes_configs>]
 ```
 
 ### frontend_worker
@@ -4577,7 +4585,7 @@ chunks:
   [tags: <map of string to string>]
 
 # How many shards will be created. Only used if schema is v10 or greater.
-[row_shards: <int>]
+[row_shards: <int> | default = 16]
 ```
 
 ### aws_storage_config
@@ -5292,6 +5300,24 @@ Named store from this example can be used by setting object_store to store-1 in
 [cos: <map of string to cos_storage_config>]
 ```
 
+### attributes_config
+
+Define actions for matching OpenTelemetry (OTEL) attributes.
+
+```yaml
+# Configures action to take on matching attributes. It allows one of
+# [structured_metadata, drop] for all attribute types. It additionally allows
+# index_label action for resource attributes
+[action: <string> | default = ""]
+
+# List of attributes to configure how to store them or drop them altogether
+[attributes: <list of strings>]
+
+# Regex to choose attributes to configure how to store them or drop them
+# altogether
+[regex: <Regexp>]
+```
+
 ## Runtime Configuration file
 
 Loki has a concept of "runtime config" file, which is simply a file that is reloaded while Loki is running. It is used by some Loki components to allow operator to change some aspects of Loki configuration without restarting it. File is specified by using `-runtime-config.file=<filename>` flag and reload period (which defaults to 10 seconds) can be changed by `-runtime-config.reload-period=<duration>` flag. Previously this mechanism was only used by limits overrides, and flags were called `-limits.per-user-override-config=<filename>` and `-limits.per-user-override-period=10s` respectively. These are still used, if `-runtime-config.file=<filename>` is not specified.
@@ -5345,7 +5371,8 @@ place in the `limits_config` section:
 configure a runtime configuration file:
 
     ```
-    runtime_config: overrides.yaml
+    runtime_config:
+      file: overrides.yaml
     ```
 
     In the `overrides.yaml` file, add `unordered_writes` for each tenant

docs/sources/configure/index.template

@@ -152,7 +152,8 @@ place in the `limits_config` section:
 configure a runtime configuration file:
 
     ```
-    runtime_config: overrides.yaml
+    runtime_config:
+      file: overrides.yaml
     ```
 
     In the `overrides.yaml` file, add `unordered_writes` for each tenant

docs/sources/get-started/labels/structured-metadata.md

@@ -5,25 +5,19 @@ description: Describes how to enable structure metadata for logs and how to quer
 ---
 # What is structured metadata
 
-{{% admonition type="warning" %}}
-Structured metadata is an experimental feature and is subject to change in future releases of Grafana Loki.
-{{% /admonition %}}
-
 {{% admonition type="warning" %}}
 Structured metadata was added to chunk format V4 which is used if the schema version is greater or equal to `13`. (See [Schema Config]({{< relref "../../storage#schema-config" >}}) for more details about schema versions. )
 {{% /admonition %}}
 
-One of the powerful features of Loki is parsing logs at query time to extract metadata and build labels out of it.
-However, the parsing of logs at query time comes with a cost which can be significantly high for, as an example,
-large JSON blobs or a poorly written query using complex regex patterns.
+Selecting proper, low cardinality labels is critical to operating and querying Loki effectively. Some metadata, especially infrastructure related metadata, can be difficult to embed in log lines, and is too high cardinality to effectively store as indexed labels (and therefore reducing performance of the index).
 
-In addition, the data extracted from logs at query time is usually high cardinality, which can’t be stored
-in the index as it would increase the cardinality too much, and therefore reduce the performance of the index.
-
-Structured metadata is a way to attach metadata to logs without indexing them. Examples of useful metadata are
-trace IDs, user IDs, and any other label that is often used in queries but has high cardinality and is expensive
+Structured metadata is a way to attach metadata to logs without indexing them or including them in the log line content itself. Examples of useful metadata are
+kubernetes pod names, process ID's, or any other label that is often used in queries but has high cardinality and is expensive
 to extract at query time.
 
+Structured metadata can also be used to query commonly needed metadata from log lines without needing to apply a parser at query time. Large json blobs or a poorly written query using complex regex patterns, for example, come with a high performance cost. Examples of useful metadata include trace IDs or user IDs.
+
+
 ## Attaching structured metadata to log lines
 
 You have the option to attach structured metadata to log lines in the push payload along with each log line and the timestamp.
@@ -34,25 +28,37 @@ See the [Promtail: Structured metadata stage]({{< relref "../../send-data/promta
 
 With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash]({{< relref "../../send-data/logstash/_index.md" >}}).
 
+{{% admonition type="warning" %}}
+There are defaults for how much structured metadata can be attached per log line.
+```
+# Maximum size accepted for structured metadata per log line.
+# CLI flag: -limits.max-structured-metadata-size
+[max_structured_metadata_size: <int> | default = 64KB]
+
+# Maximum number of structured metadata entries per log line.
+# CLI flag: -limits.max-structured-metadata-entries-count
+[max_structured_metadata_entries_count: <int> | default = 128]
+```
+{{% /admonition %}}
+
 ## Querying structured metadata
 
 Structured metadata is extracted automatically for each returned log line and added to the labels returned for the query.
 You can use labels of structured metadata to filter log line using a [label filter expression]({{< relref "../../query/log_queries#label-filter-expression" >}}).
 
-For example, if you have a label `trace_id` attached to some of your log lines as structured metadata, you can filter log lines using:
+For example, if you have a label `pod` attached to some of your log lines as structured metadata, you can filter log lines using:
 
 ```logql
-{job="example"} | trace_id="0242ac120002"
+{job="example"} | pod="myservice-abc1234-56789"`
 ```
 
 Of course, you can filter by multiple labels of structured metadata at the same time:
 
 ```logql
-{job="example"} | trace_id="0242ac120002" | user_id="superUser123"
+{job="example"} | pod="myservice-abc1234-56789" | trace_id="0242ac120002"
 ```
 
-Note that since structured metadata is extracted automatically to the results labels, some metric queries might return
-an error like `maximum of series (50000) reached for a single query`. You can use the [Keep]({{< relref "../../query/log_queries#keep-labels-expression" >}}) and [Drop]({{< relref "../../query/log_queries#drop-labels-expression" >}}) stages to filter out labels that you don't need.
+Note that since structured metadata is extracted automatically to the results labels, some metric queries might return an error like `maximum of series (50000) reached for a single query`. You can use the [Keep]({{< relref "../../query/log_queries#keep-labels-expression" >}}) and [Drop]({{< relref "../../query/log_queries#drop-labels-expression" >}}) stages to filter out labels that you don't need.
 For example:
 
 ```logql

docs/sources/operations/query-fairness/_index.md

@@ -115,7 +115,7 @@ you would usually want to avoid this scenario and control yourself where the hea
 
 When using Grafana as the Loki user interface, you can, for example, create multiple data sources
 with the same tenant, but with a different additional HTTP header
-`X-Loki-Scope-Actor` and restrict which Grafana user can use which data source.
+`X-Loki-Actor-Path` and restrict which Grafana user can use which data source.
 
 Alternatively, if you have a proxy for authentication in front of Loki, you can
 pass the (hashed) user from the authentication as downstream header to Loki.

docs/sources/reference/api.md

@@ -1178,11 +1178,15 @@ Deletes all the rule groups in a namespace (including the namespace itself). Thi
 ### List rules
 
 ```
-GET /prometheus/api/v1/rules
+GET /prometheus/api/v1/rules?type={alert|record}&file={}&rule_group={}&rule_name={}
 ```
 
 Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded.
 
+The `type` parameter is optional. If set, only the specified type of rule is returned.
+
+The `file`, `rule_group` and `rule_name` parameters are optional, and can accept multiple values. If set, the response content is filtered accordingly.
+
 For more information, refer to the [Prometheus rules](https://prometheus.io/docs/prometheus/latest/querying/api/#rules) documentation.
 
 ### List alerts

docs/sources/release-notes/v2-9.md

@@ -9,6 +9,8 @@ Grafana Labs is excited to announce the release of Loki 2.9.0 Here's a summary o
 
 ## Features and enhancements
 
+- **Structured metadata**: The [Structured Metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) feature, which was introduced as experimental in release 2.9.0, is generally available as of release 2.9.4.
+
 -  **Query Language Improvements**: Several improvements to the query language that speed up line parsing and regex matching. [PR #8646](https://github.com/grafana/loki/pull/8646), [PR #8659](https://github.com/grafana/loki/pull/8659), [PR #8724](https://github.com/grafana/loki/pull/8724), [PR #8734](https://github.com/grafana/loki/pull/8734), [PR #8739](https://github.com/grafana/loki/pull/8739), [PR #8763](https://github.com/grafana/loki/pull/8763), [PR #8890](https://github.com/grafana/loki/pull/8890), [PR #8914](https://github.com/grafana/loki/pull/8914)
 
 -  **Remote rule evaluation**: Rule evaluation can now be handled by queriers to improve speed. [PR #8744](https://github.com/grafana/loki/pull/8744) [PR #8848](https://github.com/grafana/loki/pull/8848)
@@ -33,13 +35,13 @@ Grafana Labs is excited to announce the release of Loki 2.9.0 Here's a summary o
 
 ## Bug fixes
 
-### 2.9.1 (2023-09-14)
-
-* Update Docker base images to mitigate security vulnerability CVE-2022-48174
-* Fix bugs in indexshipper (`tsdb`, `boltdb-shipper`) that could result in not showing all ingested logs in query results.
-
 ### 2.9.2 (2023-10-16)
 
 * Upgrade go to v1.21.3, golang.org/x/net to v0.17.0 and grpc-go to v1.56.3 to patch CVE-2023-39325 / CVE-2023-44487
 
 For a full list of all changes and fixes, look at the [CHANGELOG](https://github.com/grafana/loki/blob/release-2.9.x/CHANGELOG.md).
+
+### 2.9.1 (2023-09-14)
+
+* Update Docker base images to mitigate security vulnerability CVE-2022-48174
+* Fix bugs in indexshipper (`tsdb`, `boltdb-shipper`) that could result in not showing all ingested logs in query results.

docs/sources/send-data/otel/_index.md

@@ -9,10 +9,6 @@ weight:  250
 
 # Ingesting logs to Loki using OpenTelemetry Collector
 
-{{% admonition type="warning" %}}
-OpenTelemetry logs ingestion is an experimental feature and is subject to change in future releases of Grafana Loki.
-{{% /admonition %}}
-
 Loki natively supports ingesting OpenTelemetry logs over HTTP.
 For ingesting logs to Loki using the OpenTelemetry Collector, you must use the [`otlphttp` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter).
 
@@ -69,7 +65,7 @@ service:
 
 ## Format considerations
 
-Since the OpenTelemetry protocol  differs from the Loki storage model, here is how data in the OpenTelemetry format will be mapped to the Loki data model during ingestion:
+Since the OpenTelemetry protocol differs from the Loki storage model, here is how data in the OpenTelemetry format will be mapped by default to the Loki data model during ingestion, which can be changed as explained later:
 
 - Index labels: Resource attributes map well to index labels in Loki, since both usually identify the source of the logs. Because Loki has a limit of 30 index labels, we have selected the following resource attributes to be stored as index labels, while the remaining attributes are stored as [Structured Metadata]({{< relref "../../get-started/labels/structured-metadata" >}}) with each log entry:
   - cloud.availability_zone
@@ -116,3 +112,102 @@ Things to note before ingesting OpenTelemetry logs to Loki:
 - Stringification of non-string Attribute values
 
   While converting Attribute values in OTLP to Index label values or Structured Metadata, any non-string values are converted to string using [AsString method from the OTEL collector lib](https://github.com/open-telemetry/opentelemetry-collector/blob/ab3d6c5b64701e690aaa340b0a63f443ff22c1f0/pdata/pcommon/value.go#L353).
+
+### Changing the default mapping of OTLP to Loki Format
+
+Loki supports [per tenant]({{< relref "../../configure#limits_config" >}}) OTLP config which lets you change the default mapping of OTLP to Loki format for each tenant.
+It currently only supports changing the storage of Attributes. Here is how the config looks like:
+
+```yaml
+# OTLP log ingestion configurations
+otlp_config:
+  # Configuration for Resource Attributes to store them as index labels or
+  # Structured Metadata or drop them altogether
+  resource_attributes:
+    # Configure whether to ignore the default list of Resource Attributes to be
+    # stored as Index Labels and only use the given Resource Attributes config
+    [ignore_defaults: <boolean>]
+
+    [attributes_config: <list of attributes_configs>]
+
+  # Configuration for Scope Attributes to store them as Structured Metadata or
+  # drop them altogether
+  [scope_attributes: <list of attributes_configs>]
+
+  # Configuration for Log Attributes to store them as Structured Metadata or
+  # drop them altogether
+  [log_attributes: <list of attributes_configs>]
+
+attributes_config:
+  # Configures action to take on matching Attributes. It allows one of
+  # [structured_metadata, drop] for all Attribute types. It additionally allows
+  # index_label action for Resource Attributes
+  [action: <string> | default = ""]
+
+  # List of attributes to configure how to store them or drop them altogether
+  [attributes: <list of strings>]
+
+  # Regex to choose attributes to configure how to store them or drop them
+  # altogether
+  [regex: <Regexp>]
+```
+
+Here are some example configs to change the default mapping of OTLP to Loki format:
+
+#### Example 1:
+
+```yaml
+otlp_config:
+  resource_attributes:
+    attributes_config:
+      - action: index_label
+        attributes:
+          - service.group
+```
+
+With the example config, here is how various kinds of Attributes would be stored:
+* Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
+* Store remaining Resource Attributes as Structured Metadata.
+* Store all the Scope and Log Attributes as Structured Metadata.
+
+#### Example 2:
+
+```yaml
+otlp_config:
+  resource_attributes:
+    ignore_defaults: true
+    attributes_config:
+      - action: index_label
+        regex: service.group
+```
+
+With the example config, here is how various kinds of Attributes would be stored:
+* **Only** store `service.group` Resource Attribute as index labels.
+* Store remaining Resource Attributes as Structured Metadata.
+* Store all the Scope and Log Attributes as Structured Metadata.
+
+#### Example 2:
+
+```yaml
+otlp_config:
+  resource_attributes:
+    attributes_config:
+      - action: index_label
+        regex: service.group
+  scope_attributes:
+    - action: drop
+      attributes:
+        - method.name
+  log_attributes:
+    - action: structured_metadata
+      attributes:
+        - user.id
+    - action: drop
+      regex: .*
+```
+
+With the example config, here is how various kinds of Attributes would be stored:
+* Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
+* Store remaining Resource Attributes as Structured Metadata.
+* Drop Scope Attribute named `method.name` and store all other Scope Attributes as Structured Metadata.
+* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.

docs/sources/setup/install/docker.md

@@ -9,7 +9,7 @@ weight: 400
 # Install Loki with Docker or Docker Compose
 
 You can install Loki and Promtail with Docker or Docker Compose if you are evaluating, testing, or developing Loki.
-For production, we recommend installing with Tanka or Helm.
+For production, Grafana recommends installing with Tanka or Helm.
 
 The configuration acquired with these installation instructions run Loki as a single binary.
 

docs/sources/setup/install/helm/reference.md

@@ -856,7 +856,7 @@ false
 			<td>string</td>
 			<td></td>
 			<td><pre lang="json">
-"v1.8.4"
+"v1.8.6"
 </pre>
 </td>
 		</tr>

go.mod

@@ -119,7 +119,6 @@ require (
 	github.com/DmitriyVTitov/size v1.5.0
 	github.com/IBM/go-sdk-core/v5 v5.13.1
 	github.com/IBM/ibm-cos-sdk-go v1.10.0
-	github.com/aws/smithy-go v1.11.1
 	github.com/axiomhq/hyperloglog v0.0.0-20230201085229-3ddf4bad03dc
 	github.com/d4l3k/messagediff v1.2.1
 	github.com/efficientgo/core v1.0.0-rc.2
@@ -183,6 +182,7 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.1 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sso v1.11.1 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sts v1.16.1 // indirect
+	github.com/aws/smithy-go v1.11.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
 	github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe // indirect