diff --git a/docs/sources/alert/_index.md b/docs/sources/alert/_index.md index e1e857bb84280..0a9ec4c8419f6 100644 --- a/docs/sources/alert/_index.md +++ b/docs/sources/alert/_index.md @@ -167,7 +167,7 @@ ruler: url: http://localhost:9090/api/v1/write ``` -Further configuration options can be found under [ruler]({{< relref "../configure#ruler" >}}). +Further configuration options can be found under [ruler](https://grafana.com/docs/loki//configure/#ruler). ### Operations diff --git a/docs/sources/configure/bp-configure.md b/docs/sources/configure/bp-configure.md index 4765fb299fb0e..b31b9de0e34b1 100644 --- a/docs/sources/configure/bp-configure.md +++ b/docs/sources/configure/bp-configure.md @@ -14,7 +14,7 @@ Loki can cache data at many levels, which can drastically improve performance. D ## Time ordering of logs -Loki [accepts out-of-order writes]({{< relref "../configure#accept-out-of-order-writes" >}}) _by default_. +Loki [accepts out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes) _by default_. This section identifies best practices when Loki is _not_ configured to accept out-of-order writes. One issue many people have with Loki is their client receiving errors for out of order log entries. This happens because of this hard and fast rule within Loki: diff --git a/docs/sources/get-started/components.md b/docs/sources/get-started/components.md index 886556b635075..9d48a1d5381c7 100644 --- a/docs/sources/get-started/components.md +++ b/docs/sources/get-started/components.md @@ -35,7 +35,9 @@ Currently the only way the distributor mutates incoming data is by normalizing l The distributor can also rate limit incoming logs based on the maximum per-tenant bitrate. It does this by checking a per tenant limit and dividing it by the current number of distributors. This allows the rate limit to be specified per tenant at the cluster level and enables us to scale the distributors up or down and have the per-distributor limit adjust accordingly. For instance, say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/second before limiting. Now, say another large tenant joins the cluster and we need to spin up 10 more distributors. The now 20 distributors will adjust their rate limits for tenant A to `(10MB / 20 distributors) = 500KB/s`! This is how global limits allow much simpler and safer operation of the Loki cluster. -**Note: The distributor uses the `ring` component under the hood to register itself amongst its peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring configuration]({{< relref "../configure#distributor" >}}).** +{{% admonition type="note" %}} +The distributor uses the `ring` component under the hood to register itself amongst its peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring configuration](https://grafana.com/docs/loki//configure/#distributor). +{{% /admonition %}} ### Forwarding @@ -142,7 +144,7 @@ deduplicated. ### Timestamp Ordering -Loki is configured to [accept out-of-order writes]({{< relref "../configure#accept-out-of-order-writes" >}}) by default. +Loki is configured to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes) by default. When not configured to accept out-of-order writes, the ingester validates that ingested log lines are in order. When an ingester receives a log line that doesn't follow the expected order, the line diff --git a/docs/sources/get-started/deployment-modes.md b/docs/sources/get-started/deployment-modes.md index 639a22d5f0a4b..e8ce9d36be4c4 100644 --- a/docs/sources/get-started/deployment-modes.md +++ b/docs/sources/get-started/deployment-modes.md @@ -52,7 +52,7 @@ The simplest mode of operation is the monolithic deployment mode. You enable mon Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day. -You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the [`ring` section]({{< relref "../configure#common" >}}) of the `loki.yaml` file to share state between all instances, but the recommendation is to use simple scalable mode if you need to scale your deployment. +You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the [`ring` section](https://grafana.com/docs/loki//configure/#common) of the `loki.yaml` file to share state between all instances, but the recommendation is to use simple scalable mode if you need to scale your deployment. You can configure high availability by running two Loki instances using `memberlist_config` configuration and a shared object store and setting the `replication_factor` to `3`. You route traffic to all the Loki instances in a round robin fashion. diff --git a/docs/sources/get-started/hash-rings.md b/docs/sources/get-started/hash-rings.md index e136b0b9136b1..e7b53066a9733 100644 --- a/docs/sources/get-started/hash-rings.md +++ b/docs/sources/get-started/hash-rings.md @@ -53,7 +53,7 @@ For each node, the key-value store holds: ## Configuring rings -Define [ring configuration]({{< relref "../configure#common" >}}) within the `common.ring_config` block. +Define [ring configuration](https://grafana.com/docs/loki//configure/#common) within the `common.ring_config` block. Use the default `memberlist` key-value store type unless there is a compelling reason to use a different key-value store type. diff --git a/docs/sources/operations/automatic-stream-sharding.md b/docs/sources/operations/automatic-stream-sharding.md index 90372aff964f3..f5b27eb9ec047 100644 --- a/docs/sources/operations/automatic-stream-sharding.md +++ b/docs/sources/operations/automatic-stream-sharding.md @@ -12,20 +12,28 @@ existing streams. When properly tuned, this should eliminate issues where log pr per-stream rate limit. **To enable automatic stream sharding:** -1. Edit the global [limits_config]({{< relref "../configure#limits_config" >}}) of the Loki configuration file: +1. Edit the global [`limits_config`](https://grafana.com/docs/loki//configure/#limits_config) of the Loki configuration file: + ```yaml limits_config: shard_streams: enabled: true ``` -2. Optionally lower the `desired_rate` in bytes if you find that the system is still hitting the `per_stream_rate_limit`: + +1. Optionally lower the `desired_rate` in bytes if you find that the system is still hitting the `per_stream_rate_limit`: + ```yaml limits_config: shard_streams: enabled: true desired_rate: 2097152 #2MiB ``` -3. Optionally enable `logging_enabled` for debugging stream sharding. **Note**: this may affect the ingestion performance of Loki. + +1. Optionally enable `logging_enabled` for debugging stream sharding. + {{% admonition type="note" %}} + This may affect the ingestion performance of Loki. + {{% /admonition %}} + ```yaml limits_config: shard_streams: diff --git a/docs/sources/operations/blocking-queries.md b/docs/sources/operations/blocking-queries.md index 42e7a10bcd14f..97352c5b95785 100644 --- a/docs/sources/operations/blocking-queries.md +++ b/docs/sources/operations/blocking-queries.md @@ -9,7 +9,7 @@ In certain situations, you may not be able to control the queries being sent to may be intentionally or unintentionally expensive to run, and they may affect the overall stability or cost of running your service. -You can block queries using [per-tenant overrides]({{< relref "../configure#runtime-configuration-file" >}}), like so: +You can block queries using [per-tenant overrides](https://grafana.com/docs/loki//configure/#runtime-configuration-file), like so: ```yaml overrides: @@ -34,8 +34,9 @@ overrides: - hash: 2943214005 # hash of {stream="stdout",pod="loki-canary-9w49x"} types: filter,limited ``` - -NOTE: changes to these configurations **do not require a restart**; they are defined in the [runtime configuration file]({{< relref "../configure#runtime-configuration-file" >}}). +{{% admonition type="note" %}} +Changes to these configurations **do not require a restart**; they are defined in the [runtime configuration file](https://grafana.com/docs/loki//configure/#runtime-configuration-file). +{{% /admonition %}} The available query types are: diff --git a/docs/sources/operations/overrides-exporter.md b/docs/sources/operations/overrides-exporter.md index 6fe4c5aaac504..3f510ea1aa062 100644 --- a/docs/sources/operations/overrides-exporter.md +++ b/docs/sources/operations/overrides-exporter.md @@ -10,7 +10,7 @@ Loki is a multi-tenant system that supports applying limits to each tenant as a ## Context -Configuration updates to tenant limits can be applied to Loki without restart via the [`runtime_config`]({{< relref "../configure#runtime_config" >}}) feature. +Configuration updates to tenant limits can be applied to Loki without restart via the [`runtime_config`](https://grafana.com/docs/loki//configure/#runtime_config) feature. ## Example diff --git a/docs/sources/operations/recording-rules.md b/docs/sources/operations/recording-rules.md index 476a151bb3df6..8ebfcd7fa53c4 100644 --- a/docs/sources/operations/recording-rules.md +++ b/docs/sources/operations/recording-rules.md @@ -7,7 +7,7 @@ description: Working with recording rules. Recording rules are evaluated by the `ruler` component. Each `ruler` acts as its own `querier`, in the sense that it executes queries against the store without using the `query-frontend` or `querier` components. It will respect all query -[limits]({{< relref "../configure#limits_config" >}}) put in place for the `querier`. +[limits](https://grafana.com/docs/loki//configure/#limits_config) put in place for the `querier`. Loki's implementation of recording rules largely reuses Prometheus' code. @@ -70,8 +70,8 @@ so a `Persistent Volume` should be utilised. ### Per-Tenant Limits Remote-write can be configured at a global level in the base configuration, and certain parameters tuned specifically on -a per-tenant basis. Most of the configuration options [defined here]({{< relref "../configure#ruler" >}}) -have [override options]({{< relref "../configure#limits_config" >}}) (which can be also applied at runtime!). +a per-tenant basis. Most of the configuration options [defined here](https://grafana.com/docs/loki//configure/#ruler) +have [override options](https://grafana.com/docs/loki//configure/#limits_config) (which can be also applied at runtime!). ### Tuning diff --git a/docs/sources/operations/storage/boltdb-shipper.md b/docs/sources/operations/storage/boltdb-shipper.md index db772e9d7eee3..1fa81ae57c39a 100644 --- a/docs/sources/operations/storage/boltdb-shipper.md +++ b/docs/sources/operations/storage/boltdb-shipper.md @@ -105,14 +105,14 @@ Within Kubernetes, if you are not using an Index Gateway, we recommend running Q An Index Gateway downloads and synchronizes the BoltDB index from the Object Storage in order to serve index queries to the Queriers and Rulers over gRPC. This avoids running Queriers and Rulers with a disk for persistence. Disks can become costly in a big cluster. -To run an Index Gateway, configure [StorageConfig]({{< relref "../../configure#storage_config" >}}) and set the `-target` CLI flag to `index-gateway`. -To connect Queriers and Rulers to the Index Gateway, set the address (with gRPC port) of the Index Gateway with the `-boltdb.shipper.index-gateway-client.server-address` CLI flag or its equivalent YAML value under [StorageConfig]({{< relref "../../configure#storage_config" >}}). +To run an Index Gateway, configure [StorageConfig](https://grafana.com/docs/loki//configure/#storage_config) and set the `-target` CLI flag to `index-gateway`. +To connect Queriers and Rulers to the Index Gateway, set the address (with gRPC port) of the Index Gateway with the `-boltdb.shipper.index-gateway-client.server-address` CLI flag or its equivalent YAML value under [StorageConfig](https://grafana.com/docs/loki//configure/#storage_config). When using the Index Gateway within Kubernetes, we recommend using a StatefulSet with persistent storage for downloading and querying index files. This can obtain better read performance, avoids [noisy neighbor problems](https://en.wikipedia.org/wiki/Cloud_computing_issues#Performance_interference_and_noisy_neighbors) by not using the node disk, and avoids the time consuming index downloading step on startup after rescheduling to a new node. ### Write Deduplication disabled -Loki does write deduplication of chunks and index using Chunks and WriteDedupe cache respectively, configured with [ChunkStoreConfig]({{< relref "../../configure#chunk_store_config" >}}). +Loki does write deduplication of chunks and index using Chunks and WriteDedupe cache respectively, configured with [ChunkStoreConfig](https://grafana.com/docs/loki//configure/#chunk_store_config). The problem with write deduplication when using `boltdb-shipper` though is ingesters only keep uploading boltdb files periodically to make them available to all the other services which means there would be a brief period where some of the services would not have received updated index yet. The problem due to that is if an ingester which first wrote the chunks and index goes down and all the other ingesters which were part of replication scheme skipped writing those chunks and index due to deduplication, we would end up missing those logs from query responses since only the ingester which had the index went down. This problem would be faced even during rollouts which is quite common. diff --git a/docs/sources/operations/storage/schema/_index.md b/docs/sources/operations/storage/schema/_index.md index af0b644a23f3e..5a569b19644cc 100644 --- a/docs/sources/operations/storage/schema/_index.md +++ b/docs/sources/operations/storage/schema/_index.md @@ -12,6 +12,32 @@ Loki uses the defined schemas to determine which format to use when storing and Use of a schema allows Loki to iterate over the storage layer without requiring migration of existing data. +## New Loki installs +For a new Loki install with no previous data, here is an example schema configuration with recommended values + +``` +schema_config: + configs: + - from: 2024-04-01 + object_store: s3 + store: tsdb + schema: v13 + index: + prefix: index_ + period: 24h +``` + + +| Property | Description | +|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------| +| from | for a new install, this must be a date in the past, use a recent date. Format is YYYY-MM-DD. | +| object_store | s3, azure, gcs, alibabacloud, bos, cos, swift, filesystem, or a named_store (see [StorageConfig](https://grafana.com/docs/loki//configure/#storage_config)). | +| store | `tsdb` is the current and only recommended value for store. | +| schema | `v13` is the most recent schema and recommended value. | +| prefix: | any value without spaces is acceptable. | +| period: | must be `24h`. | + + ## Changing the schema Here are items to consider when changing the schema; if schema changes are not done properly, a scenario can be created which prevents data from being read. diff --git a/docs/sources/operations/storage/table-manager/_index.md b/docs/sources/operations/storage/table-manager/_index.md index ada43137117fc..0405cd74e17bc 100644 --- a/docs/sources/operations/storage/table-manager/_index.md +++ b/docs/sources/operations/storage/table-manager/_index.md @@ -39,7 +39,7 @@ to store chunks, are not managed by the Table Manager, and a custom bucket polic should be set to delete old data. For detailed information on configuring the Table Manager, refer to the -[`table_manager`]({{< relref "../../../configure#table_manager" >}}) +[`table_manager`](https://grafana.com/docs/loki//configure/#table_manager) section in the Loki configuration document. @@ -48,10 +48,10 @@ section in the Loki configuration document. A periodic table stores the index or chunk data relative to a specific period of time. The duration of the time range of the data stored in a single table and its storage type is configured in the -[`schema_config`]({{< relref "../../../configure#schema_config" >}}) configuration +[`schema_config`](https://grafana.com/docs/loki//configure/#schema_config) configuration block. -The [`schema_config`]({{< relref "../../../configure#schema_config" >}}) can contain +The [`schema_config`](https://grafana.com/docs/loki//configure/#schema_config) can contain one or more `configs`. Each config, defines the storage used between the day set in `from` (in the format `yyyy-mm-dd`) and the next config, or "now" in the case of the last schema config entry. @@ -105,7 +105,7 @@ order to make sure that the new table is ready once the current table end period is reached. The `creation_grace_period` property - in the -[`table_manager`]({{< relref "../../../configure#table_manager" >}}) +[`table_manager`](https://grafana.com/docs/loki//configure/#table_manager) configuration block - defines how long before a table should be created. @@ -149,7 +149,7 @@ documentation. A table can be active or inactive. A table is considered **active** if the current time is within the range: -- Table start period - [`creation_grace_period`]({{< relref "../../../configure#table_manager" >}}) +- Table start period - [`creation_grace_period`](https://grafana.com/docs/loki//configure/#table_manager) - Table end period + max chunk age (hardcoded to `12h`) ![active_vs_inactive_tables](./table-manager-active-vs-inactive-tables.png) diff --git a/docs/sources/operations/storage/tsdb.md b/docs/sources/operations/storage/tsdb.md index c7c11052b191a..1953b8ef65e26 100644 --- a/docs/sources/operations/storage/tsdb.md +++ b/docs/sources/operations/storage/tsdb.md @@ -75,7 +75,7 @@ We've added a user per-tenant limit called `tsdb_max_query_parallelism` in the ` ### Dynamic Query Sharding -Previously we would statically shard queries based on the index row shards configured [here]({{< relref "../../configure#period_config" >}}). +Previously we would statically shard queries based on the index row shards configured [here](https://grafana.com/docs/loki//configure/#period_config). TSDB does Dynamic Query Sharding based on how much data a query is going to be processing. We additionally store size(KB) and number of lines for each chunk in the TSDB index which is then used by the [Query Frontend]({{< relref "../../get-started/components#query-frontend" >}}) for planning the query. Based on our experience from operating many Loki clusters, we have configured TSDB to aim for processing 300-600 MBs of data per query shard. @@ -83,4 +83,4 @@ This means with TSDB we will be running more, smaller queries. ### Index Caching not required -TSDB is a compact and optimized format. Loki does not currently use an index cache for TSDB. If you are already using Loki with other index types, it is recommended to keep the index caching until all of your existing data falls out of [retention]({{< relref "./retention" >}}) or your configured `max_query_lookback` under [limits_config]({{< relref "../../configure#limits_config" >}}). After that, we suggest running without an index cache (it isn't used in TSDB). +TSDB is a compact and optimized format. Loki does not currently use an index cache for TSDB. If you are already using Loki with other index types, it is recommended to keep the index caching until all of your existing data falls out of [retention](https://grafana.com/docs/loki//operations/storage/retention/)) or your configured `max_query_lookback` under [limits_config](https://grafana.com/docs/loki//configure/#limits_config). After that, we suggest running without an index cache (it isn't used in TSDB). diff --git a/docs/sources/release-notes/v2-4.md b/docs/sources/release-notes/v2-4.md index 98c61a9d8c6eb..6465503e3df34 100644 --- a/docs/sources/release-notes/v2-4.md +++ b/docs/sources/release-notes/v2-4.md @@ -13,9 +13,9 @@ Loki 2.4 focuses on two items: ## Features and enhancements -* [**Loki no longer requires logs to be sent in perfect chronological order.**]({{< relref "../configure#accept-out-of-order-writes" >}}) Support for out of order logs is one of the most highly requested features for Loki. The strict ordering constraint has been removed. +* [**Loki no longer requires logs to be sent in perfect chronological order.**](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes) Support for out of order logs is one of the most highly requested features for Loki. The strict ordering constraint has been removed. * Scaling Loki is now easier with a hybrid deployment mode that falls between our single binary and our microservices. The [Simple scalable deployment]({{< relref "../get-started/deployment-modes" >}}) scales Loki with new `read` and `write` targets. Where previously you would have needed Kubernetes and the microservices approach to start tapping into Loki’s potential, it’s now possible to do this in a simpler way. -* The new [`common` section]({{< relref "../configure#common" >}}) results in a 70% smaller Loki configuration. Pair that with updated defaults and Loki comes out of the box with more appropriate defaults and limits. Check out the [example local configuration](https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml) as the new reference for running Loki. +* The new [`common` section](https://grafana.com/docs/loki//configure/#common) results in a 70% smaller Loki configuration. Pair that with updated defaults and Loki comes out of the box with more appropriate defaults and limits. Check out the [example local configuration](https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml) as the new reference for running Loki. * [**Recording rules**]({{< relref "../alert#recording-rules" >}}) are no longer an experimental feature. We've given them a more resilient implementation which leverages the existing write ahead log code in Prometheus. * The new [**Promtail Kafka Consumer**]({{< relref "../send-data/promtail/scraping#kafka" >}}) can easily get your logs out of Kafka and into Loki. * There are **nice LogQL enhancements**, thanks to the amazing Loki community. LogQL now has [group_left and group_right]({{< relref "../query#many-to-one-and-one-to-many-vector-matches" >}}). And, the `label_format` and `line_format` functions now support [working with dates and times]({{< relref "../query/template_functions#now" >}}). diff --git a/docs/sources/release-notes/v2-5.md b/docs/sources/release-notes/v2-5.md index ea7d015479a1c..9ffc947349a7d 100644 --- a/docs/sources/release-notes/v2-5.md +++ b/docs/sources/release-notes/v2-5.md @@ -14,7 +14,7 @@ It has been nearly 6 months since Loki 2.4 was released, and we’ve been busy m - **[Binary operations are now significantly faster](https://github.com/grafana/loki/pull/5317)**, taking full advantage of Loki's parallelism. - **[A new schema is available](https://github.com/grafana/loki/pull/5054)**, which uses more path prefixes to avoid rate limits on S3. - That same schema change **[was also added to the filesystem store](https://github.com/grafana/loki/pull/5291)**, which avoids using one directory to store every chunk. -- A new capability for **[hedging requests to storage](https://github.com/grafana/loki/pull/4826)** improves performance on highly parallelized queries. Refer to the [hedging configuration]({{< relref "../configure#storage_config" >}}) under the `storage_config` block for more information. +- A new capability for **[hedging requests to storage](https://github.com/grafana/loki/pull/4826)** improves performance on highly parallelized queries. Refer to the [hedging configuration](https://grafana.com/docs/loki//configure/#storage_config) under the `storage_config` block for more information. - Promtail has several new ways to ingest logs: - The **[ability to do service discovery and tailing directly from the Docker daemon](https://github.com/grafana/loki/pull/4911)**. - **[Fetching logs directly from Cloudflare](https://github.com/grafana/loki/pull/4813)**. diff --git a/docs/sources/send-data/fluentbit/_index.md b/docs/sources/send-data/fluentbit/_index.md index 5b92fd799353b..8903a57c56c4c 100644 --- a/docs/sources/send-data/fluentbit/_index.md +++ b/docs/sources/send-data/fluentbit/_index.md @@ -229,7 +229,7 @@ Buffering refers to the ability to store the records somewhere, and while they a The blocking state with some of the input plugins is not acceptable, because it can have an undesirable side effect on the part that generates the logs. Fluent Bit implements a buffering mechanism that is based on parallel processing. Therefore, it cannot send logs in order. There are two ways of handling the out-of-order logs: -- Configure Loki to [accept out-of-order writes]({{< relref "../../configure#accept-out-of-order-writes" >}}). +- Configure Loki to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes). - Configure the Loki output plugin to use the buffering mechanism based on [`dque`](https://github.com/joncrlsn/dque), which is compatible with the Loki server strict time ordering: diff --git a/docs/sources/send-data/fluentd/_index.md b/docs/sources/send-data/fluentd/_index.md index 9ba016c6f1a4d..d18a3bb13ef74 100644 --- a/docs/sources/send-data/fluentd/_index.md +++ b/docs/sources/send-data/fluentd/_index.md @@ -152,7 +152,7 @@ Use with the `remove_keys kubernetes` option to eliminate metadata from the log. ### Multi-worker usage -Loki enables out-of-order inserts by default; refer to [accept out-of-order writes]({{< relref "../../configure#accept-out-of-order-writes" >}}). +Loki enables out-of-order inserts by default; refer to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes). If out-of-order inserts are _disabled_, attempting to insert a log entry with an earlier timestamp after a log entry with identical labels but a later timestamp, the insert will fail with `HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order`. Therefore, in order to use this plugin in a multi worker Fluentd setup, you'll need to include the worker ID in the labels or otherwise [ensure log streams are always sent to the same worker](https://docs.fluentd.org/deployment/multi-process-workers#less-than-worker-n-greater-than-directive). For example, using [fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier): diff --git a/docs/sources/send-data/lambda-promtail/_index.md b/docs/sources/send-data/lambda-promtail/_index.md index 7c1daa2efde5c..93ec4b575884f 100644 --- a/docs/sources/send-data/lambda-promtail/_index.md +++ b/docs/sources/send-data/lambda-promtail/_index.md @@ -212,4 +212,4 @@ Instead we can pipeline Cloudwatch logs to a set of Promtails, which can mitigat 1) Using Promtail's push api along with the `use_incoming_timestamp: false` config, we let Promtail determine the timestamp based on when it ingests the logs, not the timestamp assigned by cloudwatch. Obviously, this means that we lose the origin timestamp because Promtail now assigns it, but this is a relatively small difference in a real time ingestion system like this. 2) In conjunction with (1), Promtail can coalesce logs across Cloudwatch log streams because it's no longer susceptible to out-of-order errors when combining multiple sources (lambda invocations). -One important aspect to keep in mind when running with a set of Promtails behind a load balancer is that we're effectively moving the cardinality problems from the number of log streams -> number of Promtails. If you have not configured Loki to [accept out-of-order writes]({{< relref "../../configure#accept-out-of-order-writes" >}}), you'll need to assign a Promtail-specific label on each Promtail so that you don't run into out-of-order errors when the Promtails send data for the same log groups to Loki. This can easily be done via a configuration like `--client.external-labels=promtail=${HOSTNAME}` passed to Promtail. +One important aspect to keep in mind when running with a set of Promtails behind a load balancer is that we're effectively moving the cardinality problems from the number of log streams -> number of Promtails. If you have not configured Loki to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes), you'll need to assign a Promtail-specific label on each Promtail so that you don't run into out-of-order errors when the Promtails send data for the same log groups to Loki. This can easily be done via a configuration like `--client.external-labels=promtail=${HOSTNAME}` passed to Promtail. diff --git a/docs/sources/send-data/promtail/_index.md b/docs/sources/send-data/promtail/_index.md index b1201137fc217..18e3b97214694 100644 --- a/docs/sources/send-data/promtail/_index.md +++ b/docs/sources/send-data/promtail/_index.md @@ -95,7 +95,7 @@ Important details are: to resume work from the last scraped line and process the rest of the remaining 55%. * Since decompression and pushing can be very fast, depending on the size of your compressed file Loki will rate-limit your ingestion. In that case you - might configure Promtail's [`limits` stage]({{< relref "./stages/limit" >}}) to slow the pace or increase [ingestion limits on Loki]({{< relref "../../configure#limits_config" >}}) + might configure Promtail's [`limits` stage](https://grafana.com/docs/loki//send-data/promtail/configuration/#limits_config) to slow the pace or increase [ingestion limits](https://grafana.com/docs/loki//configure/#limits_config) on Loki. * Log rotations on compressed files **are not supported as of now** (log rotation is fully supported for normal files), mostly because it requires us modifying Promtail to rely on file inodes instead of file names. If you'd like to see support for it, create a new diff --git a/docs/sources/send-data/promtail/stages/limit.md b/docs/sources/send-data/promtail/stages/limit.md index d70c7bc5a9989..e7a85f13bcd3a 100644 --- a/docs/sources/send-data/promtail/stages/limit.md +++ b/docs/sources/send-data/promtail/stages/limit.md @@ -14,7 +14,7 @@ The `limit` stage is a rate-limiting stage that throttles logs based on several ## Limit stage schema This pipeline stage places limits on the rate or burst quantity of log lines that Promtail pushes to Loki. -The concept of having distinct burst and rate limits mirrors the approach to limits that can be set for Loki's distributor component: `ingestion_rate_mb` and `ingestion_burst_size_mb`, as defined in [limits_config]({{< relref "../../../configure#limits_config" >}}). +The concept of having distinct burst and rate limits mirrors the approach to limits that can be set for Loki's distributor component: `ingestion_rate_mb` and `ingestion_burst_size_mb`, as defined in [limits_config](https://grafana.com/docs/loki//configure/#limits_config). ```yaml limit: diff --git a/docs/sources/send-data/promtail/troubleshooting/_index.md b/docs/sources/send-data/promtail/troubleshooting/_index.md index 16812228ef7e3..e35c89a172983 100644 --- a/docs/sources/send-data/promtail/troubleshooting/_index.md +++ b/docs/sources/send-data/promtail/troubleshooting/_index.md @@ -201,7 +201,7 @@ from there. This means that if new log entries have been read and pushed to the ingester between the last sync period and the crash, these log entries will be sent again to the ingester on Promtail restart. -If Loki is not configured to [accept out-of-order writes]({{< relref "../../../configure#accept-out-of-order-writes" >}}), Loki will reject all log lines received in +If Loki is not configured to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes), Loki will reject all log lines received in what it perceives is out of order. If Promtail happens to crash, it may re-send log lines that were sent prior to the crash. The default diff --git a/docs/sources/setup/migrate/migrate-to-tsdb/_index.md b/docs/sources/setup/migrate/migrate-to-tsdb/_index.md new file mode 100644 index 0000000000000..963913e21ef9e --- /dev/null +++ b/docs/sources/setup/migrate/migrate-to-tsdb/_index.md @@ -0,0 +1,74 @@ +--- +title: Migrate to TSDB +menuTitle: Migrate to TSDB +description: Migration guide for moving from any of the older indexes to TSDB +weight: 100 +keywords: + - migrate + - tsdb +--- + +# Migrate to TSDB + +[TSDB]({{< relref "../../../operations/storage/tsdb" >}}) is the recommended index type for Loki and is where the current development lies. +If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types](https://grafana.com/docs/loki//configure/storage/#index-storage) that have been deprecated, +we strongly recommend migrating to TSDB. + + +### Configure TSDB index for an upcoming period + +To begin the migration, add a new [period_config](https://grafana.com/docs/loki//configure/#period_config) entry in your [schema_config](https://grafana.com/docs/loki//configure/#schema_config). +You can read more about schema config [here](https://grafana.com/docs/loki//configure/storage/#schema-config). + +{{% admonition type="note" %}} +You must roll out the new `period_config` change to all Loki components in order for it to take effect. +{{% /admonition %}} + +This example adds a new `period_config` which configures Loki to start using the TSDB index for the data ingested starting from `2023-10-20`. + +``` +schema_config: + configs: + - from: 2023-01-01 + store: boltdb-shipper + object_store: filesystem + schema: v11 + index: + prefix: index_ + period: 24h + - from: 2023-10-20 ① + store: tsdb ② + object_store: filesystem ③ + schema: v13 ④ + index: + prefix: index_ + period: 24h +``` + +1. You must set the new period `from` to a date in the future. + +1. Update the new period to use TSDB as the index type by setting `store: tsdb`. + +1. This sample configuration uses filesystem as the storage in both the periods. If you want to use a different storage for the TSDB index and chunks, you can specify a different `object_store` in the new period. + +1. Update the schema to v13 which is the recommended version at the time of writing. Please refer to the [configure page](https://grafana.com/docs/loki//configure/#period_config) for the current recommend version. + +### Configure TSDB shipper + +It's also important that you configure the `tsdb_shipper` block in [storage_config](https://grafana.com/docs/loki//configure/#storage_config). Specifically the following options: +- `active_index_directory`: directory where ingesters would write index files which will then be uploaded by shipper to configured storage. +- `cache_location`: cache location for downloading index files from the storage for use in query path. + +``` +storage_config: + tsdb_shipper: + active_index_directory: /data/tsdb-index + cache_location: /data/tsdb-cache +``` + +### Run compactor + +We strongly recommended running the [compactor]({{< relref "../../../operations/storage/retention#compactor" >}}) when using TSDB index. It is responsible for running compaction and retention on TSDB index. +Not running index compaction will result in sub-optimal query performance. + +Please refer to the [compactor section]({{< relref "../../../operations/storage/retention#compactor" >}}) for more information and configuration examples.