Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Added Simple Scalable Deployment guide for AWS #14327

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d8dd1c7
added SSD guide for AWS
Jayclifford345 Oct 1, 2024
fccc7dc
Merge branch 'main' into aws-helm
Jayclifford345 Oct 1, 2024
3305c96
fixed title
Jayclifford345 Oct 1, 2024
ff2b1db
fixed trailing slashes
Jayclifford345 Oct 1, 2024
933ee34
updated microservice ref
Jayclifford345 Oct 1, 2024
edf4605
updated index
Jayclifford345 Oct 1, 2024
7886f0d
Update docs/sources/setup/install/helm/install-microservices/_index.md
Jayclifford345 Oct 1, 2024
c7cd341
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
bcbc2af
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
a94257c
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
119676b
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
af08bcc
Merge branch 'main' into aws-helm
Jayclifford345 Oct 1, 2024
108a3f0
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 2, 2024
f77d2d7
added recommended changes
Jayclifford345 Oct 3, 2024
6b72e1e
Merge branch 'aws-helm' of https://github.com/grafana/loki into aws-helm
Jayclifford345 Oct 3, 2024
7103f1c
fixed formatting
Jayclifford345 Oct 3, 2024
8225c4c
fixed namespace
Jayclifford345 Oct 3, 2024
271ef6d
added suggestion around seperating chunk and ruler bucket
Jayclifford345 Oct 7, 2024
d9767d2
Added suggestions by trevor
Jayclifford345 Oct 8, 2024
ae277ce
Merge branch 'main' into aws-helm
Jayclifford345 Oct 8, 2024
f553894
Update docs/sources/setup/install/helm/install-microservices/_index.md
Jayclifford345 Oct 8, 2024
dd57de1
added Poyzans recommendations
Jayclifford345 Oct 8, 2024
972878f
added Davids suggestions
Jayclifford345 Oct 10, 2024
07dfee8
Merge branch 'main' into aws-helm
Jayclifford345 Oct 15, 2024
b5144ec
fixed typo
Jayclifford345 Oct 15, 2024
6c244c7
Merge branch 'aws-helm' of https://github.com/grafana/loki into aws-helm
Jayclifford345 Oct 15, 2024
40903b4
Merge branch 'main' into aws-helm
Jayclifford345 Oct 15, 2024
109c349
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 17, 2024
1a9970b
added authentication
Jayclifford345 Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/sources/setup/install/helm/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,17 @@ This guide references the Loki Helm chart version 3.0 or greater and contains th

If you are installing Grafana Enterprise Logs, follow the [GEL Helm installation](https://grafana.com/docs/enterprise-logs/<ENTERPRISE_LOGS_VERSION>/setup/helm/).

## Recommended Installation

The recommended installation method for initial deployments is to use the [Loki Simple Scalable Helm chart]({{< relref "./install-scalable" >}}). This chart provides a simple scalable deployment mode for Loki, separating execution paths into read, write, and backend targets. For small to medium-sized deployments, this chart is a good starting point.
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved

### Cloud Deployment Guides

The following guides provide step-by-step instructions for deploying Loki on cloud providers:

- [Deploy Loki Simple Scalable Helm chart on AWS]({{< relref "./install-scalable/aws" >}})


## Reference

[Values reference]({{< relref "./reference" >}})
172 changes: 93 additions & 79 deletions docs/sources/setup/install/helm/install-microservices/_index.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
title: Install the microservice Helm chart
title: Loki Microservice Helm chart
menuTitle: Install microservice Loki
description: Installing Loki in microservice (distributed) mode using the Helm chart.
weight: 300
keywords:
---

# Install the microservice Helm chart
# Loki Microservice Helm chart

This Helm Chart deploys Grafana Loki on Kubernetes.

Expand All @@ -21,16 +21,17 @@ The default Helm chart deploys the following components:
- **QueryFrontend component** (2 replicas, maxUnavailable: 1): Manages frontend queries. Up to 1 replica can be unavailable during updates.
- **QueryScheduler component** (2 replicas): Schedules queries.

It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example.
{{< admonition type="note" >}}
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
We do not recommended to run Microservice mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example.
{{< /admonition >}}

**Prerequisites**
## Prerequisites

- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/).
- A running Kubernetes cluster.
- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to the [caching section](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/caching/).


**To deploy Loki in microservice mode (with MinIO):**
## Deploying the Helm chart for development and testing


1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm:
Expand All @@ -49,46 +50,45 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu

```yaml
loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
tracing:
enabled: true
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4

#gateway:
# ingress:
# enabled: true
# hosts:
# - host: FIXME
# paths:
# - path: /
# pathType: Prefix
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4
pattern_ingester:
enabled: true
limits_config:
allow_structured_metadata: true
volume_enabled: true
retention_period: 672h
compactor:
retention_enabled: true
delete_request_store: s3


deploymentMode: Distributed

ingester:
replicas: 3
replicas: 3 # To ensure data durability with replication
querier:
replicas: 3
replicas: 3 # Improve query performance via parallelism
maxUnavailable: 2
queryFrontend:
replicas: 2
maxUnavailable: 1
queryScheduler:
replicas: 2
distributor:
replicas: 3
replicas: 3
maxUnavailable: 2
compactor:
replicas: 1
Expand All @@ -100,21 +100,26 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu
replicas: 0
bloomGateway:
replicas: 0

# Enable minio for storage
minio:
enabled: true

Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
# Zero out replica counts of other deployment modes

backend:
replicas: 0
replicas: 0
read:
replicas: 0
replicas: 0
write:
replicas: 0
replicas: 0

singleBinary:
replicas: 0
replicas: 0

# This exposes the Loki gateway so it can be written to and queried externaly
gateway:
service:
type: LoadBalancer


# Enable minio for storage
minio:
enabled: true
```

4. Install or upgrade the Loki deployment.
Expand All @@ -138,7 +143,7 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu
loki-canary-8thrx 1/1 Running 0 167m
loki-canary-h965l 1/1 Running 0 167m
loki-canary-th8kb 1/1 Running 0 167m
loki-chunks-cache-0 0/2 Pending 0 167m
loki-chunks-cache-0 2/2 Running 0 167m
loki-compactor-0 1/1 Running 0 167m
loki-compactor-1 1/1 Running 0 167m
loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m
Expand All @@ -165,7 +170,7 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu

## Object Storage Configuration

After testing Loki with MinIO, it is recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers:
After testing Loki with MinIO, we recommend to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers:
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved

{{< admonition type="caution" >}}
When deploying Loki using S3 Storage **DO NOT** use the default bucket names; `chunk`, `ruler` and `admin`. Choose a unique name for each bucket. For more information see the following [security update](https://grafana.com/blog/2024/06/27/grafana-security-update-grafana-loki-and-unintended-data-write-attempts-to-amazon-s3-buckets/). This caution does not apply when you are using MinIO. When using MinIO we recommend using the default bucket names.
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -176,48 +181,57 @@ When deploying Loki using S3 Storage **DO NOT** use the default bucket names; `
```s3
# Example configuration for Loki with S3 storage

loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
region: <AWS region your bucket is in eg. `eu-west-2`>
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
bucketnames: loki-aws-bucket # Define your AWS bucket here
s3forcepathstyle: false
ingester:
chunk_encoding: snappy
tracing:
pattern_ingester:
enabled: true
querier:
limits_config:
allow_structured_metadata: true
volume_enabled: true
retention_period: 672h # 28 days retention
querier:
max_concurrent: 4

storage:
storage:
type: s3
bucketNames:
chunks: "<INSERT BUCKET NAME>"
ruler: "<INSERT BUCKET NAME>"
admin: "<INSERT BUCKET NAME>"
s3:
# s3 URL can be used to specify the endpoint, access key, secret key, and bucket name
s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name
# AWS endpoint URL
endpoint: <your-endpoint>
# AWS region where the S3 bucket is located
region: <your-region>
# AWS secret access key
secretAccessKey: <your-secret-access-key>
# AWS access key ID
accessKeyId: <your-access-key-id>
# AWS signature version (e.g., v2 or v4)
signatureVersion: <your-signature-version>
# Forces the path style for S3 (true/false)
s3ForcePathStyle: false
# Allows insecure (HTTP) connections (true/false)
insecure: false
# HTTP configuration settings
http_config: {}
s3:
# s3 URL can be used to specify the endpoint, access key, secret key, and bucket name this works well for S3 compatible storages or are hosting Loki on-premises and want to use S3 as the storage backend. Either use the s3 URL or the individual fields below (AWS endpoint, region, secret).
s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name
# AWS endpoint URL
endpoint: <your-endpoint>
# AWS region where the S3 bucket is located
region: <your-region>
# AWS secret access key
secretAccessKey: <your-secret-access-key>
# AWS access key ID
accessKeyId: <your-access-key-id>
# AWS signature version (e.g., v2 or v4)
signatureVersion: <your-signature-version>
# Forces the path style for S3 (true/false)
s3ForcePathStyle: false
# Allows insecure (HTTP) connections (true/false)
insecure: false
# HTTP configuration settings
http_config: {}

deploymentMode: Distributed

Expand Down
Loading
Loading