Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Added Simple Scalable Deployment guide for AWS #14327

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d8dd1c7
added SSD guide for AWS
Jayclifford345 Oct 1, 2024
fccc7dc
Merge branch 'main' into aws-helm
Jayclifford345 Oct 1, 2024
3305c96
fixed title
Jayclifford345 Oct 1, 2024
ff2b1db
fixed trailing slashes
Jayclifford345 Oct 1, 2024
933ee34
updated microservice ref
Jayclifford345 Oct 1, 2024
edf4605
updated index
Jayclifford345 Oct 1, 2024
7886f0d
Update docs/sources/setup/install/helm/install-microservices/_index.md
Jayclifford345 Oct 1, 2024
c7cd341
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
bcbc2af
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
a94257c
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
119676b
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
af08bcc
Merge branch 'main' into aws-helm
Jayclifford345 Oct 1, 2024
108a3f0
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 2, 2024
f77d2d7
added recommended changes
Jayclifford345 Oct 3, 2024
6b72e1e
Merge branch 'aws-helm' of https://github.com/grafana/loki into aws-helm
Jayclifford345 Oct 3, 2024
7103f1c
fixed formatting
Jayclifford345 Oct 3, 2024
8225c4c
fixed namespace
Jayclifford345 Oct 3, 2024
271ef6d
added suggestion around seperating chunk and ruler bucket
Jayclifford345 Oct 7, 2024
d9767d2
Added suggestions by trevor
Jayclifford345 Oct 8, 2024
ae277ce
Merge branch 'main' into aws-helm
Jayclifford345 Oct 8, 2024
f553894
Update docs/sources/setup/install/helm/install-microservices/_index.md
Jayclifford345 Oct 8, 2024
dd57de1
added Poyzans recommendations
Jayclifford345 Oct 8, 2024
972878f
added Davids suggestions
Jayclifford345 Oct 10, 2024
07dfee8
Merge branch 'main' into aws-helm
Jayclifford345 Oct 15, 2024
b5144ec
fixed typo
Jayclifford345 Oct 15, 2024
6c244c7
Merge branch 'aws-helm' of https://github.com/grafana/loki into aws-helm
Jayclifford345 Oct 15, 2024
40903b4
Merge branch 'main' into aws-helm
Jayclifford345 Oct 15, 2024
109c349
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 17, 2024
1a9970b
added authentication
Jayclifford345 Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 67 additions & 120 deletions docs/sources/setup/install/helm/install-scalable/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ weight: 300
keywords:
---

# Install the simple scalable Helm chart
# Loki simple scalable Helm chart
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved

This Helm Chart deploys Grafana Loki on Kubernetes.
This Helm Chart deploys Grafana Loki on Kubernetes in simple scalable mode.

This chart configures Loki to run `read`, `write`, and `backend` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#simple-scalable" >}}). Loki’s simple scalable deployment mode separates execution paths into read, write, and backend targets.

Expand All @@ -22,17 +22,21 @@ The default Helm chart deploys the following components:
- Loki Canary (1 DaemonSet)
- Gateway (1 NGINX replica)
- Minio (optional, if `minio.enabled=true`)
- Index and Chunk cache (1 replica)

{{< admonition type="note" >}}
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example.
{{< /admonition >}}

**Prerequisites**
## Prerequisites

- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/).
- A running Kubernetes cluster.
- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to [caching section]({{< relref "../../../../operations/caching" >}}).


**To deploy Loki in simple scalable mode:**
## Deploying the Helm chart for development and testing

The following steps show how to deploy the Loki Helm chart in simple scalable mode using the included MinIO as the storage backend. Our recommendation is to start here for development and testing purposes. Then configure Loki with an object storage provider when moving to production.


1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm:
Expand All @@ -49,72 +53,51 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu

3. Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage:

```yaml
loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
tracing:
enabled: true
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4

#gateway:
# ingress:
# enabled: true
# hosts:
# - host: FIXME
# paths:
# - path: /
# pathType: Prefix

deploymentMode: SimpleScalable

backend:
replicas: 3
read:
replicas: 3
write:
replicas: 3

# Enable minio for storage
minio:
enabled: true

# Zero out replica counts of other deployment modes
singleBinary:
replicas: 0

ingester:
replicas: 0
querier:
replicas: 0
queryFrontend:
replicas: 0
queryScheduler:
replicas: 0
distributor:
replicas: 0
compactor:
replicas: 0
indexGateway:
replicas: 0
bloomCompactor:
replicas: 0
bloomGateway:
replicas: 0
```

4. Install or upgrade the Loki deployment.
```yaml
loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4
pattern_ingester:
enabled: true
limits_config:
allow_structured_metadata: true
volume_enabled: true
retention_period: 672h
compactor:
retention_enabled: true
delete_request_store: s3

deploymentMode: SimpleScalable

backend:
replicas: 2
read:
replicas: 2
write:
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
replicas: 3

# Enable minio for storage
minio:
enabled: true

gateway:
service:
type: LoadBalancer
```

1. Install or upgrade the Loki deployment.
- To install:
```bash
helm install --values values.yaml loki grafana/loki
Expand Down Expand Up @@ -155,11 +138,11 @@ loki:
storage:
type: s3
bucketNames:
chunks: "<INSERT BUCKET NAME>"
chunks: "<INSERT BUCKET NAME>" #
ruler: "<INSERT BUCKET NAME>"
admin: "<INSERT BUCKET NAME>"
s3:
# s3 URL can be used to specify the endpoint, access key, secret key, and bucket name
# s3 URL can be used to specify the endpoint, access key, secret key, and bucket name this works well for S3 compatible storages or are hosting Loki on-premises and want to use S3 as the storage backend. Either use the s3 URL or the individual fields below (AWS endpoint, region, secret).
s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name
# AWS endpoint URL
endpoint: <your-endpoint>
Expand Down Expand Up @@ -190,29 +173,6 @@ write:
# Disable minio storage
minio:
enabled: false

# Zero out replica counts of other deployment modes
singleBinary:
replicas: 0

ingester:
replicas: 0
querier:
replicas: 0
queryFrontend:
replicas: 0
queryScheduler:
replicas: 0
distributor:
replicas: 0
compactor:
replicas: 0
indexGateway:
replicas: 0
bloomCompactor:
replicas: 0
bloomGateway:
replicas: 0
```
```azure
loki:
Expand Down Expand Up @@ -269,34 +229,21 @@ write:
minio:
enabled: false

# Zero out replica counts of other deployment modes
singleBinary:
replicas: 0

ingester:
replicas: 0
querier:
replicas: 0
queryFrontend:
replicas: 0
queryScheduler:
replicas: 0
distributor:
replicas: 0
compactor:
replicas: 0
indexGateway:
replicas: 0
bloomCompactor:
replicas: 0
bloomGateway:
replicas: 0
```

{{< /code >}}

To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}).

## Deploying the Loki Helm chart to a Production Environment

{{< admonition type="note" >}}
We are activley working on providing more guides for deploying Loki in production.
{{< /admonition >}}

It is recommended to run Loki at scale within in a cloud enviroment like AWS, Azure, or GCP. The below guides will show you how to deploy a minimally viable production environment.
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
- [Deploy Loki on AWS]({{< relref "./aws" >}})



## Next Steps
* Configure an agent to [send log data to Loki](/docs/loki/<LOKI_VERSION>/send-data/).
* Monitor the Loki deployment using the [Meta Monitoring Helm chart](/docs/loki/<LOKI_VERSION>/setup/install/helm/monitor-and-alert/)
Loading
Loading