allow to configure s3 subpath (bucket prefix configuration) #5889

oneacl · 2022-04-12T13:48:28Z

Is your feature request related to a problem? Please describe.

Currently a full S3 bucket is required for Loki - however, in some situations a bucket may need to be shared with other apps (one example would be aws lightsail where there's a fixed cost per bucket).

Describe the solution you'd like

Ability to specify a folder where all Loki data will be stored in a bucket

I've raised a similar issue with Mimir: grafana/mimir#1682

Thank you.

arcosx · 2022-04-22T07:07:01Z

I think this issue is very important and I am facing the same problem.

arcosx · 2022-04-22T07:13:00Z

Can this issue be assigned to me? I plan to implement a subpath feature for all object storage (s3,gcs,azure storage....).
I can give a proposal. @kavirajk @owen-d

fzyzcjy · 2022-04-23T00:31:48Z

Facing the same problem. This is very much needed since we are using that bucket for other purposes as well. Making it into a subpath will make it much neater.

Sylphe88 · 2022-06-23T09:50:13Z

Need the same feature here, so that I can fit all the observability backends in the same bucket (thanos, loki...)!

stale · 2022-08-13T19:47:41Z

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

Mark issues as revivable if we think it's a valid issue but isn't something we are likely
to prioritize in the future (the issue will still remain closed).
Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

callmeteus · 2022-08-14T00:32:55Z

Bumping it cause we are having the same problem.

zerda · 2022-09-09T04:08:53Z

Loki communicates with the S3 bucket using the BucketClient from github.com/thanos-io/thanos/pkg/objstore/s3.

Since thanos-io/thanos#5337 has been merged, implementing bucket prefix in loki should be much simpler.

oneacl · 2022-12-14T20:02:33Z

@dimitarvdimitrov - thank you for grafana/mimir#1686

was wondering if there are any plans to retrofit this into loki as well?

Thank you.

nicon89 · 2022-12-17T15:13:25Z

Is there any update on this?

davinkevin · 2022-12-31T12:04:43Z

+1000 on this one! Really important when you want to backup a complete monitoring bucket with all [metrics|traces|logs] in the same state.

rpartapsing · 2023-04-18T15:06:23Z

Is there any update on this one?

eddiewong007 · 2023-06-01T02:57:36Z

Any updates? Really need this feature to keep it align with mimir and tempo!

matschaffer-roblox · 2023-08-23T01:38:01Z

I currently set up one loki per k8s cluster (via the helm chart) and this would be handy to allow all my clusters in a given environment (dev, prod, etc) to have a common log storage bucket rather than ENV x CLUSTER buckets.

For now I'll just live with the dozen or so extra buckets produced by the above multiplication.

**What this PR does / why we need it**: Adds a new option under the aws stanza named key_prefix. This is useful if you have many Loki installations and do not want to create a million buckets. This is different than `compactor.shared_store_key_prefix` because it also affects eg the `loki_cluster_seed.json` file. **Which issue(s) this PR fixes**: Fixes #5889 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e) --------- Signed-off-by: Edward Welch <[email protected]> Co-authored-by: Ed Welch <[email protected]>

dragonball-wooboo · 2024-01-31T06:21:47Z

Is there any update on this one?

rubenvw-ngdata · 2024-02-16T12:19:42Z

Also interested in how this is done for the helm chart. Is it documented somewhere?

adil-jeel · 2024-03-02T08:18:24Z

Is there any update on this one?

awoimbee · 2024-03-20T11:25:47Z

Hi, I'm a bit lost, I want to use this feature that should be available since #10096 got merged (it's in the code) but with loki 2.9.4 I get:

failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
  line 71: field object_prefix not found in type storage.Config

I set loki.storage_config.object_prefix to loki/ in the helm chart, it translates in the loki configmap to storage_config.object_prefix.

A quick search of the issues show that people use this feature, but how can it work if it's not accepted by storage.Config ?
strings ~/Downloads/loki-linux-amd64 | grep -E '[Oo]bject_?[Pp]refix' | grep -v -E '[lL]og_?[Oo]bject' returns nothing ?

EDIT: goddammit there is a release-2.9.x branch, using the main branch works

tculp · 2024-03-20T18:28:24Z

On that note, this feature has now been merged to main for nearly 5 whole months but not included in 2.9.3, 2.9.4, or 2.9.5. Could this please get included in a release, and also updated in the helm chart, added to documentation, etc?

@slim-bean I see you're the one who merged the original feature, so I'm tagging you for visibility

…a#10096) **What this PR does / why we need it**: Adds a new option under the aws stanza named key_prefix. This is useful if you have many Loki installations and do not want to create a million buckets. This is different than `compactor.shared_store_key_prefix` because it also affects eg the `loki_cluster_seed.json` file. **Which issue(s) this PR fixes**: Fixes grafana#5889 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](grafana@d10549e) --------- Signed-off-by: Edward Welch <[email protected]> Co-authored-by: Ed Welch <[email protected]>

oneacl mentioned this issue Apr 12, 2022

Feature: allow to configure s3 subpath grafana/mimir#1682

Closed

kavirajk added component/storage cloudstorage/s3 and removed cloudstorage/s3 labels Apr 13, 2022

darintay mentioned this issue Aug 2, 2022

S3 bucket prefix should be configurable #1760

Closed

dannykopping mentioned this issue Sep 28, 2022

Dynamic prefix with S3 #5779

Closed

dannykopping added the keepalive An issue or PR that will be kept alive and never marked as stale. label Sep 28, 2022

This was referenced Sep 28, 2022

Support nested folders inside s3 bucket #5690

Closed

Allow for multiple store configurations from the same provider #7276

Closed

mapno mentioned this issue Feb 10, 2023

Allow to configure subpaths within S3 buckets grafana/tempo#2100

Closed

aschleck mentioned this issue Jul 28, 2023

Storage: Allow setting a constant prefix for all created keys #10096

Merged

7 tasks

slim-bean closed this as completed in #10096 Oct 31, 2023

tculp mentioned this issue Mar 26, 2024

Release "store objects under prefix object storage" feature #12364

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow to configure s3 subpath (bucket prefix configuration) #5889

allow to configure s3 subpath (bucket prefix configuration) #5889

oneacl commented Apr 12, 2022 •

edited by kavirajk

Loading

arcosx commented Apr 22, 2022

arcosx commented Apr 22, 2022 •

edited

Loading

fzyzcjy commented Apr 23, 2022 •

edited

Loading

Sylphe88 commented Jun 23, 2022

stale bot commented Aug 13, 2022

callmeteus commented Aug 14, 2022

zerda commented Sep 9, 2022

oneacl commented Dec 14, 2022

nicon89 commented Dec 17, 2022

davinkevin commented Dec 31, 2022

rpartapsing commented Apr 18, 2023

eddiewong007 commented Jun 1, 2023

matschaffer-roblox commented Aug 23, 2023 •

edited

Loading

dragonball-wooboo commented Jan 31, 2024

rubenvw-ngdata commented Feb 16, 2024

adil-jeel commented Mar 2, 2024

awoimbee commented Mar 20, 2024 •

edited

Loading

tculp commented Mar 20, 2024

allow to configure s3 subpath (bucket prefix configuration) #5889

allow to configure s3 subpath (bucket prefix configuration) #5889

Comments

oneacl commented Apr 12, 2022 • edited by kavirajk Loading

arcosx commented Apr 22, 2022

arcosx commented Apr 22, 2022 • edited Loading

fzyzcjy commented Apr 23, 2022 • edited Loading

Sylphe88 commented Jun 23, 2022

stale bot commented Aug 13, 2022

callmeteus commented Aug 14, 2022

zerda commented Sep 9, 2022

oneacl commented Dec 14, 2022

nicon89 commented Dec 17, 2022

davinkevin commented Dec 31, 2022

rpartapsing commented Apr 18, 2023

eddiewong007 commented Jun 1, 2023

matschaffer-roblox commented Aug 23, 2023 • edited Loading

dragonball-wooboo commented Jan 31, 2024

rubenvw-ngdata commented Feb 16, 2024

adil-jeel commented Mar 2, 2024

awoimbee commented Mar 20, 2024 • edited Loading

tculp commented Mar 20, 2024

oneacl commented Apr 12, 2022 •

edited by kavirajk

Loading

arcosx commented Apr 22, 2022 •

edited

Loading

fzyzcjy commented Apr 23, 2022 •

edited

Loading

matschaffer-roblox commented Aug 23, 2023 •

edited

Loading

awoimbee commented Mar 20, 2024 •

edited

Loading