Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TSDB migration error #14054

Open
surekha3 opened this issue Sep 5, 2024 · 1 comment
Open

TSDB migration error #14054

surekha3 opened this issue Sep 5, 2024 · 1 comment

Comments

@surekha3
Copy link

surekha3 commented Sep 5, 2024

Describe the bug
We have loki running in simple scalable mode in AWS EKS cluster with 2.8.2 version. Storage format taken is boltDB shipper form. I want to migrate it to TSDB form with object storage as s3. Though it's working well with existing s3 bucket config, when configured TSDB, it's not picking up the same bucket even tried to modified the configuration in many ways

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki 2.8.2 version with simple scalable mode in EKS cluster
  2. Below is the config used before migration
    loki:
    auth_enabled: false
    externalConfigSecretName: "loki"
    readinessProbe:
    null

-- Limits config

limits_config:
retention_period: 4392h # Adjust as needed - Default retention to 1 month
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m

-- Storage config. Providing this will automatically populate all necessary storage configs in the templated config.
storage:
bucketNames:
chunks:
ruler:
type: s3
s3:
region: us-east-1
http_config:
idle_conn_timeout: 300s
response_header_timeout: 300s

-- Additional storage config

storage_config:
aws:
sse_encryption: true

  1. Below is the config used for migrationloki:
    config: |
    schema_config:
    configs:
    - from: 2022-01-11
    store: boltdb-shipper
    object_store: s3
    schema: v12
    index:
    prefix: loki_index_
    period: 24h
    - from: 2024-09-07
    store: tsdb
    object_store: s3
    schema: v12
    index:
    prefix: loki_index_
    period: 24h

-- Limits config

limits_config:
retention_period: 4392h # Adjust as needed - Default retention to 1 month
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m

-- Additional storage config

storage_config:
aws:
s3: s3://s3.us-east-1.amazonaws.com
bucketnames:
chunks: ""
ruler: ""
region: us-east-1
s3forcepathstyle: true
sse_encryption: true
tsdb_shipper:
active_index_directory: /var/loki/tsdb-index
cache_location: /var/loki/tsdb-cache
cache_ttl: 168h
shared_store: s3

Expected behavior
Data storing in boltDB shipper form should be migrated to TSDB format

Environment:

  • Infrastructure: AWS EKS cluster
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output

level=warn ts=2024-09-05T05:28:21.066645268Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout ("5m0s"). This behavior will change in the next major to always use the default per-tenant timeout ("5m")."
level=info ts=2024-09-05T05:28:21.068204901Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)"
level=info ts=2024-09-05T05:28:21.068699895Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2024-09-05T05:28:21.069391196Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - embedded-cache"
level=warn ts=2024-09-05T05:28:21.069528094Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
level=warn ts=2024-09-05T05:28:21.069543989Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
level=error ts=2024-09-05T05:28:21.069758759Z caller=log.go:171 msg="error running loki" err="at least one bucket name must be specified\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.buckets\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:332\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.NewS3ObjectClient\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:152\ngithub.com/grafana/loki/pkg/storage.NewObjectClient\n\t/src/loki/pkg/storage/factory.go:483\ngithub.com/grafana/loki/pkg/storage.NewChunkClient\n\t/src/loki/pkg/storage/factory.go:325\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:185\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/src/loki/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:147\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:655\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror creating object client\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:187\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/src/loki/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:147\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:655\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

@surekha3
Copy link
Author

surekha3 commented Sep 6, 2024

Can anyone help me with this please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants