Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] invalid memory address or nil pointer dereference #11410

Closed
AtlanCI opened this issue Dec 8, 2023 · 7 comments
Closed

[Bug] invalid memory address or nil pointer dereference #11410

AtlanCI opened this issue Dec 8, 2023 · 7 comments
Assignees
Labels
type/bug Somehing is not working as expected

Comments

@AtlanCI
Copy link

AtlanCI commented Dec 8, 2023

Describe the bug
I'm starting the loki all-in-one program and I'm getting panic.

To Reproduce
Steps to reproduce the behavior:

  1. use this config
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9060

ruler:
  storage:
    type: local
    local:
      directory: /home/server/loki/ruler
  rule_path: /tmp/scratch
  enable_api: true

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
  - from: 2023-04-27
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb:
    directory: /home/server/loki/index   

  filesystem:
    directory: /home/server/loki/chunks  

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  query_timeout: 5m 

  max_query_lookback: 672h



table_manager:
  retention_deletes_enabled: true
  retention_period: 672h
  1. Started Loki (2.9.2)

Expected behavior
normal startup

Environment:

  • Infrastructure: bare-metal
  • OS: CentOS Linux release 7.6.1810 (Core)

Screenshots, Promtail config, or terminal output
stack log

panic: runtime error: invalid memory address or nil pointer dereference
goroutine 475 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2548240, 0xc000b3bbd8})
        /drone/src/pkg/storage/stores/shipper/util/util.go:95 +0x47
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
        /drone/src/pkg/storage/stores/shipper/util/util.go:54 +0x45
panic({0x2548240?, 0xc000b3bbd8?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
        /drone/src/vendor/go.etcd.io/bbolt/freelist.go:265 +0x1a
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
        /drone/src/vendor/go.etcd.io/bbolt/db.go:323 +0xaf
sync.(*Once).doSlow(0xc000b3ed08?, 0x10?)
        /usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000b3eb40?)
        /drone/src/vendor/go.etcd.io/bbolt/db.go:316 +0x45
go.etcd.io/bbolt.Open({0xc000af2dc0, 0x36}, 0x1b6?, 0xc0010fbf48)
        /drone/src/vendor/go.etcd.io/bbolt/db.go:293 +0x458
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
        /drone/src/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000af2dc0, 0x36}, 0xc000b525a0)
        /drone/src/pkg/storage/stores/shipper/util/util.go:62 +0x105
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile in goroutine 1
        /drone/src/pkg/storage/stores/shipper/util/util.go:42 +0x86
@JStickler JStickler added the type/bug Somehing is not working as expected label Dec 11, 2023
@yashkundu
Copy link

Hi, @JStickler , I would like to pick it up, can you please assign this to me.

@JStickler
Copy link
Contributor

@yashkundu Sorry just saw that you'd tagged me, I've been out sick. No need to wait to have something assigned to you, you can work on the fix and submit a PR at any time.

@matheushent
Copy link

Out of the sudden, I started being hit by this error. Full trace is below:

panic: runtime error: invalid memory address or nil pointer dereference
goroutine 290 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc0000107f8})
	/src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
	/src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc0000107f8})
	/usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
	/src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
	/src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc0004021c8?, 0x10?)
	/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000402000?)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc000649260, 0x2b}, 0x1b6?, 0xc000517f48)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
	/src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000649260, 0x2b}, 0xc0005468a0)
	/src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
	/src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc000649260, 0x2b})
	/src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708c20, 0x20}, 0xc00073e9e0)
	/src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708c20, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
	/src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
	/src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
	/src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
	/src/loki/pkg/loki/loki.go:422 +0x56
main.main()
	/src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creapanic: runtime error: invalid memory address or nil pointer dereference
goroutine 303 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc000010e28})
	/src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
	/src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc000010e28})
	/usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
	/src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
	/src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc000844408?, 0x10?)
	/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000844240?)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc0008367b0, 0x2b}, 0x1b6?, 0xc000517f48)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
	/src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc0008367b0, 0x2b}, 0xc000547aa0)
	/src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
	/src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc0008367b0, 0x2b})
	/src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
	/src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
	/src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
	/src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
	/src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
	/src/loki/pkg/loki/loki.go:422 +0x56
main.main()
	/src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creapanic: runtime error: invalid memory address or nil pointer dereference
goroutine 308 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc000010f60})
	/src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
	/src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc000010f60})
	/usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
	/src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
	/src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc000844f48?, 0x10?)
	/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000844d80?)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc000836a50, 0x2b}, 0x1b6?, 0xc000517f48)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
	/src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000836a50, 0x2b}, 0xc000547e60)
	/src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
	/src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc000836a50, 0x2b})
	/src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
	/src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
	/src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
	/src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
	/src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
	/src/loki/pkg/loki/loki.go:422 +0x56
main.main()
	/src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creapanic: runtime error: invalid memory address or nil pointer dereference
goroutine 311 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc000011008})
	/src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
	/src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc000011008})
	/usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
	/src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
	/src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc000845608?, 0x10?)
	/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000845440?)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc000836c00, 0x2b}, 0x1b6?, 0xc000517f48)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
	/src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000836c00, 0x2b}, 0xc00005f260)
	/src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
	/src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc000836c00, 0x2b})
	/src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
	/src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
	/src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
	/src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
	/src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
	/src/loki/pkg/loki/loki.go:422 +0x56
main.main()
	/src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creaunexpected fault address 0x7fa3a956c008
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fa3a956c008 pc=0x197beb1]

goroutine 1 [running]:
runtime.throw({0x234deaf?, 0x18?})
	/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc0009a6bb0 sp=0xc0009a6b80 pc=0x438f3d
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:834 +0x125 fp=0xc0009a6c10 sp=0xc0009a6bb0 pc=0x44fcc5
go.etcd.io/bbolt.(*elemRef).isLeaf(...)
	/src/loki/vendor/go.etcd.io/bbolt/cursor.go:387
go.etcd.io/bbolt.(*Cursor).first(0xc0009a6d20)
	/src/loki/vendor/go.etcd.io/bbolt/cursor.go:170 +0x51 fp=0xc0009a6c70 sp=0xc0009a6c10 pc=0x197beb1
go.etcd.io/bbolt.(*Cursor).First(0xc0009a6d20)
	/src/loki/vendor/go.etcd.io/bbolt/cursor.go:36 +0x118 fp=0xc0009a6cf0 sp=0xc0009a6c70 pc=0x197b718
go.etcd.io/bbolt.(*Bucket).ForEach(0xc000843898, 0xc0009a6d60)
	/src/loki/vendor/go.etcd.io/bbolt/bucket.go:389 +0x70 fp=0xc0009a6d50 sp=0xc0009a6cf0 pc=0x19791d0
go.etcd.io/bbolt.(*Tx).ForEach(0xc000845aa0?, 0xc0009a6da8?)
	/src/loki/vendor/go.etcd.io/bbolt/tx.go:128 +0x49 fp=0xc0009a6d88 sp=0xc0009a6d50 pc=0x19888c9
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir.func1(0xc0009a6de0?)
	/src/loki/pkg/storage/stores/shipper/index/table.go:382 +0x39 fp=0xc0009a6db8 sp=0xc0009a6d88 pc=0x1b96339
go.etcd.io/bbolt.(*DB).View(0xc000836d20?, 0xc0009a6f38)
	/src/loki/vendor/go.etcd.io/bbolt/db.go:772 +0x82 fp=0xc0009a6e20 sp=0xc0009a6db8 pc=0x197f7e2
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
	/src/loki/pkg/storage/stores/shipper/index/table.go:381 +0x63b fp=0xc0009a6fa8 sp=0xc0009a6e20 pc=0x1b95edb
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
	/src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46 fp=0xc0009a7000 sp=0xc0009a6fa8 pc=0x1b928a6
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674 fp=0xc0009a71f8 sp=0xc0009a7000 pc=0x1b97f34
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
	/src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b fp=0xc0009a7320 sp=0xc0009a71f8 pc=0x1b9662b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef fp=0xc0009a7718 sp=0xc0009a7320 pc=0x1b98eef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
	/src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7 fp=0xc0009a79d0 sp=0xc0009a7718 pc=0x1b989b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/factory.go:190 +0xa13 fp=0xc0009a8a10 sp=0xc0009a79d0 pc=0x1bd58f3
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:268 +0xd46 fp=0xc0009ab558 sp=0xc0009a8a10 pc=0x1bdbe06
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
	/src/loki/pkg/storage/store.go:164 +0x213 fp=0xc0009ab778 sp=0xc0009ab558 pc=0x1bda973
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
	/src/loki/pkg/storage/store.go:147 +0xa3b fp=0xc0009abbe0 sp=0xc0009ab778 pc=0x1bda5fb
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
	/src/loki/pkg/loki/modules.go:632 +0x594 fp=0xc0009adb90 sp=0xc0009abbe0 pc=0x1d58214
github.com/grafana/loki/pkg/loki.(*Loki).initStore-fm()
	<autogenerated>:1 +0x26 fp=0xc0009adba8 sp=0xc0009adb90 pc=0x1d62dc6
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a fp=0xc0009adcb0 sp=0xc0009adba8 pc=0x1a5b5ea
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8 fp=0xc0009addd0 sp=0xc0009adcb0 pc=0x1a5b358
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
	/src/loki/pkg/loki/loki.go:422 +0x56 fp=0xc0009adf48 sp=0xc0009addd0 pc=0x1d4da96
main.main()
	/src/loki/cmd/loki/main.go:105 +0xe55 fp=0xc0009b1f80 sp=0xc0009adf48 pc=0x1d64455
runtime.main()
	/usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc0009b1fe0 sp=0xc0009b1f80 pc=0x43b827
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0009b1fe8 sp=0xc0009b1fe0 pc=0x46fc41

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007afb0 sp=0xc00007af90 pc=0x43bc56
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc00007afe0 sp=0xc00007afb0 pc=0x43ba90
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007afe8 sp=0xc00007afe0 pc=0x46fc41
created by runtime.init.6
	/usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007b780 sp=0xc00007b760 pc=0x43bc56
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc00007b7c8 sp=0xc00007b780 pc=0x425dde
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00007b7e0 sp=0xc00007b7c8 pc=0x41b046
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007b7e8 sp=0xc00007b7e0 pc=0x46fc41
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc000098000?, 0x27d62e8?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007bf70 sp=0xc00007bf50 pc=0x43bc56
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x3a079c0)
	/usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc00007bfa0 sp=0xc00007bf70 pc=0x423cb3
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc00007bfc8 sp=0xc00007bfa0 pc=0x4242a5
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x41afe6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x46fc41
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x43bfd2?, 0x43b9c5?, 0x0?, 0x0?, 0xc00007a770?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007a628 sp=0xc00007a608 pc=0x43bc56
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00007a7e0 sp=0xc00007a628 pc=0x41a087
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007a7e8 sp=0xc00007a7e0 pc=0x46fc41
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x4c7772aa202?, 0xc00009cba0?, 0xa0?, 0xc7?, 0xc00007c7a8?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007c750 sp=0xc00007c730 pc=0x43bc56
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00007c7e0 sp=0xc00007c750 pc=0x41cdb1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x46fc41
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x4c77768cddb?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000076750 sp=0xc000076730 pc=0x43bc56
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000767e0 sp=0xc000076750 pc=0x41cdb1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x46fc41
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 52 [select]:
runtime.gopark(0xc000077f48?, 0x2?, 0x0?, 0x0?, 0xc000077f14?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000077d90 sp=0xc000077d70 pc=0x43bc56
runtime.selectgo(0xc000077f48, 0xc000077f10, 0x0?, 0x0, 0x3?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000077ed0 sp=0xc000077d90 pc=0x44bf1e
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9 fp=0xc000077fe0 sp=0xc000077ed0 pc=0x16855f9
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x46fc41
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
	/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
runtime.gopark(0xc000078788?, 0x3?, 0x90?, 0x0?, 0xc000078772?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000785f8 sp=0xc0000785d8 pc=0x43bc56
runtime.selectgo(0xc000078788, 0xc00007876c, 0xc0004cc200?, 0x0, 0xc000078788?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000078738 sp=0xc0000785f8 pc=0x44bf1e
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad fp=0xc0000787c8 sp=0xc000078738 pc=0x16fcfad
go.opencensus.io/stats/view.init.0.func1()
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc0000787e0 sp=0xc0000787c8 pc=0x16fc226
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x46fc41
created by go.opencensus.io/stats/view.init.0
	/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
runtime.gopark(0xc0002ccf58?, 0x2?, 0xf0?, 0x5f?, 0xc0002ccf4c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002ccdd8 sp=0xc0002ccdb8 pc=0x43bc56
runtime.selectgo(0xc0002ccf58, 0xc0002ccf48, 0xc0002ccf40?, 0x0, 0xc000969408?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002ccf18 sp=0xc0002ccdd8 pc=0x44bf1e
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89 fp=0xc0002ccf88 sp=0xc0002ccf18 pc=0x8d1989
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d fp=0xc0002ccfc8 sp=0xc0002ccf88 pc=0x8d184d
github.com/uber/jaeger-client-go.NewRemotelyControlledSampler.func1()
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x26 fp=0xc0002ccfe0 sp=0xc0002ccfc8 pc=0x8d0f46
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002ccfe8 sp=0xc0002ccfe0 pc=0x46fc41
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
	/src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
runtime.gopark(0xc0009a1ed0?, 0x441d75?, 0x18?, 0x1f?, 0x458e6f?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0009a1ec0 sp=0xc0009a1ea0 pc=0x43bc56
runtime.chanrecv(0xc0008b09c0, 0xc0009a1fb0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0009a1f50 sp=0xc0009a1ec0 pc=0x406f9d
runtime.chanrecv2(0x5f5e100?, 0x853125?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0009a1f78 sp=0xc0009a1f50 pc=0x406ad8
github.com/go-kit/log.WithFlushPeriod.func1.1()
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5 fp=0xc0009a1fe0 sp=0xc0009a1f78 pc=0x564c25
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0009a1fe8 sp=0xc0009a1fe0 pc=0x46fc41
created by github.com/go-kit/log.WithFlushPeriod.func1
	/src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
runtime.gopark(0xc0002cdf88?, 0x2?, 0x4?, 0x0?, 0xc0002cdf6c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002cddf0 sp=0xc0002cddd0 pc=0x43bc56
runtime.selectgo(0xc0002cdf88, 0xc0002cdf68, 0xc000536480?, 0x0, 0x405d3d?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002cdf30 sp=0xc0002cddf0 pc=0x44bf1e
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc fp=0xc0002cdfc0 sp=0xc0002cdf30 pc=0x8bbefc
github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn.func1()
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x2a fp=0xc0002cdfe0 sp=0xc0002cdfc0 pc=0x8bbe0a
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cdfe8 sp=0xc0002cdfe0 pc=0x46fc41
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
	/src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
runtime.gopark(0xc000078f78?, 0x2?, 0x60?, 0x78?, 0xc000078f14?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000078d80 sp=0xc000078d60 pc=0x43bc56
runtime.selectgo(0xc000078f78, 0xc000078f10, 0xc000078f28?, 0x0, 0x28042c8?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000078ec0 sp=0xc000078d80 pc=0x44bf1e
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde fp=0xc000078fc8 sp=0xc000078ec0 pc=0x8ce3fe
github.com/uber/jaeger-client-go.NewRemoteReporter.func1()
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x26 fp=0xc000078fe0 sp=0xc000078fc8 pc=0x8ce086
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x46fc41
created by github.com/uber/jaeger-client-go.NewRemoteReporter
	/src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
runtime.gopark(0xc0009ff200?, 0xc00009aae0?, 0xe0?, 0xaa?, 0xc0002cefa8?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002ceee8 sp=0xc0002ceec8 pc=0x43bc56
runtime.chanrecv(0xc0009e1ec0, 0xc0002cefc8, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0002cef78 sp=0xc0002ceee8 pc=0x406f9d
runtime.chanrecv2(0x2807380?, 0xc000667890?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0002cefa0 sp=0xc0002cef78 pc=0x406ad8
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66 fp=0xc0002cefe0 sp=0xc0002cefa0 pc=0x9edca6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cefe8 sp=0xc0002cefe0 pc=0x46fc41
created by github.com/grafana/dskit/services.(*BasicService).AddListener
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
runtime.gopark(0xc0014f3e88?, 0x2?, 0x8?, 0xdc?, 0xc0014f3e3c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014f3ca0 sp=0xc0014f3c80 pc=0x43bc56
runtime.selectgo(0xc0014f3e88, 0xc0014f3e38, 0xc0000a7a90?, 0x0, 0xc0009e1f18?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014f3de0 sp=0xc0014f3ca0 pc=0x44bf1e
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e fp=0xc0014f3ee0 sp=0xc0014f3de0 pc=0xe74c6e
github.com/grafana/dskit/kv/memberlist.(*KV).running-fm({0x28042c8?, 0xc0000a7a90?})
	<autogenerated>:1 +0x39 fp=0xc0014f3f08 sp=0xc0014f3ee0 pc=0xe874f9
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8 fp=0xc0014f3fc8 sp=0xc0014f3f08 pc=0x9eca48
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1.2()
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x26 fp=0xc0014f3fe0 sp=0xc0014f3fc8 pc=0x9ec666
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0014f3fe8 sp=0xc0014f3fe0 pc=0x46fc41
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
	/src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
runtime.gopark(0xc0002c9f80?, 0x2?, 0x40?, 0xd0?, 0xc0002c9f64?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002c9de0 sp=0xc0002c9dc0 pc=0x43bc56
runtime.selectgo(0xc0002c9f80, 0xc0002c9f60, 0xc0002c9f58?, 0x0, 0xc0009f9680?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002c9f20 sp=0xc0002c9de0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9 fp=0xc0002c9fb8 sp=0xc0002c9f20 pc=0x133e7b9
github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache.func1()
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x2e fp=0xc0002c9fe0 sp=0xc0002c9fb8 pc=0x133e6ae
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002c9fe8 sp=0xc0002c9fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
runtime.gopark(0xc0002cbf80?, 0x2?, 0xa0?, 0xdb?, 0xc0002cbf64?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002cbde0 sp=0xc0002cbdc0 pc=0x43bc56
runtime.selectgo(0xc0002cbf80, 0xc0002cbf60, 0xc0002cbf58?, 0x0, 0xc0009f9e60?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002cbf20 sp=0xc0002cbde0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9 fp=0xc0002cbfb8 sp=0xc0002cbf20 pc=0x133e7b9
github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache.func1()
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x2e fp=0xc0002cbfe0 sp=0xc0002cbfb8 pc=0x133e6ae
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
	/src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
runtime.gopark(0xc0002cace8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0002cad50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002cacd0 sp=0xc0002cacb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0002caf18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0002cad60 sp=0xc0002cacd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0008f94b0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0002cad88 sp=0xc0002cad60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0002cafc8 sp=0xc0002cad88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0002cafe0 sp=0xc0002cafc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cafe8 sp=0xc0002cafe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 227 [chan receive]:
runtime.gopark(0xc0003a2ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a2d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a2cd0 sp=0xc0003a2cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a2f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a2d60 sp=0xc0003a2cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0004aaa00?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a2d88 sp=0xc0003a2d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a2fc8 sp=0xc0003a2d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a2fe0 sp=0xc0003a2fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a2fe8 sp=0xc0003a2fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 228 [chan receive]:
runtime.gopark(0xc0003a0ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a0d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a0cd0 sp=0xc0003a0cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a0f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a0d60 sp=0xc0003a0cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0008f94b0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a0d88 sp=0xc0003a0d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a0fc8 sp=0xc0003a0d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a0fe0 sp=0xc0003a0fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a0fe8 sp=0xc0003a0fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 229 [chan receive]:
runtime.gopark(0xc0003a1ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a1d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a1cd0 sp=0xc0003a1cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a1f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a1d60 sp=0xc0003a1cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0008f94b0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a1d88 sp=0xc0003a1d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a1fc8 sp=0xc0003a1d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a1fe0 sp=0xc0003a1fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a1fe8 sp=0xc0003a1fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 230 [chan receive]:
runtime.gopark(0xc00039fce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00039fd50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00039fcd0 sp=0xc00039fcb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00039ff18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00039fd60 sp=0xc00039fcd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00039fd88 sp=0xc00039fd60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00039ffc8 sp=0xc00039fd88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00039ffe0 sp=0xc00039ffc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00039ffe8 sp=0xc00039ffe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 231 [chan receive]:
runtime.gopark(0xc000569ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000569d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000569cd0 sp=0xc000569cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000569f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000569d60 sp=0xc000569cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000569d88 sp=0xc000569d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000569fc8 sp=0xc000569d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000569fe0 sp=0xc000569fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 232 [chan receive]:
runtime.gopark(0xc00056ace8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00056ad50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00056acd0 sp=0xc00056acb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00056af18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00056ad60 sp=0xc00056acd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00056ad88 sp=0xc00056ad60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00056afc8 sp=0xc00056ad88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00056afe0 sp=0xc00056afc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00056afe8 sp=0xc00056afe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 233 [chan receive]:
runtime.gopark(0xc00056bce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00056bd50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00056bcd0 sp=0xc00056bcb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00056bf18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00056bd60 sp=0xc00056bcd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00056bd88 sp=0xc00056bd60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00056bfc8 sp=0xc00056bd88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00056bfe0 sp=0xc00056bfc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00056bfe8 sp=0xc00056bfe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 234 [chan receive]:
runtime.gopark(0xc000568ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000568d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000568cd0 sp=0xc000568cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000568f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000568d60 sp=0xc000568cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000568d88 sp=0xc000568d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000568fc8 sp=0xc000568d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000568fe0 sp=0xc000568fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000568fe8 sp=0xc000568fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 235 [chan receive]:
runtime.gopark(0xc000565ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000565d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000565cd0 sp=0xc000565cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000565f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000565d60 sp=0xc000565cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000565d88 sp=0xc000565d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000565fc8 sp=0xc000565d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000565fe0 sp=0xc000565fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 236 [chan receive]:
runtime.gopark(0xc000564ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000564d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000564cd0 sp=0xc000564cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000564f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000564d60 sp=0xc000564cd0 pc=0x406f9d
runtime.chanrecv2(0x1?, 0xc0003c7500?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000564d88 sp=0xc000564d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000564fc8 sp=0xc000564d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000564fe0 sp=0xc000564fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000564fe8 sp=0xc000564fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 237 [chan receive]:
runtime.gopark(0xc000566ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000566d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000566cd0 sp=0xc000566cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000566f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000566d60 sp=0xc000566cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000566d88 sp=0xc000566d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000566fc8 sp=0xc000566d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000566fe0 sp=0xc000566fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000566fe8 sp=0xc000566fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 238 [chan receive]:
runtime.gopark(0xc000567ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000567d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000567cd0 sp=0xc000567cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000567f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000567d60 sp=0xc000567cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000567d88 sp=0xc000567d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000567fc8 sp=0xc000567d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000567fe0 sp=0xc000567fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 239 [chan receive]:
runtime.gopark(0xc000351ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000351d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000351cd0 sp=0xc000351cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000351f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000351d60 sp=0xc000351cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000351d88 sp=0xc000351d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000351fc8 sp=0xc000351d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000351fe0 sp=0xc000351fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000351fe8 sp=0xc000351fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 240 [chan receive]:
runtime.gopark(0xc0003a5ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a5d50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a5cd0 sp=0xc0003a5cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a5f18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a5d60 sp=0xc0003a5cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a5d88 sp=0xc0003a5d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a5fc8 sp=0xc0003a5d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a5fe0 sp=0xc0003a5fc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a5fe8 sp=0xc0003a5fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 241 [chan receive]:
runtime.gopark(0xc00039ece8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00039ed50?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00039ecd0 sp=0xc00039ecb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00039ef18, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00039ed60 sp=0xc00039ecd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00039ed88 sp=0xc00039ed60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00039efc8 sp=0xc00039ed88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00039efe0 sp=0xc00039efc8 pc=0x1377be6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00039efe8 sp=0xc00039efe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 242 [select]:
runtime.gopark(0xc00099c778?, 0x2?, 0x0?, 0x0?, 0xc00099c744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00099c5b0 sp=0xc00099c590 pc=0x43bc56
runtime.selectgo(0xc00099c778, 0xc00099c740, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00099c6f0 sp=0xc00099c5b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc00099c7c8 sp=0xc00099c6f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc00099c7e0 sp=0xc00099c7c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00099c7e8 sp=0xc00099c7e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 243 [select]:
runtime.gopark(0xc00099cf78?, 0x2?, 0x0?, 0x0?, 0xc00099cf44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00099cdb0 sp=0xc00099cd90 pc=0x43bc56
runtime.selectgo(0xc00099cf78, 0xc00099cf40, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00099cef0 sp=0xc00099cdb0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc00099cfc8 sp=0xc00099cef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc00099cfe0 sp=0xc00099cfc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00099cfe8 sp=0xc00099cfe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 244 [select]:
runtime.gopark(0xc0003c3778?, 0x2?, 0x0?, 0x0?, 0xc0003c3744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c35b0 sp=0xc0003c3590 pc=0x43bc56
runtime.selectgo(0xc0003c3778, 0xc0003c3740, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c36f0 sp=0xc0003c35b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c37c8 sp=0xc0003c36f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c37e0 sp=0xc0003c37c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c37e8 sp=0xc0003c37e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 245 [select]:
runtime.gopark(0xc0003c3f78?, 0x2?, 0x12?, 0x0?, 0xc0003c3f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c3db0 sp=0xc0003c3d90 pc=0x43bc56
runtime.selectgo(0xc0003c3f78, 0xc0003c3f40, 0x0?, 0x0, 0xd40010?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c3ef0 sp=0xc0003c3db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c3fc8 sp=0xc0003c3ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c3fe0 sp=0xc0003c3fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c3fe8 sp=0xc0003c3fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 246 [select]:
runtime.gopark(0xc0003c4778?, 0x2?, 0x2?, 0x0?, 0xc0003c4744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c45b0 sp=0xc0003c4590 pc=0x43bc56
runtime.selectgo(0xc0003c4778, 0xc0003c4740, 0xd40010?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c46f0 sp=0xc0003c45b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c47c8 sp=0xc0003c46f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c47e0 sp=0xc0003c47c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c47e8 sp=0xc0003c47e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 247 [select]:
runtime.gopark(0xc0003c4f78?, 0x2?, 0x0?, 0x0?, 0xc0003c4f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c4db0 sp=0xc0003c4d90 pc=0x43bc56
runtime.selectgo(0xc0003c4f78, 0xc0003c4f40, 0xc0003c4e70?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c4ef0 sp=0xc0003c4db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c4fc8 sp=0xc0003c4ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c4fe0 sp=0xc0003c4fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c4fe8 sp=0xc0003c4fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 248 [select]:
runtime.gopark(0xc0003c5778?, 0x2?, 0x0?, 0x0?, 0xc0003c5744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c55b0 sp=0xc0003c5590 pc=0x43bc56
runtime.selectgo(0xc0003c5778, 0xc0003c5740, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c56f0 sp=0xc0003c55b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c57c8 sp=0xc0003c56f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c57e0 sp=0xc0003c57c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c57e8 sp=0xc0003c57e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 249 [select]:
runtime.gopark(0xc0003c5f78?, 0x2?, 0x10?, 0x0?, 0xc0003c5f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c5db0 sp=0xc0003c5d90 pc=0x43bc56
runtime.selectgo(0xc0003c5f78, 0xc0003c5f40, 0x0?, 0x0, 0x12?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c5ef0 sp=0xc0003c5db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c5fc8 sp=0xc0003c5ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c5fe0 sp=0xc0003c5fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c5fe8 sp=0xc0003c5fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 250 [select]:
runtime.gopark(0xc0003c6778?, 0x2?, 0x1?, 0x0?, 0xc0003c6744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c65b0 sp=0xc0003c6590 pc=0x43bc56
runtime.selectgo(0xc0003c6778, 0xc0003c6740, 0x12?, 0x0, 0x2?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c66f0 sp=0xc0003c65b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c67c8 sp=0xc0003c66f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c67e0 sp=0xc0003c67c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c67e8 sp=0xc0003c67e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 251 [select]:
runtime.gopark(0xc0003c6f78?, 0x2?, 0x0?, 0x0?, 0xc0003c6f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c6db0 sp=0xc0003c6d90 pc=0x43bc56
runtime.selectgo(0xc0003c6f78, 0xc0003c6f40, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c6ef0 sp=0xc0003c6db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c6fc8 sp=0xc0003c6ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c6fe0 sp=0xc0003c6fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c6fe8 sp=0xc0003c6fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 252 [select]:
runtime.gopark(0xc000447778?, 0x2?, 0xe0?, 0x4e?, 0xc000447744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004475b0 sp=0xc000447590 pc=0x43bc56
runtime.selectgo(0xc000447778, 0xc000447740, 0xc00030d2b0?, 0x0, 0xc00030d370?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0004476f0 sp=0xc0004475b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0004477c8 sp=0xc0004476f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0004477e0 sp=0xc0004477c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004477e8 sp=0xc0004477e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 253 [select]:
runtime.gopark(0xc000447f78?, 0x2?, 0x40?, 0x4?, 0xc000447f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000447db0 sp=0xc000447d90 pc=0x43bc56
runtime.selectgo(0xc000447f78, 0xc000447f40, 0xc00030c520?, 0x0, 0xc00030c540?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000447ef0 sp=0xc000447db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc000447fc8 sp=0xc000447ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc000447fe0 sp=0xc000447fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000447fe8 sp=0xc000447fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 254 [select]:
runtime.gopark(0xc0003c0778?, 0x2?, 0x0?, 0x0?, 0xc0003c0744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c05b0 sp=0xc0003c0590 pc=0x43bc56
runtime.selectgo(0xc0003c0778, 0xc0003c0740, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c06f0 sp=0xc0003c05b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c07c8 sp=0xc0003c06f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c07e0 sp=0xc0003c07c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c07e8 sp=0xc0003c07e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 255 [select]:
runtime.gopark(0xc0003c0f78?, 0x2?, 0x0?, 0x0?, 0xc0003c0f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c0db0 sp=0xc0003c0d90 pc=0x43bc56
runtime.selectgo(0xc0003c0f78, 0xc0003c0f40, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c0ef0 sp=0xc0003c0db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c0fc8 sp=0xc0003c0ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c0fe0 sp=0xc0003c0fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c0fe8 sp=0xc0003c0fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 256 [select]:
runtime.gopark(0xc0003c1778?, 0x2?, 0x0?, 0x0?, 0xc0003c1744?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c15b0 sp=0xc0003c1590 pc=0x43bc56
runtime.selectgo(0xc0003c1778, 0xc0003c1740, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c16f0 sp=0xc0003c15b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c17c8 sp=0xc0003c16f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c17e0 sp=0xc0003c17c8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c17e8 sp=0xc0003c17e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 257 [select]:
runtime.gopark(0xc0003c1f78?, 0x2?, 0x0?, 0x0?, 0xc0003c1f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c1db0 sp=0xc0003c1d90 pc=0x43bc56
runtime.selectgo(0xc0003c1f78, 0xc0003c1f40, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c1ef0 sp=0xc0003c1db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c1fc8 sp=0xc0003c1ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c1fe0 sp=0xc0003c1fc8 pc=0x1377b86
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c1fe8 sp=0xc0003c1fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
	/src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 263 [select]:
runtime.gopark(0xc0003a3f88?, 0x2?, 0x68?, 0xdd?, 0xc0003a3f4c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a3dd0 sp=0xc0003a3db0 pc=0x43bc56
runtime.selectgo(0xc0003a3f88, 0xc0003a3f48, 0xc000064428?, 0x0, 0xc0003a3f58?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003a3f10 sp=0xc0003a3dd0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads.(*tableManager).loop(0xc000741dc0)
	/src/loki/pkg/storage/stores/indexshipper/uploads/table_manager.go:65 +0x15d fp=0xc0003a3fc8 sp=0xc0003a3f10 pc=0x1b8e33d
github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads.NewTableManager.func1()
	/src/loki/pkg/storage/stores/indexshipper/uploads/table_manager.go:51 +0x26 fp=0xc0003a3fe0 sp=0xc0003a3fc8 pc=0x1b8e1a6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a3fe8 sp=0xc0003a3fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads.NewTableManager
	/src/loki/pkg/storage/stores/indexshipper/uploads/table_manager.go:51 +0x294

goroutine 141 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002c8cd0 sp=0xc0002c8cb0 pc=0x43bc56
runtime.netpollblock(0x0?, 0x40482f?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0002c8d08 sp=0xc0002c8cd0 pc=0x434297
internal/poll.runtime_pollWait(0x7fa3a9b0db38, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0002c8d28 sp=0xc0002c8d08 pc=0x469d69
internal/poll.(*pollDesc).wait(0xc0001c0800?, 0x26c0?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0002c8d50 sp=0xc0002c8d28 pc=0x4e5f12
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001c0800)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0002c8df8 sp=0xc0002c8d50 pc=0x4eb81d
net.(*netFD).accept(0xc0001c0800)
	/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0002c8eb0 sp=0xc0002c8df8 pc=0x6ee9b5
net.(*TCPListener).accept(0xc0002f5308)
	/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0002c8ed8 sp=0xc0002c8eb0 pc=0x70a1a5
net.(*TCPListener).Accept(0xc0002f5308)
	/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc0002c8f08 sp=0xc0002c8ed8 pc=0x708f5d
github.com/grafana/dskit/kv/memberlist.(*TCPTransport).tcpListen(0xc000846400, {0x2801bd0, 0xc0002f5308})
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/tcp_transport.go:202 +0xa9 fp=0xc0002c8fb8 sp=0xc0002c8f08 pc=0xe81ac9
github.com/grafana/dskit/kv/memberlist.NewTCPTransport.func2()
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/tcp_transport.go:178 +0x2e fp=0xc0002c8fe0 sp=0xc0002c8fb8 pc=0xe819ae
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002c8fe8 sp=0xc0002c8fe0 pc=0x46fc41
created by github.com/grafana/dskit/kv/memberlist.NewTCPTransport
	/src/loki/vendor/github.com/grafana/dskit/kv/memberlist/tcp_transport.go:178 +0x7d3

goroutine 142 [select]:
runtime.gopark(0xc000079f98?, 0x2?, 0x0?, 0x0?, 0xc000079f7c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000079e00 sp=0xc000079de0 pc=0x43bc56
runtime.selectgo(0xc000079f98, 0xc000079f78, 0x405d3d?, 0x0, 0xc000079fa0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000079f40 sp=0xc000079e00 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).streamListen(0xc0000d5a20)
	/src/loki/vendor/github.com/hashicorp/memberlist/net.go:222 +0x96 fp=0xc000079fc8 sp=0xc000079f40 pc=0xe4e8d6
github.com/hashicorp/memberlist.newMemberlist.func3()
	/src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:227 +0x26 fp=0xc000079fe0 sp=0xc000079fc8 pc=0xe4b3a6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.newMemberlist
	/src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:227 +0x9c5

goroutine 143 [select]:
runtime.gopark(0xc00007cf98?, 0x2?, 0x0?, 0x0?, 0xc00007cf8c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007ce00 sp=0xc00007cde0 pc=0x43bc56
runtime.selectgo(0xc00007cf98, 0xc00007cf88, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00007cf40 sp=0xc00007ce00 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).packetListen(0xc0000d5a20)
	/src/loki/vendor/github.com/hashicorp/memberlist/net.go:346 +0xc5 fp=0xc00007cfc8 sp=0xc00007cf40 pc=0xe4fc25
github.com/hashicorp/memberlist.newMemberlist.func4()
	/src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:228 +0x26 fp=0xc00007cfe0 sp=0xc00007cfc8 pc=0xe4b346
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.newMemberlist
	/src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:228 +0xa0a

goroutine 144 [select]:
runtime.gopark(0xc00007d718?, 0x2?, 0x0?, 0x0?, 0xc00007d714?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007d598 sp=0xc00007d578 pc=0x43bc56
runtime.selectgo(0xc00007d718, 0xc00007d710, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00007d6d8 sp=0xc00007d598 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).packetHandler(0xc0000d5a20)
	/src/loki/vendor/github.com/hashicorp/memberlist/net.go:490 +0x77 fp=0xc00007d7c8 sp=0xc00007d6d8 pc=0xe50d17
github.com/hashicorp/memberlist.newMemberlist.func5()
	/src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:229 +0x26 fp=0xc00007d7e0 sp=0xc00007d7c8 pc=0xe4b2e6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x46fc41
created by github.com/hashicorp/memberlist.newMemberlist
	/src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:229 +0xa4d

goroutine 273 [select]:
runtime.gopark(0xc00007df78?, 0x2?, 0x40?, 0x0?, 0xc00007df54?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007ddd8 sp=0xc00007ddb8 pc=0x43bc56
runtime.selectgo(0xc00007df78, 0xc00007df50, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00007df18 sp=0xc00007ddd8 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0x0?, 0x12a05f200, 0xc000480240, 0xc0009eac00, 0xc0002bafc0)
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:149 +0xb7 fp=0xc00007dfa8 sp=0xc00007df18 pc=0xe5c517
github.com/hashicorp/memberlist.(*Memberlist).schedule.func2()
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:121 +0x36 fp=0xc00007dfe0 sp=0xc00007dfa8 pc=0xe5c3d6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:121 +0x218

goroutine 274 [select]:
runtime.gopark(0xc0003c7f90?, 0x2?, 0x0?, 0x0?, 0xc0003c7f6c?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c7de8 sp=0xc0003c7dc8 pc=0x43bc56
runtime.selectgo(0xc0003c7f90, 0xc0003c7f68, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c7f28 sp=0xc0003c7de8 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger(0xc0000d5a20, 0xc0009eac00)
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:173 +0xb7 fp=0xc0003c7fc0 sp=0xc0003c7f28 pc=0xe5c697
github.com/hashicorp/memberlist.(*Memberlist).schedule.func3()
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:127 +0x2a fp=0xc0003c7fe0 sp=0xc0003c7fc0 pc=0xe5c36a
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c7fe8 sp=0xc0003c7fe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:127 +0x345

goroutine 275 [select]:
runtime.gopark(0xc0003c7778?, 0x2?, 0xc0?, 0x82?, 0xc0003c7754?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c75d8 sp=0xc0003c75b8 pc=0x43bc56
runtime.selectgo(0xc0003c7778, 0xc0003c7750, 0x0?, 0x0, 0xc0003c7750?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c7718 sp=0xc0003c75d8 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc000469dd0?, 0xbebc200, 0xc0004802a0, 0xc0009eac00, 0xc0002bb010)
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:149 +0xb7 fp=0xc0003c77a8 sp=0xc0003c7718 pc=0xe5c517
github.com/hashicorp/memberlist.(*Memberlist).schedule.func4()
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:133 +0x36 fp=0xc0003c77e0 sp=0xc0003c77a8 pc=0xe5c316
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c77e8 sp=0xc0003c77e0 pc=0x46fc41
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/src/loki/vendor/github.com/hashicorp/memberlist/state.go:133 +0x498

My environment:
EKS 1.23
CPU arch: amd64

@dbg-raghulkrishna
Copy link

any update on this? i am facing the same issue

@hinorashi
Copy link

encountering the same here:

  • EKS 1.29
  • helm chart 5.48.0
  • loki 2.9.6

@hinorashi
Copy link

encountering the same here:

  • EKS 1.29
  • helm chart 5.48.0
  • loki 2.9.6

I fixed it but without any help from loki pod log or event: the disk is full -> extend the PV

@AtlanCI
Copy link
Author

AtlanCI commented Sep 21, 2024

This problem is because the data has been corrupted, clear all the data can be。

@AtlanCI AtlanCI closed this as completed Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Somehing is not working as expected
Projects
None yet
Development

No branches or pull requests

6 participants