Allow for multiple store configurations from the same provider

[dannykopping]

Is your feature request related to a problem? Please describe.
Loki currently supports period_configs for specifying index & chunk schema changes over time. A user can select a store with each config, and this effectively allows users to switch providers should they need to. Queries that span multiple periods will route requests to the appropriate store.

Example:

storage_config:
    azure:
        account_key: ${AZURE_BLOB_SECRET_ACCESS_KEY}
        account_name: lokiprod
        container_name: data

schema_config:
    configs:
        - from: "2020-07-31"
            index:
            period: 24h
            prefix: loki_prod_index_
            object_store: azure
            schema: v11
            store: boltdb-shipper
        - from: "2022-01-24"
            index:
            period: 24h
            prefix: loki_prod_index_
            object_store: azure
            schema: v12 # <-- changed from v11 to v12
            store: boltdb-shipper

We currently do not support multiple store configurations for the same provider, however.

At Grafana Labs, we require this feature due to a misconfiguration of an Azure Blob bucket. Its replication policy was set to be LRS (locally redundant within a zone), but we actually need it to be ZRS (zonally redundant within a region). Migrating the data from one bucket to another is impractical and slow, so we would prefer to setup a new bucket, and read/write new chunks/indexes to this bucket. Data in the old bucket will be deleted according to its retention period.

Describe the solution you'd like
I would like to be able to define multiple store configurations from the same provider, and use these configurations within period_configs definitions.

Additionally, if we could add a path prefix to prepend to all path lookups in the bucket, that would also solve some outstanding feature-requests (listed below), and it seems like related work. This is not required to deliver on the main value of this ticket, but it might be possible with some drive-by refactoring that needs to be done anyway.

storage_config:
    azure:
        account_key: ${AZURE_BLOB_SECRET_ACCESS_KEY}
        account_name: lokiprod
        container_name: data
        type: azure # <--- new "type" field, derive from key name for backwards-compatibility?
    azure-new:
        account_key: ${AZURE_BLOB_SECRET_ACCESS_KEY}
        account_name: lokiprod
        container_name: data
        chunk_path: "/path/to/chunks" # <--- new prefix (optional)
        index_path: "/path/to/indexes" # <--- new prefix (optional)
        type: azure

schema_config:
    configs:
        - from: "2020-07-31"
            index:
            period: 24h
            prefix: loki_prod_index_
            object_store: azure
            schema: v11
            store: boltdb-shipper
        - from: "2022-01-24"
            index:
            period: 24h
            prefix: loki_prod_index_
            object_store: azure
            schema: v12 # <-- changed from v11 to v12
            store: boltdb-shipper
        - from: "2022-09-29"
            index:
            period: 24h
            prefix: loki_prod_index_
            object_store: azure-new # <-- changed to use new object store
            schema: v12
            store: boltdb-shipper

Describe alternatives you've considered
There is no meaningful alternative right now.

Additional context
Related issues: #5889, possibly even #5757

I've hacked up a basic implementation which works, but it's very rough. It also adds to the common -> storage config, which I think might be a better place for this than storage_config, but I'm not sure what'll work out to be easiest or most useful.

[dannykopping]

Reopening since the PR that closed this goes some way to addressing this, but isn't the full solution AFAIK.

[priyanshigarg1303]

getting the error "CrashLoopBackOff" everytime the loki pod is restarted or even after recreation. The version upgrade. changes in the sku_name from Premium_LRS to Premium_ZRS has also been tried. But still the issue is not getting resolved. Im using statefulset for the same.

**level=error ts=2023-09-06T04:04:20.391118868Z caller=index_set.go:285 table-name=loki_index_19580 msg="sync failed, retrying it" err="invalid database"
level=error ts=2023-09-06T04:04:20.391148368Z caller=index_set.go:103 table-name=loki_index_19580 msg="failed to initialize table loki_i
ndex_19580, cleaning it up" err="invalid database"
level=error ts=2023-09-06T04:04:20.391275769Z caller=log.go:171 msg="error running loki" err="invalid database\nerror creating index cli
ent\ngithub.com/grafana/loki/pkg/storage.(store).storeForPeriod\n\t/src/loki/pkg/storage/store.go:270\ngithub.com/grafana/loki/pkg/stor
age.(store).init\n\t/src/loki/pkg/storage/store.go:164\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:
147\ngithub.com/grafana/loki/pkg/loki.