[mimir-distributed] Ingester rollout-group conflicts with other charts

[lindeskar]

Describe the bug

The mimir-distributed Helm chart generates Ingester StatefulSets with labels for rollout-operator. Ex.:

mimir/operations/helm/tests/large-values-generated/mimir-distributed/templates/ingester/ingester-statefulset.yaml

Lines 51 to 52 in 8571f6c

	name: "ingester-zone-a"
	rollout-group: ingester

The loki Helm chart with deploymentMode: Distributed and zoneAwareReplication enabled (default) generates StatefulSets with the same label values. Ex.:

name: ingester-zone-a
rollout-group: ingester

Deploying the two charts to the same Namespace means rollout-operator will select both Mimir and Loki StatefulSets and get confused about the rollout status. For me one of the Mimir Ingester Pods is constantly being recreated.

Issue grafana/loki#13168 is created for the same issue in the Loki repository.

To Reproduce

Steps to reproduce the behavior:

1. Deploy mimir-distributed chart with default values
2. Deploy loki chart with distributed-values.yaml

Expected behavior

rollout-operator handles Mimir and Loki Ingesters as separate rollout-groups.

Environment

• Infrastructure: Kubernetes
• Deployment tool: Helm
• Chart version: 5.3.0

Additional Context

From rollout-operator Pod:

level=info ts=2024-06-07T08:17:49.880277442Z msg="StatefulSet status is reporting all pods ready, but the rollout operator has found some not-Ready pods" statefulset=loki-ingester-zone-a not_ready_pods=mimir-ingester-zone-a-0

[dimitarvdimitrov]

you should deploy them in different namespaces

[lindeskar]

Sure, probably the easiest solution, though our setup runs LGTM in one Namespace today.

[lindeskar]

The related issue in Loki was resolved with an optional prefix added in grafana/loki#15063

Could this be ok to add for Mimir too?