Setup PrometheusRules within Mimir ruler

[marieroque]

First work: #2521 (comment)

[QuantumEnigmaa]

I have a PR upstream waiting for approval that will add a specific sa for the ruler allowing to have a specific bucket for it. The PR : grafana/mimir#7132

[QuentinBisson]

Upstream fix released in mimir-app 0.4.2 we can move forward

[QuantumEnigmaa]

With the current config mimir-ruler is crashlooping fith the following message :

ts=2024-02-14T12:53:56.724145913Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"

Still, the mimir-ruler pod is using the mimir-ruler sa wich has the correct irsa annotation with the correct AWS role arn, which in turn has the correct policy to access the ruler bucket.

[QuentinBisson]

Does the ruler need access to the block_storage section or is it denied access when accessing the new bucket ?

[QuantumEnigmaa]

That's what I'm trying to understand :/

[QuentinBisson]

Well yes it is https://github.com/grafana/mimir/blob/e8e77847191e2bbcb9263a6c12ea5c207d93ca53/pkg/mimir/sanity_check.go#L130 :(

[QuentinBisson]

Maybe we can add some extraPermissions in here https://github.com/giantswarm/object-storage-operator/blob/f377196f346dc79e2bbb8d66d5bb81b4dc4a028c/api/v1alpha1/bucket_types.go#L45 to be able to allow the role access to the other bucket?

[QuantumEnigmaa]

Maybe we can add some extraPermissions [...] to be able to allow the role access to the other bucket?

I guess that would be the most straightforward way to do it 👍
Can you take care of it or should I do it ?

[QuentinBisson]

How do you prefer we proceed ? If you want to do it, have fun, otherwise I can :)

[QuantumEnigmaa]

I'll always want to do things myself but right now I'm busy with PMO so for the sake of efficiency, I think it's better if you take care of it (considering you have time for this obviously). 🙂

[QuentinBisson]

I'll try to work on it tomorrow then

[QuentinBisson]

Fixes object storage to support extra buckets giantswarm/object-storage-operator#89