Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm podAntiAffinity matchLabels should include instance and name #13349

Open
Skaronator opened this issue Jun 28, 2024 · 0 comments
Open

Helm podAntiAffinity matchLabels should include instance and name #13349

Skaronator opened this issue Jun 28, 2024 · 0 comments

Comments

@Skaronator
Copy link

Describe the bug
I'm deploy the loki helm chart in my existing monitoring namespace. The monitoring namespace already contains loki-distributed and mimir-distributed helm chart. So there are already a few (50) pods running.

After deploying the loki helm-chart, I noticed that the gateway pods won't schedule.

0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

After quick debug I noticed that the affinity rule only matches the component label:

affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/component: gateway
topologyKey: kubernetes.io/hostname

The issue is that this is not enough. The mimir-distributed chart has a component with gateway label as well, resulting in "no free nodes".

To Reproduce
Steps to reproduce the behavior:

  1. Use a single node or 3 node cluster with 3 replicas for the mimir gateway.
  2. Deploy mimir-distributed in a namespace
  3. Deploy loki helm chart in the same namespace
  4. See that the loki gateway pods cannot be scheduled.

Expected behavior

The default matchLabels in podAntiAffinity should be more restricted. e.g. similar to the serviceMatchLabels.

The gateway service has more strict labels as you can see:

  selector:
    app.kubernetes.io/component: gateway
    app.kubernetes.io/instance: loki
    app.kubernetes.io/name: loki

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: Helm

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants