[Question] Applying NodeSelector and Tolerations to Flux extenstion

[philip-bradshaw]

Describe scenario

We are using the flux v2 cluster extension for Azure AKS clusters and are facing an issue where right now the flux components are not restricted as to what nodes they can deploy onto, and so are being scheduled onto different nodes across different node pools. Ideally we want to restrict the components to the system node pool which we've locked down with the taint CriticalAddonsOnly=true:NoSchedule as recommended here; https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli#system-and-user-node-pools

I know the components get deployed onto the cluster using the helm chart azure-k8s-flux and we've been able to pass in values into it via the terraform resource azurerm_kubernetes_cluster_extension, however it appears to be a private helm chart and we don't know if it supports setting nodeSelector's or tolerations.

Question
Has anyone faced a similar issue trying to restrict where flux is deployed? Does the azure-k8s-flux helm chart support setting nodeSelector's or tolerations?

[PixelRobots]

I did a blog post a while ago about installing it using the critical add-ons toleration but using the cli.

https://pixelrobots.co.uk/2022/09/installing-flux-gitops-aks-cluster-extension-to-a-tainted-node-pool/

So it should support tolerations.

[philip-bradshaw]

@PixelRobots Thanks for the link! That certainly helps with ensuring Flux can be deployed onto a system node pool with the taint CriticalAddonsOnly=true:NoSchedule, however it still leaves the pods as being eligible to be deployed onto other nodes as well. Ideally we need a way to set the nodeSelector or affinity to really restrict where the pods are deployed.

[PixelRobots]

Good point. You could add taints to your other node pools but then that can cause headaches for you later on.

Have you tried using helm get values --all on the helm release to see all of the values you can use for it?

I am currently not at a computer, away on holiday so can't test for you.

Thanks

Richard

[philip-bradshaw]

@PixelRobots I have been digging into the helm values for the flux installation and it looks like there is a common block that is passed in by default that set's the nodeSelector and tolerations;

common:
  fluxReleaseVersion: v2.4.0
  nodeSelector:
    kubernetes.io/os: linux
  tolerations:
  - key: CriticalAddonsOnly
    operator: Exists

[philip-bradshaw]

Setting common.nodeSelector does indeed work and the nodeSelector defined get's added to the workloads. It's worth mentioning that for the following nodeSelector;

Node-Selectors:
  node-pool=system

It has to be defined as common.nodeSelector.node-pool=system, otherwise you get issues parsing the key.