AKS clusters using Azure CNI should use Azure CNI Overlay

[BenjaminEngeset]

Rule request

Suggested rule change

AKS clusters using Azure CNI should use Azure CNI Overlay networking which currently is in preview. Azure CNI Overlay currently has some limitations that must be taken in consideration.

With Azure CNI Overlay, the cluster nodes are deployed into an Azure Virtual Network subnet, whereas pods are assigned IP addresses from a private CIDR logically different from the VNet hosting the nodes. Pod and node traffic within the cluster use an overlay network, and Network Address Translation (via the node's IP address) is used to reach resources outside the cluster. This solution saves a significant amount of VNet IP addresses and enables you to seamlessly scale your cluster to very large sizes. An added advantage is that the private CIDR can be reused in different AKS clusters, truly extending the IP space available for containerized applications in AKS.

Applies to the following

The rule applies to the following:

• Resource type: [Microsoft.ContainerService/managedClusters]

Additional context

Azure CNI Overlay preview
Azure CNI Overlay
Template reference

[BernieWhite]

@bengeset96 Nice, we just need to understand how this aligns to the Well-Architected Framework? What pillar do you feel this falls into?

@ms-sambell What are your thoughts?

[BenjaminEngeset]

@BernieWhite

Performance efficiency pillar. I think many of the topics supports overlay, particularly Consider scalability and Plan for capacity.

Disadvantages with Azure CNI Overlay so far as I know are following:

• Does not work well with virtual nodes.
• When you have resources outside the cluster that needs to reach pods directly.
• Application Gateway as an ingress controller is not supported.
• Windows not supported.

Might be "sketchy" to implement this rule, but I also feel it could add some value for those that want to run explicitly with Preview baseline.

Could also be interesting to inform customers using Kubenet with a Warning or Information level about Azure CNI Overlay as it offers some benefits (also slightly disadvantages).

Would be great to get feedback from @ms-sambell as well.

[BenjaminEngeset]

@ms-sambell

Any thoughts that you would like to share?

[ms-sambell]

Hey @bengeset96 - My notifications are muted so I missed this (will fix it).

So the new CNI overlay network is great but there are some limitations right now. So I don't think we can create a generic rule to recommend all customers use it. I believe one of the limitations is the AppGateway add-on breaks.

There's a good story around writing a recommendation for customers using kubenet to review it though, so maybe that's an awareness rule? or a rule that assesses if the app-gateway add-on isn't enabled to recommend it.

[BenjaminEngeset]

Yes, awareness would be fine for this. Agreed upon this, I'll try to create something and we'll see if it fits.

[BenjaminEngeset]

Hey @ms-sambell

I've started on looking on this and I can see that there are two ways to deploy AGIC for an AKS cluster. With the trough AKS approach as an add-on it seems that I can just look out for properties.addonProfiles.ingressApplicationGateway.enabled -eq $true, but for the Helm option I'm really struggling to find out what to look for. Any idea?