Upgrade to Kubernetes 1.29

[smerle33]

Per https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar, Microsoft stops supporting Kubernetes 1.28 in November 2024.

As such, we must upgrade our Kubernetes clusters to 1.29.

Last issue for 1.28 was: #4144

---

Task list:

• Upgrade kubectl within packer-images All-in-One

  • First upgrade the updatecli manifest - chore(updatecli): Track 1.29x line of kubectl versions packer-images#1358
  • Then apply the updatecli PR that is produced by this change: - Bump kubectl version to 1.29.8 packer-images#1359
  • Trigger a release of the packer-images - https://github.com/jenkins-infra/packer-images/releases/tag/1.90.0
  • Then apply the Kubernetes-Management PR that will use this new version - Bump agent templates for infra.ci.jenkins.io (packer-image 1.90.0) kubernetes-management#5604

• Check AKS changelog for 1.29 - (details)

• Upgrade AKS cluster infracijio-agents-1 (details)

• Upgrade AKS cluster cijenkinsio-agents-1 (details)

• Upgrade AKS cluster privatek8s (details)

• Upgrade AKS cluster publick8s (details)

• Add Logo ;)

• Open issue for next Kubernetes version

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.27 #3948
2. 92% Upgrade to Kubernetes 1.26 #3683
3. 92% Upgrade to Kubernetes 1.25 #3582
4. 92% Upgrade to Kubernetes 1.24 #3387
5. 92% Upgrade to Kubernetes 1.23 #3053

[dduportal]

No milestone yet: we should schedule this in August 2024

[dduportal]

Ping @jayfranco999 as discussed during the team meeting, you can start working on this topic with the following tasks:

• Bump the kubectl version line (start with the updatecli manifest, and then let's use the automated PRs) from 1.28.* to 1.29.*
• Prepare the migration of the infra.ci.jenkins.io-agents1 AKS cluster by opening draft PRs in the same way as we did for 1.28

[dduportal]

kubectl 1.29(.8) is now used on our agents:

• Bump kubectl version to 1.29.8 packer-images#1359
• Bump agent templates for infra.ci.jenkins.io (packer-image 1.90.0) kubernetes-management#5604

[dduportal]

Check of the Kubernetes 1.29 and AKS changelogs:

Nothing for us on

• https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#urgent-upgrade-notes (kubeadm only)
• https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#deprecation => no resources
• NOTE: Azure portal mentions we're using a deprecated resource (ComponentStatus). Culprit is the datadog helm chart (see Is ComponentStatuses RBAC desirable? DataDog/helm-charts#961) but it should not block anything from 1.29. Remains to be checked for 1.30!

From https://github.com/Azure/AKS/blob/master/CHANGELOG.md:

Starting Kubernetes 1.29, the default cgroups implementation on Azure Linux AKS nodes will be cgroupsv2. Older versions of Java, .NET and NodeJS do not support memory querying v2 memory constraints and this will lead to out of memory (OOM) issues for workloads. Please test your applications for cgroupsv2 compliance, and read the FAQ for cgroupsv2.

✅ We already were using cgroups v2 on ubuntu node since #2982

Changes to kube-reserved memory reservations are now in effect in AKS 1.29. The optimized reservation logic reduces kube-reserved memory by up to 20% depending on the node configuration. For existing 1.29 node pools created prior to 2/26, please perform a node pool update or recreate to see these changes.

✅ Optimization, we don't really care

Workload Identity is now supported as a setting for static PVs on Managed Blob/File CSI drivers in 1.29.

✅ Could be interesting for our statically defined PVs but no changes

Clusters running Kubernetes 1.29 or later will have kubernetes.azure.com/managedby=aks label to tigera-operator deployment in Calico clusters

✅ We don't use Calico

Effective starting with Kubernetes version 1.29, when you deploy Azure Kubernetes Service (AKS) clusters across multiple availability zones, AKS now utilizes zone-redundant storage (ZRS) to create managed disks within built-in storage classes. ZRS ensures synchronous replication of your Azure managed disk across multiple Azure availability zones in your chosen region. This redundancy strategy enhances the resilience of your applications and safeguards your data against datacenter failures. Refer to Storage concept for more information.

✅ We don't use zone redundant cluster

Advanced Container Networking Services can be enabled on Cilium-enabled clusters with Kubernetes v1.29.0 or greater, and on Retina-enabled clusters with Kubernetes v1.21.0 or greater for Advanced Network Observability.

✅ We don't use Cilium

Other changes are usual component version bumps (azure-csi, cni, etc.) , CVEs fixes and bufixes in the 1.29.x line

=> we can proceed!

[dduportal]

infracijio-agents-1

• Announcement: no announcement needed as the cluster is not used yet
• Disable kubernetes management for infracijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - chore(infracijioagents1): removing temporarily from management to upgrade to kubernetes 1.29 kubernetes-management#5598

Warning

Terraform upgrade must be performed manually otherwise it will upgrade the cluster where agents are currently performing, leading to build failure

• Upgrade cluster to 1.29 with Terraform (Azure)
  • PR with checks all green: feat(infracijioagents1): Upgrade cluster to kub 1.29.8 azure#806
  • Prepare local (admin laptop) terraform plan
  • Disable Terraform Azure Job in infra.ci
  • Merge the PR
  • ⚠️ Note that AKS version is not always exactly the same patch as kubectl . Always use the AKS changelog to check (or the az CLI). Hotfix PR: hotfix(infra.ci.jenkins.io-agents1) use a supported version of AKS azure#807
  • Deploy from local admin laptop
  • Enable Terraform Azure Job in infra.ci and run it once with success
• Enable kubernetes management for infracijio-agents-1 cluster and check it runs successfully - https://github.com/jenkins-infra/kubernetes-management/pull/ID

---

Edit: upgrade finished with success!

[dduportal]

cijioagents1

We plan to handle this upgrade Tuesday 22 August around 14:00 UTC

• Announcements:
  • status.jenkins.io - open ci.jio-agents1 1.29.x upgrade operation status#532
  • developer mailing list - https://groups.google.com/g/jenkinsci-dev/c/534G42tOf1A
  • IRC #jenkins-infra -
  • Add a banner on ci.jenkins.io -
• Disable kubernetes management for cijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - chore: disable cijioagents1 management kubernetes-management#5609
• Disable the 2 clouds defined in ci.jenkins.io on puppet (jenkins-infra/jenkins-infra) for the operation ci.jenkins.io-agents-1-bom and ci.jenkins.io-agents-1 - feat(ci.jenkins.io) disable Kubernetes Cloud Agents for https://github.com/jenkins-infra/helpdesk/issues/4161#issuecomment-2304516597 jenkins-infra#3619
• Terraform PR to upgrade cluster to 1.29.7 but NO merge - feat(ci.jio-agents-1) bump AKS to 1.29.7 azure#808
  • Verify all checks are ok
  • Merge the PR
  • Verify upgraded is performed with no error
• Enable kubernetes management for ci.jio-agents-1 cluster and check it runs successfully - Revert "chore: disable cijioagents1 management" kubernetes-management#5611
• Re-Enable the 2 ci.jio puppet clouds and check container agents are scheduled with the Infra/acceptance-test job - Revert "feat(ci.jenkins.io) disable Kubernetes Cloud Agents for https://github.com/jenkins-infra/helpdesk/issues/4161#issuecomment-2304516597" jenkins-infra#3620
• Close Annoucements
  • status.jenkins.io - close AKS 1.29 upgrade for ci.jio status#534
  • developer mailing list + jenkins-infra mailing
  • IRC #jenkins-infra
  • Remove the banner on ci.jenkins.io

[dduportal]

Update: privatek8s cluster upgrade:

• Announcement:
  • Open datetime announcement on status.jenkins.io - open AKS 1.29 operations for public and private clusters status#535
  • IRC #jenkins-infra
• Disable kubernetes management for privatek8s cluster (remove it from Jenkinsfile_k8s) - chore: exclude privatek8s cluster while upgrading to 1.29 kubernetes-management#5614
• Terraform PR to upgrade cluster - feat(privatek8s) bump to AKS 1.29.7 azure#809
• Merge Terraform PR and let the upgrade happen
• Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1334
• Enable kubernetes management and revert exclusion of privatek8s cluster, checking both diff and apply run successfully - Revert "chore: exclude privatek8s cluster while upgrading to 1.29" kubernetes-management#5615
• Check for services healthiness (pagerduty, datadog, manual control, logging in on infra.ci.jenkins.io/release.ci.jenkins.io with the Infra Agents Health job https://release.ci.jenkins.io/job/Infra%20Agents%20Health/)
• Close announcements

[dduportal]

Update: publick8s cluster upgrade:

• Announcement:
  • Open datetime announcement on status.jenkins.io - open AKS 1.29 operations for public and private clusters status#535
  • developer mailing list (Authentication outage due to LDAP restart) - https://groups.google.com/g/jenkinsci-dev/c/fycWymHPS4w
  • IRC #jenkins-infra
• Disable kubernetes management for publick8s cluster (remove it from Jenkinsfile_k8s) - chore: exclude publick8s cluster while upgrading to 1.29 kubernetes-management#5616
• Terraform PR to upgrade cluster - feat(publick8s) bump to AKS 1.29.7 azure#810
• Merge Terraform PR and let the upgrade happen
• Check the LDAP restart successfully once upgrade is performed. If the pod restarted and is running, try the service by logging-in on ci.jenkins.io and repo.jenkins-ci.org
• Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1336
• Enable kubernetes management and revert exclusion of publick8s cluster, checking both diff and apply run successfully - Revert "chore: exclude publick8s cluster while upgrading to 1.29" kubernetes-management#5617
• Check the pagerduty alerts are all gone
• Close announcements

[dduportal]

Next upgrade: #4258

[dduportal]