feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents #715

smerle33 · 2024-06-04T14:20:56Z

as per jenkins-infra/helpdesk#3923 (comment)
kubernetes cluster within the sponsored subscription of azure

split in 3 PR:
- creation of the cluster (this one)
- creation of the nodes
- creation of kubernetes-admin-sa with the module

depends on jenkins-infra/azure-net#249 for the network definition

locals.tf

lemeurherve · 2024-06-10T12:51:44Z

locals.tf

+  publick8s_compute_zones                 = [3]
+  cijenkinsio_agents_1_compute_zones      = [1]
+  infracijenkinsio_agents_1_compute_zones = [1]
+
+  infraci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255


Suggested change

publick8s_compute_zones = [3]

cijenkinsio_agents_1_compute_zones = [1]

infracijenkinsio_agents_1_compute_zones = [1]

infraci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255

infraci_jenkins_io_agents_1_compute_zones = [1]

infraci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255

publick8s_compute_zones = [3]

Nit: ordering and regrouping values, with cijenkinsio_agents_1_compute_zones moved above, and keep the same variable name format.

lemeurherve · 2024-06-10T12:52:24Z

locals.tf

  }
  ci_jenkins_io_fqdn              = "ci.jenkins.io"
-  ci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14"
+  ci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255


Suggested change

ci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255

ci_jenkins_io_agents_1_compute_zones = [1]

ci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255

nit: moved from below and keep the same variable name format.

lemeurherve · 2024-06-10T13:01:26Z

infraci.jenkins.io-kubernetes-sponsored-agents.tf

+    node_count                   = 3 # 3 nodes for HA as per AKS best practises
+    vnet_subnet_id               = data.azurerm_subnet.infraci_jenkins_io_kubernetes_agent_sponsorship.id
+    tags                         = local.default_tags
+    zones                        = local.cijenkinsio_agents_1_compute_zones


Suggested change

zones = local.cijenkinsio_agents_1_compute_zones

zones = local.ci_jenkins_io_agents_1_compute_zones

(nit, keep same variable name format)

The idea is to have the cluster Terraform's resource ID the same as the local

Yes, I also suggested the same change in locals so all/most resources have a common naming strategy. (Might need more work, too much?)

We misunderstood each other: i am not in favor of the proposal. I prefer the local values to be the same as the terraform resource id, which is « cijenkinsio_agents_1_… ».

if you want to change the convention no problem but it us more than a nit as it involves migrating resources (using a moved block) and derails the PR here

Yes, I also suggested the same change in locals so all/most resources have a common naming strategy. (Might need more work, too much?)

With the last changes, I think I see the "naming" missing homogeneity. I would want to make this another PR though otherwise the review here will be too hard (side tracking from the main goal): could you open a first draft PR so we can have the discussion properly?

infraci.jenkins.io-kubernetes-sponsored-agents.tf

dduportal

Plan: 2 to add, 0 to change, 0 to destroy. => a new RG and the new cluster.

LGTM @smerle33 you can proceed!

dduportal · 2024-06-10T16:21:42Z

This PR failed to deploy with the following error:

17:43:25  │ Error: creating Kubernetes Cluster (Subscription: "1311c09f-aee0-4d6c-99a4-392c2b543204"
17:43:25  │ Resource Group Name: "infra-ci-jenkins-io-kubernetes-agents"
17:43:25  │ Kubernetes Cluster Name: "infracijenkinsio-agents-1"): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with response: {
17:43:25  │   "code": "SubnetNotAssociatedWithNATGateway",
17:43:25  │   "details": null,
17:43:25  │   "message": "Subnet '/subscriptions/1311c09f-aee0-4d6c-99a4-392c2b543204/resourceGroups/infra-ci-jenkins-io-sponsorship/providers/Microsoft.Network/virtualNetworks/infra-ci-jenkins-io-sponsorship-vnet/subnets/infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent' must have a NAT gateway associated for outbound connection.",
17:43:25  │   "subcode": ""
17:43:25  │  }

#719 created by @smerle33 to revert if we cannot fix the error easily

dduportal · 2024-06-10T16:23:21Z

This PR failed to deploy with the following error:

17:43:25  │ Error: creating Kubernetes Cluster (Subscription: "1311c09f-aee0-4d6c-99a4-392c2b543204"
17:43:25  │ Resource Group Name: "infra-ci-jenkins-io-kubernetes-agents"
17:43:25  │ Kubernetes Cluster Name: "infracijenkinsio-agents-1"): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with response: {
17:43:25  │   "code": "SubnetNotAssociatedWithNATGateway",
17:43:25  │   "details": null,
17:43:25  │   "message": "Subnet '/subscriptions/1311c09f-aee0-4d6c-99a4-392c2b543204/resourceGroups/infra-ci-jenkins-io-sponsorship/providers/Microsoft.Network/virtualNetworks/infra-ci-jenkins-io-sponsorship-vnet/subnets/infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent' must have a NAT gateway associated for outbound connection.",
17:43:25  │   "subcode": ""
17:43:25  │  }

#719 created by @smerle33 to revert if we cannot fix the error easily

Error fixed by jenkins-infra/azure-net#250. Let's retry

dduportal · 2024-06-10T16:23:35Z

New error:

"message": "The name infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent used in subnet ID (/subscriptions/1311c09f-aee0-4d6c-99a4-392c2b543204/resourceGroups/infra-ci-jenkins-io-sponsorship/providers/Microsoft.Network/virtualNetworks/infra-ci-jenkins-io-sponsorship-vnet/subnets/infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent) in agent pool systempool1 is too long for AKS to use. This name will be used as a label value on the Kubernetes Node(s) for this agent pool. There is a limit of 63 characters for Node labels. Please shorten the name so the length is less than or equal to 63. If the agent pool has already been created then it must be deleted first before making any more changes to this cluster.",
18:18:49 
 │   "subcode": "NetworkNameTooLong"

dduportal · 2024-06-10T16:44:57Z

New error:

"message": "The name infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent used in subnet ID (/subscriptions/1311c09f-aee0-4d6c-99a4-392c2b543204/resourceGroups/infra-ci-jenkins-io-sponsorship/providers/Microsoft.Network/virtualNetworks/infra-ci-jenkins-io-sponsorship-vnet/subnets/infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent) in agent pool systempool1 is too long for AKS to use. This name will be used as a label value on the Kubernetes Node(s) for this agent pool. There is a limit of 63 characters for Node labels. Please shorten the name so the length is less than or equal to 63. If the agent pool has already been created then it must be deleted first before making any more changes to this cluster.",
18:18:49 
 │   "subcode": "NetworkNameTooLong"

Should be fixed by:

=> new subnet name is 53 chars

… (shorter) name (#720) This PR aims to fix the error mentioned in #715 (comment) It updates the subnet name used for this cluster as a follow up of jenkins-infra/azure-net#252 Signed-off-by: Damien Duportal <[email protected]>

dduportal · 2024-06-10T17:04:55Z

New error:

"message": "The name infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent used in subnet ID (/subscriptions/1311c09f-aee0-4d6c-99a4-392c2b543204/resourceGroups/infra-ci-jenkins-io-sponsorship/providers/Microsoft.Network/virtualNetworks/infra-ci-jenkins-io-sponsorship-vnet/subnets/infra-ci-jenkins-io-sponsorship-vnet-infraci_jenkins_io_kubernetes-agent) in agent pool systempool1 is too long for AKS to use. This name will be used as a label value on the Kubernetes Node(s) for this agent pool. There is a limit of 63 characters for Node labels. Please shorten the name so the length is less than or equal to 63. If the agent pool has already been created then it must be deleted first before making any more changes to this cluster.",
18:18:49 
 │   "subcode": "NetworkNameTooLong"

Should be fixed by:

* [Revert "feat(infraci/agents-sponsored/cluster): add new subnet to infra ci outbound ips in sponsored" azure-net#251](https://github.com/jenkins-infra/azure-net/pull/251)

* [hotfix(vnets) shorten name for the infra.ci sponsored subnet used for kubernetes agents azure-net#252](https://github.com/jenkins-infra/azure-net/pull/252)

* [Reapply "feat(infraci/agents-sponsored/cluster): add new subnet to infra ci ou…" (#251) azure-net#253](https://github.com/jenkins-infra/azure-net/pull/253)

=> new subnet name is 53 chars

Next step: #720

as per jenkins-infra/helpdesk#3923 and following #715 this PR create 3 nodes pools: - application one in arm64 - agents in x86-64 - agents in arm64 --------- Co-authored-by: Damien Duportal <[email protected]>

as per #715 need to change the os

smerle33 added 6 commits June 10, 2024 14:10

WIP

c1263bc

wip

809a7c9

use the same ip class for pods as ci

cb26d42

cleanup

7a24b41

wip

0b144a9

feat(infra.ci/agent): new cluster in azure sponsored

3160315

smerle33 force-pushed the new/aks/sponsored/infra branch from abd7dbf to 3160315 Compare June 10, 2024 12:14

smerle33 marked this pull request as ready for review June 10, 2024 12:16

smerle33 requested a review from a team June 10, 2024 12:16

smerle33 changed the title ~~WIP~~ feat(infra.ci/agent): new cluster in azure sponsored Jun 10, 2024

smerle33 changed the title ~~feat(infra.ci/agent): new cluster in azure sponsored~~ feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents Jun 10, 2024

smerle33 requested review from dduportal and lemeurherve June 10, 2024 12:38

lemeurherve previously approved these changes Jun 10, 2024

View reviewed changes

lemeurherve reviewed Jun 10, 2024

View reviewed changes

dduportal reviewed Jun 10, 2024

View reviewed changes

infraci.jenkins.io-kubernetes-sponsored-agents.tf Outdated Show resolved Hide resolved

fix compute zone for infra and sort

1e8f372

smerle33 dismissed lemeurherve’s stale review via 1e8f372 June 10, 2024 15:05

sorting

ee7facc

smerle33 requested a review from lemeurherve June 10, 2024 15:09

dduportal approved these changes Jun 10, 2024

View reviewed changes

smerle33 merged commit 1c219f7 into jenkins-infra:main Jun 10, 2024
3 checks passed

This was referenced Jun 10, 2024

Revert "feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents" #719

Closed

feat(infraci/agents-sponsored/cluster): add new subnet to infra ci outbound ips in sponsored jenkins-infra/azure-net#250

Merged

dduportal mentioned this pull request Jun 10, 2024

hotfix(infraci.jenkins.io-kubernetes-sponsored-agents) use new subnet (shorter) name #720

Merged

smerle33 mentioned this pull request Jun 11, 2024

feat(infra/cluster/sponsored): add node pools #721

Merged

smerle33 mentioned this pull request Jun 12, 2024

feat(infracijenkinsio-agents-1): change os to azurelinux #723

Merged

smerle33 added a commit that referenced this pull request Jun 12, 2024

feat(infracijenkinsio-agents-1): change os to azurelinux (#723)

afba48e

as per #715 need to change the os

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents #715

feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents #715

smerle33 commented Jun 4, 2024 •

edited

Loading

lemeurherve Jun 10, 2024 •

edited

Loading

lemeurherve Jun 10, 2024 •

edited

Loading

lemeurherve Jun 10, 2024

dduportal Jun 10, 2024

lemeurherve Jun 10, 2024

dduportal Jun 10, 2024

dduportal Jun 10, 2024

lemeurherve Jun 10, 2024

dduportal left a comment

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

	ci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255
	ci_jenkins_io_agents_1_compute_zones = [1]
	ci_jenkins_io_agents_1_pod_cidr = "10.100.0.0/14" # 10.100.0.1 - 10.103.255.255

	zones = local.cijenkinsio_agents_1_compute_zones
	zones = local.ci_jenkins_io_agents_1_compute_zones

feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents #715

feat(infra.ci/agent): new cluster in azure sponsored for infra.ci agents #715

Conversation

smerle33 commented Jun 4, 2024 • edited Loading

lemeurherve Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

lemeurherve Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

lemeurherve Jun 10, 2024

Choose a reason for hiding this comment

dduportal Jun 10, 2024

Choose a reason for hiding this comment

lemeurherve Jun 10, 2024

Choose a reason for hiding this comment

dduportal Jun 10, 2024

Choose a reason for hiding this comment

dduportal Jun 10, 2024

Choose a reason for hiding this comment

lemeurherve Jun 10, 2024

Choose a reason for hiding this comment

dduportal left a comment

Choose a reason for hiding this comment

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

dduportal commented Jun 10, 2024

smerle33 commented Jun 4, 2024 •

edited

Loading

lemeurherve Jun 10, 2024 •

edited

Loading

lemeurherve Jun 10, 2024 •

edited

Loading