Merge pull request #11 from truefoundry/internal-initial-node-pool

moved initial node pool to internally
truefoundry · Jul 10, 2024 · 62481ab · 62481ab
2 parents bbe51fd + f727fd9
commit 62481ab
Show file tree

Hide file tree

Showing 5 changed files with 197 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -17,7 +17,7 @@ No providers.
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_aws-eks-kubernetes-cluster"></a> [aws-eks-kubernetes-cluster](#module\_aws-eks-kubernetes-cluster) | terraform-aws-modules/eks/aws | v20.15.0 |
+| <a name="module_aws-eks-kubernetes-cluster"></a> [aws-eks-kubernetes-cluster](#module\_aws-eks-kubernetes-cluster) | terraform-aws-modules/eks/aws | v20.17.2 |
 | <a name="module_eks_blueprints_addons"></a> [eks\_blueprints\_addons](#module\_eks\_blueprints\_addons) | aws-ia/eks-blueprints-addons/aws | 1.16.3 |
 
 ## Resources
@@ -28,6 +28,7 @@ No resources.
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_additional_eks_managed_node_groups"></a> [additional\_eks\_managed\_node\_groups](#input\_additional\_eks\_managed\_node\_groups) | Map of additional EKS managed node group definitions to create | `any` | `{}` | no |
 | <a name="input_cloudwatch_log_group_retention_in_days"></a> [cloudwatch\_log\_group\_retention\_in\_days](#input\_cloudwatch\_log\_group\_retention\_in\_days) | Number of days to retain log events. Default retention - 90 days | `number` | `90` | no |
 | <a name="input_cluster_access_entries"></a> [cluster\_access\_entries](#input\_cluster\_access\_entries) | value of the access entries for the EKS cluster | `any` | `{}` | no |
 | <a name="input_cluster_additional_security_group_ids"></a> [cluster\_additional\_security\_group\_ids](#input\_cluster\_additional\_security\_group\_ids) | List of additional, externally created security group IDs to attach to the cluster control plane | `list(string)` | `[]` | no |
@@ -48,6 +49,23 @@ No resources.
 | <a name="input_eks_managed_node_groups"></a> [eks\_managed\_node\_groups](#input\_eks\_managed\_node\_groups) | Map of EKS managed node group definitions to create | `any` | `{}` | no |
 | <a name="input_enable_irsa"></a> [enable\_irsa](#input\_enable\_irsa) | Determines whether to create an OpenID Connect Provider for EKS to enable IRSA | `bool` | `true` | no |
 | <a name="input_iam_role_additional_policies"></a> [iam\_role\_additional\_policies](#input\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `map(string)` | `{}` | no |
+| <a name="input_inital_node_pool_capacity_type"></a> [inital\_node\_pool\_capacity\_type](#input\_inital\_node\_pool\_capacity\_type) | capacity type for the initial node pool | `string` | `"SPOT"` | no |
+| <a name="input_initial_node_pool_ami_type"></a> [initial\_node\_pool\_ami\_type](#input\_initial\_node\_pool\_ami\_type) | AMI type for the initial node pool | `string` | `"AL2023_x86_64_STANDARD"` | no |
+| <a name="input_initial_node_pool_create_iam_role"></a> [initial\_node\_pool\_create\_iam\_role](#input\_initial\_node\_pool\_create\_iam\_role) | Create IAM role for the initial node pool | `bool` | `true` | no |
+| <a name="input_initial_node_pool_create_iam_role_policy"></a> [initial\_node\_pool\_create\_iam\_role\_policy](#input\_initial\_node\_pool\_create\_iam\_role\_policy) | Create IAM role policy for the initial node pool | `bool` | `true` | no |
+| <a name="input_initial_node_pool_create_node_template"></a> [initial\_node\_pool\_create\_node\_template](#input\_initial\_node\_pool\_create\_node\_template) | Create node template for the initial node pool | `bool` | `true` | no |
+| <a name="input_initial_node_pool_desired_size"></a> [initial\_node\_pool\_desired\_size](#input\_initial\_node\_pool\_desired\_size) | Desired size for the initial node pool | `number` | `2` | no |
+| <a name="input_initial_node_pool_enabled"></a> [initial\_node\_pool\_enabled](#input\_initial\_node\_pool\_enabled) | Create al2023 initial node pool for EKS managed node group | `bool` | `true` | no |
+| <a name="input_initial_node_pool_iam_role_additional_policies"></a> [initial\_node\_pool\_iam\_role\_additional\_policies](#input\_initial\_node\_pool\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role for the initial node pool | `map(string)` | <pre>{<br>  "karpenter": "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"<br>}</pre> | no |
+| <a name="input_initial_node_pool_iam_role_attach_cni_policy"></a> [initial\_node\_pool\_iam\_role\_attach\_cni\_policy](#input\_initial\_node\_pool\_iam\_role\_attach\_cni\_policy) | Attach CNI policy to IAM role for the initial node pool | `bool` | `true` | no |
+| <a name="input_initial_node_pool_iam_role_tags"></a> [initial\_node\_pool\_iam\_role\_tags](#input\_initial\_node\_pool\_iam\_role\_tags) | IAM role tags for the initial node pool | `map(string)` | `{}` | no |
+| <a name="input_initial_node_pool_iam_role_use_name_prefix"></a> [initial\_node\_pool\_iam\_role\_use\_name\_prefix](#input\_initial\_node\_pool\_iam\_role\_use\_name\_prefix) | Use name prefix for IAM role for the initial node pool | `bool` | `true` | no |
+| <a name="input_initial_node_pool_instance_types"></a> [initial\_node\_pool\_instance\_types](#input\_initial\_node\_pool\_instance\_types) | Instance types for the initial node pool | `list(string)` | <pre>[<br>  "c7i.large",<br>  "c7a.large",<br>  "m7i.large",<br>  "m7a.large",<br>  "r7i.large",<br>  "r7a.large",<br>  "r6i.large",<br>  "r6a.large",<br>  "c6i.large",<br>  "c6a.large",<br>  "m6a.large"<br>]</pre> | no |
+| <a name="input_initial_node_pool_labels"></a> [initial\_node\_pool\_labels](#input\_initial\_node\_pool\_labels) | Labels for the initial node pool | `map(string)` | <pre>{<br>  "truefoundry.cloud": "initial"<br>}</pre> | no |
+| <a name="input_initial_node_pool_launch_template_use_name_prefix"></a> [initial\_node\_pool\_launch\_template\_use\_name\_prefix](#input\_initial\_node\_pool\_launch\_template\_use\_name\_prefix) | Use name prefix for launch template for the initial node pool | `bool` | `true` | no |
+| <a name="input_initial_node_pool_max_size"></a> [initial\_node\_pool\_max\_size](#input\_initial\_node\_pool\_max\_size) | Maximum size for the initial node pool | `number` | `2` | no |
+| <a name="input_initial_node_pool_metadata_options"></a> [initial\_node\_pool\_metadata\_options](#input\_initial\_node\_pool\_metadata\_options) | Metadata options for the initial node pool | `map(string)` | `{}` | no |
+| <a name="input_initial_node_pool_min_size"></a> [initial\_node\_pool\_min\_size](#input\_initial\_node\_pool\_min\_size) | Minimum size for the initial node pool | `number` | `2` | no |
 | <a name="input_karpenter_fargate_profile_attach_cni_policy"></a> [karpenter\_fargate\_profile\_attach\_cni\_policy](#input\_karpenter\_fargate\_profile\_attach\_cni\_policy) | Attach CNI policy to IAM role for Karpenter Fargate profile | `bool` | `true` | no |
 | <a name="input_karpenter_fargate_profile_create_iam_role"></a> [karpenter\_fargate\_profile\_create\_iam\_role](#input\_karpenter\_fargate\_profile\_create\_iam\_role) | Create IAM role for Karpenter Fargate profile | `bool` | `true` | no |
 | <a name="input_karpenter_fargate_profile_enabled"></a> [karpenter\_fargate\_profile\_enabled](#input\_karpenter\_fargate\_profile\_enabled) | Enable Karpenter Fargate profile | `bool` | `true` | no |

diff --git a/eks.tf b/eks.tf
@@ -4,7 +4,7 @@
 
 module "aws-eks-kubernetes-cluster" {
   source                                = "terraform-aws-modules/eks/aws"
-  version                               = "v20.15.0"
+  version                               = "v20.17.2"
   cluster_name                          = var.cluster_name
   cluster_version                       = var.cluster_version
   cluster_enabled_log_types             = var.cluster_enabled_log_types
@@ -23,7 +23,7 @@ module "aws-eks-kubernetes-cluster" {
   self_managed_node_group_defaults       = var.self_managed_node_group_defaults
   self_managed_node_groups               = var.self_managed_node_groups
   eks_managed_node_group_defaults        = var.eks_managed_node_group_defaults
-  eks_managed_node_groups                = var.eks_managed_node_groups
+  eks_managed_node_groups                = local.node_groups
 
   cluster_security_group_additional_rules  = merge(local.cluster_security_group_additional_rules, var.cluster_security_group_additional_rules)
   node_security_group_additional_rules     = merge(local.node_security_group_additional_rules, var.node_security_group_additional_rules)
@@ -36,8 +36,7 @@ module "aws-eks-kubernetes-cluster" {
       create       = true
       cluster_name = var.cluster_name
       name         = local.karpenter_profile_name
-
-      subnet_ids = var.subnet_ids
+      subnet_ids   = var.subnet_ids
       selectors = [
         {
           namespace = var.karpenter_fargate_profile_namespace

diff --git a/locals.tf b/locals.tf
@@ -8,6 +8,12 @@ locals {
     var.tags
   )
 
+  karpenter_tags = merge(
+    {
+      "karpenter.sh/discovery" = var.cluster_name
+    },
+    var.tags
+  )
   cluster_security_group_additional_rules = {
     egress_nodes_ephemeral_ports_tcp = {
       description                = "To node 1025-65535"
@@ -58,5 +64,44 @@ locals {
     }
   }
 
+  node_groups = merge(var.additional_eks_managed_node_groups,
+    var.initial_node_pool_enabled ? {
+      initial = {
+        ami_type = "AL2023_x86_64_STANDARD"
+        block_device_mappings = {
+          xvda = {
+            device_name = "/dev/xvda"
+            ebs = {
+              volume_size           = 100
+              volume_type           = "gp3"
+              delete_on_termination = true
+            }
+          }
+        }
+        capacity_type                   = var.inital_node_pool_capacity_type
+        create                          = true
+        create_iam_role                 = var.initial_node_pool_create_iam_role
+        create_iam_role_policy          = var.initial_node_pool_create_iam_role_policy
+        create_launch_template          = var.initial_node_pool_create_node_template
+        min_size                        = var.initial_node_pool_min_size
+        max_size                        = var.initial_node_pool_max_size
+        desired_size                    = var.initial_node_pool_desired_size
+        iam_role_attach_cni_policy      = var.initial_node_pool_iam_role_attach_cni_policy
+        iam_role_description            = "TrueFoundry EKS initial node group role for ${var.cluster_name}"
+        iam_role_tags                   = merge(local.tags, var.initial_node_pool_iam_role_tags)
+        iam_role_use_name_prefix        = var.initial_node_pool_iam_role_use_name_prefix
+        iam_role_name                   = "${var.cluster_name}-initial"
+        instance_types                  = var.initial_node_pool_instance_types
+        launch_template_description     = "TrueFoundry AL2023 EKS initial node group launch template for ${var.cluster_name}"
+        launch_template_name            = "${var.cluster_name}-initial"
+        launch_template_use_name_prefix = var.initial_node_pool_launch_template_use_name_prefix
+        labels                          = var.initial_node_pool_labels
+        iam_role_additional_policies    = var.initial_node_pool_iam_role_additional_policies
+        name                            = "${var.cluster_name}-initial"
+        metadata_options                = var.initial_node_pool_metadata_options
+        tags                            = local.karpenter_tags
+      }
+  } : {})
+
   karpenter_profile_name = "${var.cluster_name}-karpenter"
 }
diff --git a/upgrade-guide.md b/upgrade-guide.md
@@ -0,0 +1,15 @@
+# terraform-aws-truefoundry-cluster
+This guide will help you to migrate your terraform code across versions. Keeping your terraform state to the latest version is always recommeneded
+
+## Upgrade guide from 0.5.x to 0.6.x
+
+### Pre-requisites
+1. Ensure you are running on the version [0.5.2](https://github.com/truefoundry/terraform-aws-truefoundry-cluster/releases/tag/v0.5.2)
+2. Ensure that you runnning on the EKS version `1.29` which is the default version in `0.5.2`
+
+## Upgrade changes (manual)
+1. Execute the terraform apply with version `0.6.0`. If it fails run the below command to import access entry for cluster creator
+```
+terragrunt import 'module.aws-eks-kubernetes-cluster.aws_eks_access_entry.this["cluster_creator"]' "$IAM_PRINCIPAL_ARN"
+terragrunt import 'module.aws-eks-kubernetes-cluster.aws_eks_access_policy_association.this["cluster_creator_admin"]' $CLUSTER_NAME#$IAM_PRINCIPAL_ARN#arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy
+```
diff --git a/variables.tf b/variables.tf
@@ -162,12 +162,127 @@ variable "eks_managed_node_group_defaults" {
   default     = {}
 }
 
+variable "additional_eks_managed_node_groups" {
+  description = "Map of additional EKS managed node group definitions to create"
+  type        = any
+  default     = {}
+}
+
 variable "eks_managed_node_groups" {
   description = "Map of EKS managed node group definitions to create"
   type        = any
   default     = {}
 }
 
+################################################################################
+# EKS Managed Initial Node Group
+################################################################################
+variable "initial_node_pool_enabled" {
+  description = "Create al2023 initial node pool for EKS managed node group"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_ami_type" {
+  description = "AMI type for the initial node pool"
+  type        = string
+  default     = "AL2023_x86_64_STANDARD"
+}
+
+variable "inital_node_pool_capacity_type" {
+  description = "capacity type for the initial node pool"
+  type        = string
+  default     = "SPOT"
+}
+
+variable "initial_node_pool_create_iam_role" {
+  description = "Create IAM role for the initial node pool"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_create_iam_role_policy" {
+  description = "Create IAM role policy for the initial node pool"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_create_node_template" {
+  description = "Create node template for the initial node pool"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_min_size" {
+  description = "Minimum size for the initial node pool"
+  type        = number
+  default     = 2
+}
+
+variable "initial_node_pool_max_size" {
+  description = "Maximum size for the initial node pool"
+  type        = number
+  default     = 2
+}
+
+variable "initial_node_pool_desired_size" {
+  description = "Desired size for the initial node pool"
+  type        = number
+  default     = 2
+}
+
+variable "initial_node_pool_iam_role_attach_cni_policy" {
+  description = "Attach CNI policy to IAM role for the initial node pool"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_iam_role_tags" {
+  description = "IAM role tags for the initial node pool"
+  type        = map(string)
+  default     = {}
+}
+
+variable "initial_node_pool_iam_role_additional_policies" {
+  description = "Additional policies to be added to the IAM role for the initial node pool"
+  type        = map(string)
+  default = {
+    # Required by Karpenter
+    karpenter = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
+  }
+}
+
+variable "initial_node_pool_iam_role_use_name_prefix" {
+  description = "Use name prefix for IAM role for the initial node pool"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_instance_types" {
+  description = "Instance types for the initial node pool"
+  type        = list(string)
+  default     = ["c7i.large", "c7a.large", "m7i.large", "m7a.large", "r7i.large", "r7a.large", "r6i.large", "r6a.large", "c6i.large", "c6a.large", "m6a.large"]
+}
+
+variable "initial_node_pool_launch_template_use_name_prefix" {
+  description = "Use name prefix for launch template for the initial node pool"
+  type        = bool
+  default     = true
+}
+
+variable "initial_node_pool_metadata_options" {
+  description = "Metadata options for the initial node pool"
+  type        = map(string)
+  default     = {}
+}
+
+variable "initial_node_pool_labels" {
+  description = "Labels for the initial node pool"
+  type        = map(string)
+  default = {
+    "truefoundry.cloud" = "initial"
+  }
+}
 ##################################################################################
 ## Other variables
 ##################################################################################