Skip to content

Commit

Permalink
Merge pull request #143 from ExpediaGroup/feature/add_wd_memory_limit
Browse files Browse the repository at this point in the history 
feat: Added option to set guaranteed resource on waggledance pod
  • Loading branch information
@githubjianli
githubjianli authored Apr 18, 2024
2 parents 3d67b00 + 5f60ef2 commit e6d8530
Show file tree
Hide file tree
Showing 4 changed files with 30 additions and 2 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [4.4.0] - 2024-04-18
### Added
- Add new variable `cpu_limit` and `memory_limit` to support set pod resource guaranteed.

## [4.3.2] - 2024-03-26
### Updated
- Removed the datadog explicit provider configuration as suggested here: https://developer.hashicorp.com/terraform/language/modules/develop/providers#legacy-shared-modules-with-provider-configurations
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
| aws_region | AWS region to use for resources. | string | - | yes |
| bastion_ssh_key_secret_name | Secret name in AWS Secrets Manager which stores the private key used to log in to bastions. The secret's key should be `private_key` and the value should be stored as a base64 encoded string. Max character limit for a secret's value is 4096. | string | `` | no |
| cpu | The number of CPU units to reserve for the Waggle Dance container. Valid values can be 256, 512, 1024, 2048 and 4096. Reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html | string | `1024` | no |
| cpu_limit | The number of CPU limit units to reserve for the Waggle Dance container. Valid values can be 256, 512, 1024, 2048 and 4096. It will use `cpu` * 1.25 if not specified. Reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html | string | `1024` | no |
| cpu_scale_in_cooldown | Cool down time(seconds) of scale in task by cpu usage | number | 300 | no |
| cpu_scale_out_cooldown | Cool down time(seconds) of scale out task by cpu usage | number | 120 | no |
| default_latency | Latency used for other (not primary) metastores that don't override it in their own configurations. See `latency` parameter in https://github.com/ExpediaGroup/waggle-dance/blob/main/README.md. | number | `0` | no |
Expand All @@ -32,6 +33,7 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
| k8s_max_replica_count | Max Number of k8s pod replicas to create. | number | `10` | no |
| local_metastores | List of federated Metastore endpoints directly accessible on the local network. See section [`local_metastores`](#local_metastores) for more info. | list | `<list>` | no |
| memory | The amount of memory (in MiB) used to allocate for the Waggle Dance container. Valid values: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html | string | `4096` | no |
| memory_limit | The amount of memory limit (in MiB) used to allocate for the Waggle Dance container, it will use `memory` * 1.25 if the limit is not specified. Valid values: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html | string | `null` | no |
| primary_metastore_access_type | Primary Hive Metastore access control type. | string | `READ_AND_WRITE_ON_DATABASE_WHITELIST` | no |
| primary_metastore_host | Primary Hive Metastore hostname configured in Waggle Dance. | string | `localhost` | no |
| primary_metastore_port | Primary Hive Metastore port | string | `9083` | no |
Expand Down
4 changes: 2 additions & 2 deletions k8s.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@

locals {
heapsize = ceil((var.memory * 85) / 100)
memory_limit = ceil((var.memory * 120) / 100)
memory_limit = length(var.memory_limit) != 0 ? var.memory_limit : ceil((var.memory * 120) / 100)
actuator_port = 18000
wd_port = 48869
k8s_cpu = var.cpu / 1024
k8s_cpu_limit = (var.cpu / 1024) * 1.25
k8s_cpu_limit = length(var.cpu_limit) != 0 ? var.cpu_limit / 1024 : (var.cpu / 1024) * 1.25
}

resource "kubernetes_service_account" "waggle_dance" {
Expand Down
22 changes: 22 additions & 0 deletions variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,17 @@ EOF
default = "4096"
}

variable "memory_limit" {
description = <<EOF
The amount of memory limit (in MiB) used to allocate for the Waggle Dance container. It will use memory * 1.25
if this value is not specified.
Valid values: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html
EOF

type = string
default = ""
}

variable "cpu" {
description = <<EOF
The number of CPU units to reserve for the Waggle Dance container.
Expand All @@ -148,6 +159,17 @@ EOF
default = "1024"
}

variable "cpu_limit" {
description = <<EOF
The number of CPU units limit to reserve for the Waggle Dance container.
Valid values can be 256, 512, 1024, 2048 and 4096.It will use cpu * 1.25 if this value is not specified.
Reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html
EOF

type = string
default = ""
}

variable "ingress_cidr" {
description = "Generally allowed ingress CIDR list."
type = list(string)
Expand Down

0 comments on commit e6d8530

Please sign in to comment.