Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: 25 refresh instances #39

Closed
wants to merge 3 commits into from
Closed

Conversation

cecilia-donnelly
Copy link
Member

Description

This PR adds an autoscaling group and refresh policy to the Terraform dev setup. (Only dev, since this is experimental.) We have two autoscaling groups in AWS, but neither of them are for dev. I assumed that we only wanted to autoscale the "backend" machine, not cron or task-runner, but that might be wrong.

I'm opening this as a place to discuss with @andrewatwood.

Resolves #25.

Related to #23 and #21. I suspect that fixing #21 would be a good thing to include in this PR, too.

Steps to Test

I am not sure how to test this short of redeploying dev, unfortunately. Ideas?

@xmunoz mentioned in #25 that Terraform didn't yet support
automatically refreshing instances with new code.  This is an issue
because, assuming we have multiple autoscaled instances up at a moment
when we want to do a code deploy, we need a way to refresh all of
those instances with the new code (other than clicking "Refresh" in
AWS).  Terraform now does support this, as of version 3.22.

Their issue: hashicorp/terraform-provider-aws#13785
The pull request: hashicorp/terraform-provider-aws#16678
The changelog: https://github.com/hashicorp/terraform-provider-aws/blob/v3.22.0/CHANGELOG.md

Use this new version of the Terraform AWS provider so that we can use
autoscaling.
Followed the documentation here:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group

This is quite experimental (which is why I only changed the dev
setup).  I tried to use the minimal set of parameters because most of
them are optional, but I'm not entirely sure I've given terraform
enough information to find the right autoscaling group in AWS.
Copy link
Contributor

@andrewatwood andrewatwood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assumptions about which boxes we want to scale are correct, for now. backend is the immediate issue, as taskrunner can get backed up but won't fail under load as its tasks are managed by the SQS queue. And cron we'll correctly never scale.

Looks good!

@@ -30,16 +30,31 @@ variable "perm_env" {
}
}

resource "aws_autoscaling_group" "dev_api" {
name = "backend-dev"
max_size = 5 # does this seem appropriate?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously this was set to a max of 3, but that was arbitrary! 5 seems like a a good end target, assuming we dial in appropriate scaling triggers.

@@ -30,16 +30,31 @@ variable "perm_env" {
}
}

resource "aws_autoscaling_group" "dev_api" {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question we have: do we need to create this autoscaling group or will Terraform build it?

@cecilia-donnelly
Copy link
Member Author

Closing this since it's not ready for prime time (have not made corresponding AWS changes). Will reopen as needed when autoscaling becomes a priority again.

@jasonaowen jasonaowen deleted the 25-refresh-instances branch February 9, 2022 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Devise automation for instance refresh
2 participants