Document updating Fargate Service Task Definition using CODE_DEPLOY deployment controller

[dan-cooke]

What happened?

It is impossible to update an existing Fargate service using pulumi/aws or pulumi/awsx with a deploymentController configured as CODE_DEPLOY

If you attempt this, you will receive the following error message

Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment

I am unsure where this issue is coming from, I notice previously closed issues on the subject to do with upstream TF provider

#1961

Related issues:
pulumi/pulumi-awsx#856
aws-cloudformation/cloudformation-coverage-roadmap#1529
aws/aws-cdk#23564

Example

Given the following Fargate Service task definition

const service = new awsx.ecs.FargateService(
  'templi-api',
  {
    cluster: baseInfra.requireOutput('clusterArn'),
    deploymentController: {
      type: 'CODE_DEPLOY',
    },
    assignPublicIp: true,
    continueBeforeSteadyState: true,


    taskDefinition: latestTaskDefinition.arn,
    loadBalancers: [
      {
        containerPort: 80,
        containerName: 'templi-api',
        targetGroupArn: blueTargetGroup.arn,
      },
    ],
  },
  {
    ignoreChanges: ['taskDefinition'],
  }
);

With a task definition that changes , (ie. docker image update)

You will be unable to run pulumi up more than once.

Output of pulumi about

CLI
Version 3.107.0
Go Version go1.22.0
Go Compiler gc

Plugins
NAME VERSION
nodejs unknown

Host
OS arch
Version
Arch x86_64

This project is written in nodejs: executable='/run/user/1000/fnm_multishells/23785_1713179166746/bin/node' version='v20.9.0'

Current Stack: templi/templi-api/prod

TYPE URN
pulumi:pulumi:Stack urn:pulumi:prod::templi-api::pulumi:pulumi:Stack::templi-api-prod
pulumi:providers:pulumi urn:pulumi:prod::templi-api::pulumi:providers:pulumi::default
pulumi:providers:aws urn:pulumi:prod::templi-api::pulumi:providers:aws::default_6_27_0
aws:iam/role:Role urn:pulumi:prod::templi-api::aws:iam/role:Role::api-code-deploy-role
aws:acm/certificate:Certificate urn:pulumi:prod::templi-api::aws:acm/certificate:Certificate::api-cert
aws:codedeploy/application:Application urn:pulumi:prod::templi-api::aws:codedeploy/application:Application::templi-api
aws:rds/parameterGroup:ParameterGroup urn:pulumi:prod::templi-api::aws:rds/parameterGroup:ParameterGroup::templi-database-parameter-group
aws:route53/record:Record urn:pulumi:prod::templi-api::aws:route53/record:Record::api-cert-validation-record
aws:acm/certificateValidation:CertificateValidation urn:pulumi:prod::templi-api::aws:acm/certificateValidation:CertificateValidation::api-cert-validation
pulumi:pulumi:StackReference urn:pulumi:prod::templi-api::pulumi:pulumi:StackReference::templi/templi-infra/prod
pulumi:providers:awsx urn:pulumi:prod::templi-api::pulumi:providers:awsx::default_2_5_0
aws:ec2/securityGroup:SecurityGroup urn:pulumi:prod::templi-api::aws:ec2/securityGroup:SecurityGroup::templi-api-sg
aws:lb/targetGroup:TargetGroup urn:pulumi:prod::templi-api::aws:lb/targetGroup:TargetGroup::blue-target-group
aws:rds/subnetGroup:SubnetGroup urn:pulumi:prod::templi-api::aws:rds/subnetGroup:SubnetGroup::templi-database-subnet-group
aws:iam/role:Role urn:pulumi:prod::templi-api::aws:iam/role:Role::templi-api-task-role
aws:lb/targetGroup:TargetGroup urn:pulumi:prod::templi-api::aws:lb/targetGroup:TargetGroup::green-target-group
aws:ec2/securityGroup:SecurityGroup urn:pulumi:prod::templi-api::aws:ec2/securityGroup:SecurityGroup::templi-database-sg
aws:rds/instance:Instance urn:pulumi:prod::templi-api::aws:rds/instance:Instance::templi-database
awsx:ecr:Image urn:pulumi:prod::templi-api::awsx:ecr:Image::templi-api
pulumi:providers:aws urn:pulumi:prod::templi-api::pulumi:providers:aws::default_6_9_0
pulumi:providers:docker urn:pulumi:prod::templi-api::pulumi:providers:docker::default_4_5_0
docker:index/image:Image urn:pulumi:prod::templi-api::awsx:ecr:Image$docker:index/image:Image::6f49d865-container
awsx:lb:ApplicationLoadBalancer urn:pulumi:prod::templi-api::awsx:lb:ApplicationLoadBalancer::templi-lb
aws:lb/loadBalancer:LoadBalancer urn:pulumi:prod::templi-api::awsx:lb:ApplicationLoadBalancer$aws:lb/loadBalancer:LoadBalancer::templi-lb
aws:lb/targetGroup:TargetGroup urn:pulumi:prod::templi-api::awsx:lb:ApplicationLoadBalancer$aws:lb/targetGroup:TargetGroup::templi-lb
awsx:ecs:FargateService urn:pulumi:prod::templi-api::awsx:ecs:FargateService::templi-api
aws:ecs/service:Service urn:pulumi:prod::templi-api::awsx:ecs:FargateService$aws:ecs/service:Service::templi-api
aws:lb/listener:Listener urn:pulumi:prod::templi-api::aws:lb/listener:Listener::blue-listener
aws:route53/record:Record urn:pulumi:prod::templi-api::aws:route53/record:Record::api.templi.ai
aws:lb/listener:Listener urn:pulumi:prod::templi-api::aws:lb/listener:Listener::green-listener
awsx:ecs:FargateTaskDefinition urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition::templi-api
aws:iam/role:Role urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:iam/role:Role::templi-api-execution
aws:cloudwatch/logGroup:LogGroup urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:cloudwatch/logGroup:LogGroup::templi-api
aws:codedeploy/deploymentGroup:DeploymentGroup urn:pulumi:prod::templi-api::aws:codedeploy/deploymentGroup:DeploymentGroup::api-code-deploy
aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:iam/rolePolicyAttachment:RolePolicyAttachment::templi-api-execution-9a42f520
aws:ecs/taskDefinition:TaskDefinition urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:ecs/taskDefinition:TaskDefinition::templi-api
pulumi:providers:command urn:pulumi:prod::templi-api::pulumi:providers:command::default_0_9_2
command:local:Command urn:pulumi:prod::templi-api::command:local:Command::create-deployment

Found no pending operations associated with prod

Backend
Name pulumi.com
URL https://app.pulumi.com/dan-cooke
User dan-cooke
Organizations dan-cooke, templi
Token type personal

Pulumi locates its logs in /tmp by default

Additional context

If this falls into the wontfix, working as expected category - maybe we can document this somewhere? I've spent quite some time today trying to get blue/green deployments working with ECS + CodeDeploy via Pulumi, and the experience has been extremely painful to say the least.

The suggested workaround in the meantime seems to be to use ignoreChanges on your ECS service task definition, but unfortunately that is also not working atm. pulumi/pulumi-awsx#856

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[corymhall]

@dan-cooke I saw your comment on pulumi/pulumi-awsx#856 (comment). It sounds like there might be two issues:

1. awsx does not allow you to ignoreChanges (tracked through ignoreChanges seems to have some trouble ignoring changes pulumi-awsx#856)
2. Inadequate documentation on how to perform ECS blue/green deployments (we can track with this issue)

There also might be an opportunity for a new component for triggering code deploy deployments similar to the cdk-ecs-codedeploy CDK construct. Although an argument might also be made that this is a good scenario to use the automation api.

After the first deployment, all subsequent updates have to go through the Code Deploy service, but I agree that we could make that piece much easier.

[dan-cooke]

hey @corymhall thanks for your reply!

I agree there are a few interconnected issues here, and I like your suggestion to use this issue for documentation updates - I think a piece on blue/green deployments would go a long way.

It would be useful to see how we should handle this in Pulumi, for example if the solution is to ignore taskDefinitions, then how do we subsequently update other information about our taskDefinition? ie. memory cpu etc. surely this problem has been encountered in Terraform and CloudFormation too.

Its a tricky one, because it appears to be a fundamental issue with AWS ECS x CodeDeploy rather than an IaC specific problem.

[corymhall]

Its a tricky one, because it appears to be a fundamental issue with AWS ECS x CodeDeploy rather than an IaC specific problem.

100% agree with this. I'm curious what the killer CodeDeploy features are in your opinion? Part of me wants to find a solution to this that doesn't involve CodeDeploy and just uses ECS native features.

[dan-cooke]

Good question, and honestly its something I hadn't even thought about. Perhaps the golden path here is ECS native.

But off the top of my head, the features that I love from CodeDeploy

• Rollback on CloudWatch alert - I use this to rollback on e2e test failures
• Traffic re-routing built in, easy to start with 0% and only begin routing traffic when tests pass
• UI to manually route traffic / rollback - I use this quite a bit currently as a kind of final sanity check before I release

Some of that would be quite difficult (impossible for the UI) to achieve with ECS on its own.

But I am definetly willing to explore the option, the UI is optional - eventually I will be moving to fully automated releases anyway.

[fyelci]

I'm having the same problem. There isn't enough documentation on pulumi for using blue/green deployments with ECS.

What's the expected behaviour in here? After doing the initial deployment with pulumi, is it only possible to deploy by using CodeDeploy? And can't we make any changes on task definitions by using pulumi?