There are downtime when another sync is triggered before canary deployment is complete. (alb ingress environment)

[nessa829]

• We are using argo-rollout / canary strategy / ingress with ALB configuration in AWS EKS environment(1.27).
• When i trigger a new image sync before the rollout canary deployment is complete, downtime occurs. (This is not a typical deployment process, but sometimes it happens)
• As i analyzed and followed up the argo rollout code and the actual remaining rollout logs, it progresses as follows.:

1. As soon as synchronized, rollout detects new canary (2024-02-22T01:32:14Z UTC)
2. Change service selector to new canary from old canary (2024-02-22T01:32:14Z UTC)
3. Change canary hash value of rollout status to new canary (2024-02-22T01:32:14Z UTC)
4. Change ingress weight from 50:50 to 100:0 (because new canary is not ready) (2024-02-22T01:32:14Z UTC)
5. delete old canary (2024-02-22T01:32:14Z UTC)

• I was curious that 5xx errors occurred even though the ingress weight was changed to 100:0.
• so I analyzed the lb-controller log as well, and found that an update operation request was received at almost the same time.
• However, the actual ModifyRule action to change weight triggered only after about 1 minute and 10 seconds. (From, Cloudtrail logs. February 22, 2024, 1:33:23 UTC)
• After that, when new canary is ready, it changes to 50:50 again.

Therefore, there are 3 issue points.

1. In the first place, when deployment was already in progress, another sync was accepted.
2. As soon as sync occurs, the service selector / rollout status for canary hash/ the ingress weight switches to 100:0, even before graceful shutdown
3. Delay between lb-controller’s weight change update operation request and actual lb listner ModifyRule API.

3 is not something that can be discussed here.
So, i would like to know if there are any option that i can try to prevent another sync during the rollout canary deployment, or graceful shutdown of old canary version in the first place.
i have already tried prestop and terminationGracePeriodSeconds, but it did not stop downtime.
As soon as i trigger sync, the existing canary pods are terminated immediately, and service selector / canary hash of rollout status / ingress weight changed simultaneously.

I expected this recent fix (https://github.com/argoproj/argo-rollouts/pull/3354/files) is relevant, but it seems no help to my situation.

[zachaller]

With ALB, I have found that you should always use ping pong and also enable weight verification --aws-verify-target-group.

[nessa829]

@zachaller hi, thank you for the reply.
I am already using ping pong, but tried weight verification option you suggested.
It slightly delayed the old canary terminating, but there is still downtime. ;(
When i see the log, there are failed verifications periodically as followed :

time="2024-02-29T03:36:31Z" level=warning msg="Failed to verify weight: operation error Elastic Load Balancing v2: DescribeLoadBalancers, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 060f9c79-964e-4c34-8c0e-500cd25f4e53, api error Throttling: Rate exceeded" event_reason=WeightVerifyError namespace=service-sre-test rollout=sre-test

As I wrote above, rollout ordered weight to be 100:0 on time, and lb-controller also received the order on time.
However, API (ModifyRule) happens late.
I guess maybe it is AWS ALB's problem, due to heavy api. :(