(packer-images): Add Garbage Collector for AWS

[jayfranco999]

Due to the complexity of this PR – jenkins-infra/packer-images#1430

We are splitting the tasks into 4 stages, this issue is related to handling the Garbage collector scripts foe old AWS AMI Images.

We are reintroducing the amazon login that we were using with our previous AWS jenkins account – jenkins-infra/packer-images#734

• Cleanup AWS images – ensure staging images older than 7 days are removed
• Cleanup AWS images – collect EC2 snapshots older than 1 year on us-east-2
• Cleanup AWS images – add aws.sh as daily cleanup task

Originally posted by @jayfranco999 in #4316 (comment)

[dduportal]

Update:

• We need a distinct job for garbage collection of packer-images (ref. https://github.com/jenkins-infra/kubernetes-management/blob/main/config/jenkins-jobs_infra.ci.jenkins.io.yaml) with:
  • Custom Jenkinsfile
  • Its own set of credentials (reuse the same as the principal job for now)
  • no tag discovery
• Once successfull, update the principal Jenkinsfile_k8s pipeline to remove Garbage collection from it

[jayfranco999]

Update:

jenkins-infra/kubernetes-management#5892 and jenkins-infra/packer-images#1527 adds the aws credentials and aws-gc stage to the Jenkisnfile_gc pipeline.

Once merged, we can merge jenkins-infra/packer-images#1528 which makes the garbage-collectora distinct job and speed up themainbuild onpacker-images`

[jayfranco999]

Update:

Task list to follow:

This first version works well!

Next steps:

• On this pipeline (subsequent PR):
  • Add a cron job trigger (feat(Jenkinsfile_gc) adds cron trigger expression packer-images#1531)
  • Restrict (dryrun=false) to only the main branch - (feat(jenkinsfile_gc) disables dryrun when run on main branch packer-images#1532)
  • Add AWS garbage collection
• On the Jenkinsfile_k8s standard pipeline of this repository:
  • Remove the GC stage(s) (chore(Jenkinsfile_k8s) removes GC stage from pipeline packer-images#1528)
• On the GC job configuration level, we need to:
  • Disable the GitHub checks report from infra.ci to the repository
  • Stop discovering and building PRs (We decided to keep PRs with dryrun to test script changes)
  • Stop discovering and building tags
  • Only build on the main branch (not other branch) (Not needed anymore)

garbage collector now exists as a distinct job Jenkinsfile_gc and runs on a pod agent with dryrun on PRs (but not on tags) and doesn't send github checks anymore

[jayfranco999]

• • Add AWS garbage collection

On adding the aws garbage collection, we noticed that the aws.sh script exited with an error An error occurred (UnauthorizedOperation) when calling the DescribeNetworkInterfaces operation: You are not authorized to perform this operation. User: arn:aws:iam::326712726440:user/terraform-packer-user is not authorized to perform: ec2:DescribeNetworkInterfaces because no identity-based policy allows the ec2:DescribeNetworkInterfaces action

ec2:DescribeNetworkInterfaces permission needs to be added for the IAM user terraform-packer-user

Even though the script failed, the logs reported the build step as successful, which is unexpected behaviour on infra.ci (https://infra.ci.jenkins.io/job/infra-tools/job/packer-images-gc/job/main/65/pipeline-console/?selected-node=19)

Also there was an unexpected behaviour with the aws_snapshots.sh script that could not delete snapshots with the error The snapshot snap-0999857b5cac1fd3a is currently in use by ami-0cfe784c66xxxx

Further steps will involve fixing these errors and unexpected behaviour of infra.ci

[jayfranco999]

• Disable the GitHub checks report from infra.ci to the repository

jenkins-infra/kubernetes-management#5895 partially works as the configuration was applied on garbage-collector for packer images folder on infra.ci

While tag discovery was successfully disabled, the github checks report of the garbage collector is still published on packer-images repository, we are working to fix this issue

[dduportal]

• Disable the GitHub checks report from infra.ci to the repository

Explanation: your changes were good and did disable the usual "Status checks" (e.g. one status check per stage by default, pipeline reporting and custom status checks).
But what we see in your screenshot above are "Notifications", which are a subset of "Status checks" (if any) but not only: the GitHub Branch Source plugin also sends a notification as part of the pipeline report.

If you look at the configuration below, you see the checkbox "Skip publishing Status check" is selected , which was not the case before your changes 👍

However, we now need to have the "Suppress progress updates in job check" and "Skip GitHub Branch Source notifications" checbox selected to be sure that:

• No notifications are sent at all when pipeline finished
• Any "build in progress" event should also not send any notification (when pipeline starts)

We need to allow these to be configured in the "Jenkins Jobs" helm chart in https://github.com/jenkins-infra/helm-charts/blob/575396eb4f6c813f59a36610594c12446222935c/charts/jenkins-jobs/templates/_jobDSL_jobs_multibranch.tpl#L38-L40 => let me do this right now.

[dduportal]

Update on the job configuration:

• Added a new feature on the jenkins-jobs helm chart to totally disable notifications.
  • NOTE: this feature also has a bugfix which sync. the "progress updates checks" with other checks => there are now disabled or enabled together (which fixes partailly our problem here).
  • This new feature is not enabled (yet?) on the job though (non regrerssion)

[dduportal]

Update:

• hotfix(aws-garbage-collector) enables debugging for aws.sh script packer-images#1544 did fix the aws.sh, added debug and allow the process to fail fast when facing missing permissions 👏
• Added the missing permission in https://github.com/jenkins-infra/terraform-states/commit/a39d169e096c3c9837f6f1d7609148ab098d5bf3.
• Checked the new build status: aws.sh now work and remove dangling instances and security groups 👏
• New error on the aws_snapshot.sh script: An error occurred (InvalidSnapshot.InUse) when calling the DeleteSnapshot operation: The snapshot snap-0207fe814a28f361b is currently in use by ami-006a00b674bfd3f70. Should be fixed with the same technique as in hotfix(aws-garbage-collector) enables debugging for aws.sh script packer-images#1544

[dduportal]

• We checked if aws.sh was effective. For (dangling) instances, it is not finding any with the command aws ec2 describe-instances --filters 'Name=tag:Name,Values=*Packer*' --query 'Reservations[].Instances[?LaunchTime<=2024-11-21][].InstanceId'. However, we see ~15 dangling instance (terminated)

The root cause comes from a recent change in the Packer Amazon EBS build:

The builder no longer adds a "Name": "Packer Builder" entry to the tags.

=> we need to add a tag Name with a value matching the GC filter in https://github.com/jenkins-infra/packer-images/blob/2a473a06f70ada1bc48cd985fb836119bc09ee31/sources.pkr.hcl#L25-L32

Ref. https://developer.hashicorp.com/packer/integrations/hashicorp/amazon/latest/components/builder/ebs#tags

[dduportal]

Last tasks have been done (see #4355)

@jayfranco999 reported we don't have dangling instances no more \o/