An update on rate limiting issue (TOOMANYREQUESTS)

[itaysk]

Recently, Trivy users have been experiencing delays and errors when using Trivy. This page explains the situation and mitigation steps.

What is happening?

When you install Trivy, the installed artifact contains the scanner engine but is lacking relevant security information needed to make security detections and recommendations. These external assets (sometimes referred to as "databases") are hosted in GitHub Container Registry and maintained by Trivy automatically as needed, so normally you shouldn't notice or worry about them.
GitHub offers a very generous free hosting for open source packages in GHCR, but it’s not without limits. Recently, the aggregate load that Trivy users generated on GHCR supposedly exceeded the maximum rate limit, which is currently 44000 requests per minute per namespace (that includes all images under Aqua’s organization). In such surge registry pulls will be throttled, and Trivy will be unable to proceed and report an error (TOOMANYREQUESTS).

Mitigation steps

We have been working hard over the past weeks to come up with solutions and workarounds to this problem, while remaining under the constraints of public free infrastructure. Some of the actions were implemented in Trivy and its ecosystem, while other solutions are recommended for users to implement.

Trivy team

Following are the steps that Trivy have taken to try minimizing the impact of this situation:

1. Trivy already supported users overriding the registry location. This configuration now accepts multiple alternative registry locations, and Trivy will now fall back through alternative locations before failing. Read more in Trivy's databases documentation
2. Started pushing Trivy’s databases to AWS ECR Public Gallery: public.ecr.aws/aquasecurity/trivy-db
   and to Docker Hub public registry: aquasec/trivy-db
   . Read more in Trivy's databases documentation.
3. Having the images in Docker Hub enables integration with pull-through cache solutions. Particularly, Trivy images are available in GCR Mirror which will now become the default location for Trivy’s images.
4. Introduced caching in the Trivy GitHub Action, so that consecutive runs will not need to re-pull the databases on every run. Read more here: https://github.com/aquasecurity/trivy-action#cache
5. Reduced the frequency of Trivy DB updates](ci: set to download DB daily trivy-db#444) from every 6 hours to every 24 hours. This will make Trivy need to pull the DB less often, thus reduce the load on GHCR.
6. Fixed a bug that caused Trivy to pull Java DB unnecessarily even if there was no update.
7. Note that the rate limiting issue affected Trivy's Vulnerability DB and Java Packages DB but did not affect the Checks bundle (used for IaC/misconfiguration scanning) thanks for the fact that we have been embedding it into the Trivy binary, and falling back to using the embedding version in case of errors.

Trivy users

Following are the steps that users could take to try minimizing the impact of this situation:

1. Consider hosting Trivy DBs in your own container registry. This way you will not compete on shared resources with the rest of the community. This is documentation in the self-hosting guide
2. Use a pull-through cache to keep a local copy of images and prevent overloading the origin servers. Your cloud or CI/CD platform might already have a solution for this.
3. If running from AWS, you might want to set ECR as the primary registry location , and authenticate with your AWS access key. Read more here
4. If running from GitHub Actions, please use the latest version of Trivy action that automatically leverages GitHub Actions cache. Read more here. If you are not using the official trivy-action please consider using GitHub’s actions/cache to achieve similar effect.
5. If running Trivy in other CI pipelines, use your CI platform’s cache management to persist Trivy’s cache directory between runs, so that Trivy doesn’t try to pull DBs on every single scan.

Setting expectations

We strive to make Trivy accessible and available to all. Trivy is open source and relies on public free infrastructure so it's reasonable to expect it will not scale inifinitely. We hope that the improvements that we have recently made will improve your experience and allow you to continue enjoying Trivy. If you are using Trivy at work and your business relies on it, consider an enterprise grade solution like Aqua Security which guerentees availability and reliability.