-
QuestionHello, recently trivy is having issues updating the database before scanning container images. To isolate the problem, I have introduced a separate update task which runs the update-only commands. Here is the update commands output:
From my point of view, it looks like it's a problem related to the Github container registry. I guess there is some kind of DDoS protection in place. Has anyone already faced the same issue? Or already found a solution? Best regards, TargetContainer Image ScannerNone Output FormatNone ModeStandalone Operating SystemUbuntu Version0.54.1 |
Beta Was this translation helpful? Give feedback.
Replies: 13 comments 56 replies
-
|
Hi @gwitsch ! We have encountered the same thing at CI and are still looking into the problem. |
Beta Was this translation helpful? Give feedback.
-
|
I have seen this as well |
Beta Was this translation helpful? Give feedback.
-
|
Please see https://aquasecurity.github.io/trivy/v0.55/docs/references/troubleshooting/#github-rate-limiting |
Beta Was this translation helpful? Give feedback.
-
|
For those that my have a on site registry that can act as a cache for remote content, the following environment variables can prevent the problem by making Trivy pull from the cache on not hit the ghcr pull limit: where |
Beta Was this translation helpful? Give feedback.
-
|
Adding a Github token seems to be working. At least the last couple of days, the update process worked as expected. |
Beta Was this translation helpful? Give feedback.
-
|
I added a token and 1 out of 2 scans still failed. Haven’t seen any failures since Thursday though without using token. |
Beta Was this translation helpful? Give feedback.
-
|
Sadly, Github tokens don't fix the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Same here. |
Beta Was this translation helpful? Give feedback.
-
|
We published trivy-db in Amazon ECR Public Gallery as well as GHCR. The following should work now. We'll update the documentation tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
We tried the list: 2024-10-14T14:45:11Z INFO [db] Need to update DB |
Beta Was this translation helpful? Give feedback.
-
|
@Connor-Wood i'm sorry i don't know, i just saw it in the docs |
Beta Was this translation helpful? Give feedback.
-
|
I would guess that you used a slightly wrong command line syntax. Not sure how Because the way you did it, trivy won't understand that the second repo's URI belongs to the |
Beta Was this translation helpful? Give feedback.
-
|
@knqyf263 Could you please populate the trivy-chekcs repo too on ecr? https://gallery.ecr.aws/aquasecurity/trivy-checks |
Beta Was this translation helpful? Give feedback.
-
|
@ktzsolt I'm not an owner of that. It's being discussed here. aquasecurity/trivy-checks#262 |
Beta Was this translation helpful? Give feedback.
-
|
I've not been able to get a successful Trivy run in GitLab for the last few hours due to this issue. We've seen it on and off in the past but all of our installs fail 100% now. This is causing our pipelines to fail so we can no longer merge code. |
Beta Was this translation helpful? Give feedback.
-
that makes sense, thank you for providing the context |
Beta Was this translation helpful? Give feedback.
-
The long term solution doesn't seem to be working for us as we had this in place long before these issues have been popping up. For the short term solution being using ECR how frequently is this image updated as I don't want to be pointing to an image that will only be updated short term? Is there a long-term solution we can use or are we good to permanently use the ECR image from now on? |
Beta Was this translation helpful? Give feedback.
-
@knqyf263 If you need any help with Gitlab configuration, let me know. This is what our Gitlab job looks like: We're still facing issues with the TOOMANYREQUESTS error though. As you can see we're using a cache:key: value here. This defines when the cache is re-used in another job. In this case I'm using a hardcoded string value, so each pipeline job in the same repository will use the same cache. However, we have a lot of different projects and repository and it's not possible to share cache across repositories in Gitlab (due to security reasons and cache poisoning). |
Beta Was this translation helpful? Give feedback.
-
Does this work for you? Because (on GitHub Actions btw) with I'm getting |
Beta Was this translation helpful? Give feedback.
-
It works fine for me on all of our Gitlab projects. |
Beta Was this translation helpful? Give feedback.
-
|
I found an option to create a local cache by setting the environment variable TRIVY_CACHE_DIR= and using the command trivy --download-db-only. It downloads the database to the specified path, and when scanning images, it won't connect to https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2. I think this could solve the issue. Doc : https://aquasecurity.github.io/trivy/v0.56/docs/advanced/air-gap/#populating-the-trivy-cache |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, @bryanhermass, |
Beta Was this translation helpful? Give feedback.
-
|
@kapi-caz I'm using AZ DevOps, but only specifying cache folder is not a solution, didn't try to add '--download-db-only' but here is the question, why I need this flag, and what will it affect if I'm using like this in the future ? |
Beta Was this translation helpful? Give feedback.
-
|
It’s not working with me on Az DevOps even when specifying both flags TRIVY_CACHE_DIR= + --download-db-only. |
Beta Was this translation helpful? Give feedback.
-
|
@bryanhermass @kapi-caz does anyone bypassed the error on az devops ? how ? I tried as mentioned adding two flags but still see the errors |
Beta Was this translation helpful? Give feedback.
-
|
But when you need to update your database the issue keep happen. I use github actions here and the solution that I had to use was cache for the db and config file changing the repository to new one that they sent here and the issues is happens much less now. |
Beta Was this translation helpful? Give feedback.
-
|
I tried all of the suggestions above. Running on Gitlab, enabled artifacts for the I am scanning ~6-7 images in seperate pipelines so it makes sense that I am hitting some rate-limit, but I cannot figure out how I am supposed to be configuring this cache to work properly... I think the real issue is that Am I missing a CLI flag or something? |
Beta Was this translation helpful? Give feedback.
-
|
Update: I ended up cloning the docker images to a local docker registry and using that in my CI/CD pipelines and I was able to resolve the issue. The problem is the cloning of the To solve it I used skopeo to copy the image (since it is OCI docker cannot copy it) into our artifactory instance, and then pointed my pipeline to my company's artifactory instance. This is my pipeline: container_scan:
stage: containerize
# https://aquasecurity.github.io/trivy/v0.18.3/usage/image/
image:
name: docker.io/aquasec/trivy:latest
entrypoint: [ "" ]
variables:
# No need to clone the repo
GIT_STRATEGY: none
# Variables to authenticate to our private registry
TRIVY_USERNAME: "$JFROG_CI_USER"
TRIVY_PASSWORD: "$JFROG_CI_PASSWORD"
TRIVY_AUTH_URL: "$ARTIFACTORY_REGISTRY"
# Misc.
TRIVY_NO_PROGRESS: "true"
TRIVY_CACHE_DIR: ".trivycache/"
# Configure Trivy to pull the database images from our local private images instead of ghcr.io
TRIVY_DB_REPOSITORY: artifactory.mydomain.com/docker-mirror/aquasecurity/trivy-db:2
TRIVY_JAVA_DB_REPOSITORY: artifactory.mydomain.com/docker-mirror/aquasecurity/trivy-java-db:1
# The image I want scanned
FULL_IMAGE_NAME: ${ARTIFACTORY_REGISTRY}/${CONTAINER_NAME}:${CI_COMMIT_TAG:-$CI_COMMIT_SHA}
script:
# Dump version.
- trivy --version
# Clear out the previous scan history from the cache
- time trivy clean --scan-cache
# update databases
- time trivy image --download-db-only
- time trivy image --download-java-db-only
# Builds report and puts it in the default workdir $CI_PROJECT_DIR, so `artifacts:` can take it from there
- time trivy image --exit-code 0 --format template --template "@/contrib/gitlab.tpl" --output "$CI_PROJECT_DIR/gl-container-scanning-report.json" "$FULL_IMAGE_NAME"
# Prints full report to the console
- time trivy image --exit-code 0 "$FULL_IMAGE_NAME"
# Fail on critical vulnerabilities
- time trivy image --exit-code 1 --severity CRITICAL "$FULL_IMAGE_NAME"
cache:
key: trivy-cache
paths:
- .trivycache/
artifacts:
when: always
reports:
container_scanning: gl-container-scanning-report.jsonHopefully this helps some future souls. Though it's a bit rough for people who don't have any way to mirror docker images to some local cache/registry...
Turns out this was happening because I have 10 CI/CD runners and the trivy cache only gets placed onto individual runners. So the trivy cache was runner dependent, and getting a different CI/CD runner would caus the image to get pulled again. Something to keep in mind if you are using multiple runners... |
Beta Was this translation helpful? Give feedback.
-
|
Anyone trying to run it locally as part of pre-commit checks when pushing the code upstream? default_stages:
- pre-commit
repos:
- repo: local
hooks:
- id: trivy-fs
stages:
- pre-push
name: trivy-fs
language: docker_image
entry: -v /tmp:/.cache aquasec/trivy:latest fs -c .config/trivy.yaml .
pass_filenames: false |
Beta Was this translation helpful? Give feedback.
-
|
We found that |
Beta Was this translation helpful? Give feedback.
-
|
It works like a charm. I don't find any rate limits on mirror.gcr.io. |
Beta Was this translation helpful? Give feedback.
-
|
One concern is that the old artifact is cached for a long time, and clients cannot get the latest DBs. |
Beta Was this translation helpful? Give feedback.
-
|
From my experiment, the cached artifact seems invalidated after 1 hour. I'm not sure exactly 1 hour, but it doesn't look like 1 day or 1 week. |
Beta Was this translation helpful? Give feedback.
-
|
I consider a delay of up to one hour in updating the DBs to be acceptable. I'd set |
Beta Was this translation helpful? Give feedback.
-
|
Tried @knqyf263 's answer and it worked for the trivy db but was still getting request errors for the Java DB, tried with AWS ECR gallery and that worked. |
Beta Was this translation helpful? Give feedback.
-
|
v0.57.1 with |
Beta Was this translation helpful? Give feedback.
-
|
looking good so far, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
i get this: 2024-11-19T10:47:01Z ERROR [vulndb] Failed to download artifact repo="ghcr.io/aquasecurity/trivy-db:2" err="oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:d8f527b0dde6241ad3cfa63f82e6b0a8c9a64ba55571a06063dad700b3a15390: TOOMANYREQUESTS: retry-after: 426.501µs, allowed: 44000/minute" after flipping to |
Beta Was this translation helpful? Give feedback.
-
|
for trivy-action check this: aquasecurity/trivy-action#389 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
Currrently getting an error, even with the new GCR mirror on Trivy |
Beta Was this translation helpful? Give feedback.
-
|
Is it still persisting? |
Beta Was this translation helpful? Give feedback.
-
|
It started working again after a little while. |
Beta Was this translation helpful? Give feedback.
-
|
Can confirm, it fails sometimes with BLOB_UNKNOWN as @baksetercx wrote. Much better than with ghcr but still fails sometimes. |
Beta Was this translation helpful? Give feedback.
and others
I consider a delay of up to one hour in updating the DBs to be acceptable. I'd set
mirror.gcr.ioas primary and setghcr.ioas secondary.