Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(checks): Add automated bundle testing in pipeline #7031

Open
simar7 opened this issue Jun 26, 2024 · 2 comments
Open

test(checks): Add automated bundle testing in pipeline #7031

simar7 opened this issue Jun 26, 2024 · 2 comments
Assignees
Labels
kind/testing Categorizes issue or PR as related to a unit/integration test. scan/misconfiguration Issues relating to misconfiguration scanning
Milestone

Comments

@simar7
Copy link
Member

simar7 commented Jun 26, 2024

Today when a PR is submitted in the checks repo, we have some basic testing that takes place as described in this workload. However, it offers no integration testing with Trivy itself.

For this to take place, we can do the following:

  1. Create a test environment with certain misconfigurations within the trivy-checks repo
  2. Create a bundle with the changes submitted in the PR within a GitHub Action pipeline
  3. Load this bundle into the latest release of Trivy, will be implemented via feat(misconfig): Add flag to supply checks bundle locally #7029
  4. Make assertions on the checks themselves but also warrant that the bundle is loaded successfully.

By adding this automated layer of testing two problems will be tackled:

  1. Avoiding surprise breakages when upgrading trivy-checks usage within Trivy.
  2. Enabling automated scheduled releases of trivy-checks bundles without operator intervention. More details here: feat(checks): Schedule automated releases for trivy-checks bundles #7030
@simar7 simar7 added kind/testing Categorizes issue or PR as related to a unit/integration test. scan/misconfiguration Issues relating to misconfiguration scanning labels Jun 26, 2024
@nikpivkin
Copy link
Contributor

CI in trivy-checks is already doing integration testing of the checks bundle with the latest version of Trivy and seems to meet all the requirements described here. https://github.com/aquasecurity/trivy-checks/blob/main/scripts/verify-bundle.go

@simar7
Copy link
Member Author

simar7 commented Jul 6, 2024

CI in trivy-checks is already doing integration testing of the checks bundle with the latest version of Trivy and seems to meet all the requirements described here. https://github.com/aquasecurity/trivy-checks/blob/main/scripts/verify-bundle.go

Yes but there's currently no functional verification. We also lack testing against multiple versions of Trivy as we only test latest right now.

@simar7 simar7 added this to the v0.58.0 milestone Oct 24, 2024
@simar7 simar7 assigned simar7 and nikpivkin and unassigned nikpivkin Nov 2, 2024
@simar7 simar7 modified the milestones: v0.58.0, v0.59.0 Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/testing Categorizes issue or PR as related to a unit/integration test. scan/misconfiguration Issues relating to misconfiguration scanning
Projects
Status: No status
Development

No branches or pull requests

2 participants