Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advisory data: tombstone events for withdrawn packages #576

Open
luhring opened this issue Jan 26, 2024 · 4 comments
Open

Advisory data: tombstone events for withdrawn packages #576

luhring opened this issue Jan 26, 2024 · 4 comments
Labels
enhancement New feature or request needs-triage applied to all new customer/user issues. Removed after triage occurs.

Comments

@luhring
Copy link
Member

luhring commented Jan 26, 2024

Context

We recently added some much needed validation of our advisory data into wolfictl, which is used as a CI check in our advisories repos. The validation rules relevant to this proposal are:

  1. "Fixed versions" of packages must exist in the relevant APKINDEX.
  2. "Fixed versions" must not be the first version of a package in the APKINDEX.
  3. Existing event data must not be removed or modified. Updating the status of an advisory should be achieved via appending a new event to the end of the sequence.

Rules 1 and 2 are checked across all data in the advisories repo. Rule 3 is a function of what was changed in the current PR (relative to the designated fork point).

Meanwhile... we also withdraw packages (i.e. specific APK files) from the distro from time to time.

This results in an unpleasant side effect where: a new fixed event can be added that's valid because the package version exists, then the package version is withdrawn, and then validation runs again and fails.

Proposal

(Credit to @jonjohnsonjr for this idea 🧠 )

To allow our advisory data entry workflow to satisfy our validation checks, continue with our transparent "append only" philosophy, and account for withdrawn packages, we could create a new event type to act as a tombstone entry, which says that a previously referenced fixed version no longer exists.

The impact on downstream data transformation, and on the secdb in particular, would be that we no longer report that fixed version for the advisory — the fixed information is effectively reverted to its state prior to the original fixed event.

We would update our validation rules such that:

  1. A fixed event is allowed to refer to non-existent APK version, as long as there exists an event later in the advisory's event sequence that's the "tombstone" event for that package version.
  2. A tombstone event must specify a package version that's referenced earlier in the advisory's event sequence.

It would also be great to have the dev tooling and automation help us, such as by automatically adding tombstone events as needed at time of package withdrawal.

@luhring luhring added enhancement New feature or request needs-triage applied to all new customer/user issues. Removed after triage occurs. labels Jan 26, 2024
@luhring
Copy link
Member Author

luhring commented Jan 31, 2024

Another related case to solve is this: https://github.com/wolfi-dev/advisories/actions/runs/7719825548/job/21043730046?pr=1036#step:5:26

...where it's not the fixed version that was removed, but a version before it, which would (as a side effect) cause the validation rule that "a fixed-version can't be the first version in the APK's history" to fail.

@luhring
Copy link
Member Author

luhring commented Feb 5, 2024

An alternative solution discussed recently would be to avoid storing tombstones in the advisory data itself, and instead ensure that a full list of withdrawn packages exists in the distro repo (e.g. wolfi-dev/os). Validation on advisory data could take this manifest of withdrawn versions into account, similar to how tombstones are used in the above proposal.

@jonjohnsonjr
Copy link
Contributor

For posterity:

git log -p -- withdrawn-packages.txt | grep "^+" | grep ".apk$" | cut -c2- | sort | uniq > withdrawn-packages.txt

@jonjohnsonjr
Copy link
Contributor

Okay wolfi now has this:

curl -sL https://packages.wolfi.dev/os/withdrawn-packages.txt | wc -l
     637

We should be able to key off that for deleted packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs-triage applied to all new customer/user issues. Removed after triage occurs.
Projects
None yet
Development

No branches or pull requests

2 participants