Skip to content

Commit

Permalink
OCI-attach reports from clair-scan 0.2 Task
Browse files Browse the repository at this point in the history
This attaches the "clair"-formatted output of the `clair-action` command
to the scanned image. Contrary to the `sast-snyk-check` Task only a
single variant of attachment method is supported, based on the registry
support. For quay.io OCI Distribution 1.1. Referrers API will be used.

The `clair-action` needs to be run twice as the Rego rules executed in
the `conftest-vulnerabilities` require the "quay" format, which does not
include any date information, whereas the (future) EC policy Rego rules
require the "clair" format, which does.

Resolves: https://issues.redhat.com/browse/EC-837
  • Loading branch information
zregvart committed Oct 3, 2024
1 parent f3f7493 commit a87adda
Show file tree
Hide file tree
Showing 9 changed files with 250 additions and 7 deletions.
1 change: 1 addition & 0 deletions pipelines/docker-build-multi-platform-oci-ta/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/reposito
|name|description|used in params (taskname:taskrefversion:taskparam)
|---|---|---|
|IMAGES_PROCESSED| Images processed in the task.| |
|REPORTS| Mapping of image digests to report digests| |
|SCAN_OUTPUT| Clair scan result.| |
|TEST_OUTPUT| Tekton task test output.| |
### clamav-scan:0.1 task results
Expand Down
1 change: 1 addition & 0 deletions pipelines/docker-build-oci-ta/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/reposito
|name|description|used in params (taskname:taskrefversion:taskparam)
|---|---|---|
|IMAGES_PROCESSED| Images processed in the task.| |
|REPORTS| Mapping of image digests to report digests| |
|SCAN_OUTPUT| Clair scan result.| |
|TEST_OUTPUT| Tekton task test output.| |
### clamav-scan:0.1 task results
Expand Down
1 change: 1 addition & 0 deletions pipelines/docker-build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,7 @@ This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/reposito
|name|description|used in params (taskname:taskrefversion:taskparam)
|---|---|---|
|IMAGES_PROCESSED| Images processed in the task.| |
|REPORTS| Mapping of image digests to report digests| |
|SCAN_OUTPUT| Clair scan result.| |
|TEST_OUTPUT| Tekton task test output.| |
### clamav-scan:0.1 task results
Expand Down
1 change: 1 addition & 0 deletions pipelines/java-builder/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@
|name|description|used in params (taskname:taskrefversion:taskparam)
|---|---|---|
|IMAGES_PROCESSED| Images processed in the task.| |
|REPORTS| Mapping of image digests to report digests| |
|SCAN_OUTPUT| Clair scan result.| |
|TEST_OUTPUT| Tekton task test output.| |
### clamav-scan:0.1 task results
Expand Down
1 change: 1 addition & 0 deletions pipelines/nodejs-builder/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@
|name|description|used in params (taskname:taskrefversion:taskparam)
|---|---|---|
|IMAGES_PROCESSED| Images processed in the task.| |
|REPORTS| Mapping of image digests to report digests| |
|SCAN_OUTPUT| Clair scan result.| |
|TEST_OUTPUT| Tekton task test output.| |
### clamav-scan:0.1 task results
Expand Down
1 change: 1 addition & 0 deletions pipelines/tekton-bundle-builder/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@
|name|description|used in params (taskname:taskrefversion:taskparam)
|---|---|---|
|IMAGES_PROCESSED| Images processed in the task.| |
|REPORTS| Mapping of image digests to report digests| |
|SCAN_OUTPUT| Clair scan result.| |
|TEST_OUTPUT| Tekton task test output.| |
### clamav-scan:0.1 task results
Expand Down
9 changes: 5 additions & 4 deletions task/clair-scan/0.2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,11 @@ analyzing the components of a container image and comparing them against Clair's

## Results:

| name | description |
|-------------------|--------------------------|
| TEST_OUTPUT | Tekton task test output. |
| SCAN_OUTPUT | Clair scan result. |
| name | description |
|-------------------|------------------------------------------|
| TEST_OUTPUT | Tekton task test output. |
| SCAN_OUTPUT | Clair scan result. |
| REPORTS |Mapping of image digests to report digests|

## Clair-action repository:
https://github.com/quay/clair-action
Expand Down
65 changes: 62 additions & 3 deletions task/clair-scan/0.2/clair-scan.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ spec:
description: Clair scan result.
- name: IMAGES_PROCESSED
description: Images processed in the task.
- name: REPORTS
description: Mapping of image digests to report digests
stepTemplate:
volumeMounts:
- name: trusted-ca
Expand Down Expand Up @@ -92,22 +94,36 @@ spec:
value: $(params.image-url)
- name: IMAGE_DIGEST
value: $(params.image-digest)
workingDir: /tekton/home
script: |
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
imagewithouttag=$(echo -n $IMAGE_URL | sed "s/\(.*\):.*/\1/")
images_processed_template='{"image": {"pullspec": "'"$IMAGE_URL"'", "digests": [%s]}}'
digests_processed=()
for sha_file in /tekton/home/image-manifest-*.sha; do
# the quay report format used by the Conftest rules in the
# conftest-vulnerabilities step doesn't contain the "issued" date which
# we require in the policy rules, so we resort to running clair-action
# twice to produce both quay and clair formatted output
clair_report() {
{ clair-action report --image-ref="$1" --db-path=/tmp/matcher.db --format=quay | tee "clair-result-$2.json"; } && \
{ clair-action report --image-ref="$1" --db-path=/tmp/matcher.db --format=clair > "clair-report-$2.json"; }
}
for sha_file in image-manifest-*.sha; do
if [ -e "$sha_file" ]; then
arch_sha=$(cat "$sha_file")
arch=$(basename "$sha_file" | sed 's/image-manifest-//;s/.sha//')
arch_specific_digest="$imagewithouttag@$arch_sha"
echo "Running clair-action on $arch image manifest."
# run the scan for each image manifest in the image index
clair-action report --image-ref=$arch_specific_digest --db-path=/tmp/matcher.db --format=quay | tee /tekton/home/clair-result-$arch.json || true
clair_report "${arch_specific_digest}" "${arch}" || true
digests_processed+=("\"$arch_sha\"")
fi
Expand All @@ -117,7 +133,48 @@ spec:
# add the image_index to the processed digests list and store the result in a file
images_processed=$(echo "${images_processed_template/\[%s]/[$digests_processed_string]}")
echo "$images_processed" > /tekton/home/images-processed.json
echo "$images_processed" > images-processed.json
- name: oci-attach-report
image: quay.io/konflux-ci/oras:latest@sha256:56589c1c9132aeaccffad2fc2fd1c3b612741b961e177a91abeb81cb0d859ff2
workingDir: /tekton/home
env:
- name: IMAGE_URL
value: $(params.image-url)
script: |
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
if ! compgen -G "clair-report-*.json" > /dev/null; then
echo 'No Clair reports generated. Skipping upload.'
exit 0
fi
echo "Selecting auth"
select-oci-auth "$IMAGE_URL" > "$HOME/auth.json"
repository="${IMAGE_URL/:*/}"
arch() {
report_file="$1"
arch="${report_file/*-}"
echo "${arch/.json/}"
}
MEDIA_TYPE='application/vnd.redhat.clair-report+json'
reports_json=""
for f in clair-report-*.json; do
digest=$(cat "image-manifest-$(arch "$f").sha")
image_ref="${repository}@${digest}"
echo "Attaching $f to ${image_ref}"
report_digest="$(oras attach --no-tty --format go-template='{{.digest}}' --registry-config "$HOME/auth.json" --artifact-type "${MEDIA_TYPE}" "${image_ref}" "$f:${MEDIA_TYPE}")"
# shellcheck disable=SC2016
reports_json="$(yq --output-format json --indent=0 eval-all '. as $i ireduce ({}; . * $i)' <(echo "${reports_json}") <(echo "${digest}: ${report_digest}"))"
done
echo "${reports_json}" > reports.json
- name: conftest-vulnerabilities
image: quay.io/redhat-appstudio/konflux-test:v1.4.7@sha256:cf6808a3bd605630a5d9f20595ff7c43f8645c00381219d32f5a11e88fe37072
# per https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting
Expand Down Expand Up @@ -206,6 +263,8 @@ spec:
echo "$scan_result" | tee "$(results.SCAN_OUTPUT.path)"
cat /tekton/home/images-processed.json | tee $(results.IMAGES_PROCESSED.path)
# shellcheck disable=SC2154
cat /tekton/home/reports.json > "$(results.REPORTS.path)"
note="Task $(context.task.name) completed: Refer to Tekton task result SCAN_OUTPUT for vulnerabilities scanned by Clair."
TEST_OUTPUT=$(make_result_json -r "SUCCESS" -t "$note")
Expand Down
177 changes: 177 additions & 0 deletions task/clair-scan/0.2/spec/clair_scan_spec.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
#!/bin/env bash

set -o errexit
set -o pipefail
set -o nounset

eval "$(shellspec - -c) exit 1"

task_path=clair-scan.yaml

if [[ -f "../${task_path}" ]]; then
task_path="../${task_path}"
fi


extract_script() {
script="$(mktemp --tmpdir script_XXXXXXXXXX.sh)"
yq -r ".spec.steps[] | select(.name == \"$1\").script" "${task_path}" > "${script}"
chmod +x "${script}"

echo "${script}"
}

# array containing files/directories to remove on test exit
cleanup=()
trap 'rm -rf "${cleanup[@]}"' EXIT

# Extract the get-vulnerabilities Step script so we can test it
get_vulnerabilities_script="$(extract_script get-vulnerabilities)"
cleanup+=("${get_vulnerabilities_script}")

testdir() {
testdir="$(mktemp -d)" && cleanup+=("${testdir}") && cd "${testdir}"

AfterEach 'rm -rf "$testdir"'
}

clair_report() {
echo "report --image-ref=registry.io/repository/image@$1 --db-path=/tmp/matcher.db --format=$2"
}


Describe "get vulnerabilities"
BeforeEach testdir

export IMAGE_URL=registry.io/repository/image:tag
export IMAGE_DIGEST=sha256:f0cacc1a

It "generates reports and images-processed.json"
Mock clair-action
clair_action_args+=("$*")
%preserve clair_action_args
# expecting the --format parameter to be the last one
echo "report in ${clair_action_args[-1]#*--format=} format"
End
echo "sha256:f0cacc1a" > image-manifest-amd64.sha
echo "sha256:cc1af0ca" > image-manifest-arm64.sha

When call "${get_vulnerabilities_script}"
The output should eq "Running clair-action on amd64 image manifest.
report in quay format
Running clair-action on arm64 image manifest.
report in quay format"
The contents of file "images-processed.json" should equal '{"image": {"pullspec": "registry.io/repository/image:tag", "digests": ["sha256:f0cacc1a","sha256:cc1af0ca"]}}'
The contents of file "clair-result-amd64.json" should equal 'report in quay format'
The contents of file "clair-report-amd64.json" should equal 'report in clair format'
The contents of file "clair-result-arm64.json" should equal 'report in quay format'
The contents of file "clair-report-arm64.json" should equal 'report in clair format'
The variable clair_action_args[@] should eq "$(clair_report sha256:f0cacc1a quay) "\
"$(clair_report sha256:f0cacc1a clair) "\
"$(clair_report sha256:cc1af0ca quay) "\
"$(clair_report sha256:cc1af0ca clair)"
End

It "fails in clair-action quay report"
Mock clair-action
clair_action_args+=("$*")
%preserve clair_action_args
[[ "$*" == *--format=quay* ]] && echo "didn't work out" && exit 1
End
echo "sha256:f0cacc1a" > image-manifest-amd64.sha

When call "${get_vulnerabilities_script}"
The output should eq "Running clair-action on amd64 image manifest.
didn't work out"
The contents of file "images-processed.json" should equal '{"image": {"pullspec": "registry.io/repository/image:tag", "digests": ["sha256:f0cacc1a"]}}'
The contents of file "clair-result-amd64.json" should equal "didn't work out"
The file "clair-report-amd64.json" should not exist
The variable clair_action_args[@] should eq "$(clair_report sha256:f0cacc1a quay)"
End
End

# Extract the oci-attach-report Step script so we can test it
oci_attach_report_script="$(extract_script oci-attach-report)"
cleanup+=("${oci_attach_report_script}")

oras_attach() {
echo "attach --no-tty --format go-template={{.digest}} --registry-config $HOME/auth.json --artifact-type application/vnd.redhat.clair-report+json registry.io/repository/image@$1 $2:application/vnd.redhat.clair-report+json"
}

Describe "OCI attach report"
BeforeEach testdir

export IMAGE_URL=registry.io/repository/image:tag

It "skips attachments if no reports generated"
Mock select-oci-auth
echo select-oci-auth should not be called
End

Mock oras
echo oras should not be called
End

When call "${oci_attach_report_script}"
The output should eq "No Clair reports generated. Skipping upload."
End

It "attaches for single architecture"
export HOME="${testdir}"

Mock select-oci-auth
echo selected auth
End

Mock oras
oras_args+=("$*")
%preserve oras_args
echo report-digest
End

echo "sha256:f0cacc1a" > image-manifest-amd64.sha
touch clair-report-amd64.json

When call "${oci_attach_report_script}"
The output should eq "Selecting auth
Attaching clair-report-amd64.json to registry.io/repository/image@sha256:f0cacc1a"
The contents of file "auth.json" should equal "selected auth"
The variable oras_args[@] should eq "$(oras_attach sha256:f0cacc1a clair-report-amd64.json)"
The contents of file "reports.json" should equal '{"sha256:f0cacc1a":"report-digest"}'
End

It "attaches for multiple architecture"
export HOME="${testdir}"

Mock select-oci-auth
echo selected auth
End

Mock oras
oras_args+=("$*")
%preserve oras_args
for a in "$@"; do
if [[ "$a" == *@sha256:* ]]; then
echo "sha256:$(echo "${a/*@sha256:/}" | rev)"
break
fi
done
End

echo "sha256:f0cacc1a" > image-manifest-amd64.sha
echo "sha256:cc1af0ca" > image-manifest-arm64.sha
echo "sha256:f01acacc" > image-manifest-ppc64le.sha
touch clair-report-{amd64,arm64,ppc64le}.json

When call "${oci_attach_report_script}"
The output should eq "Selecting auth
Attaching clair-report-amd64.json to registry.io/repository/image@sha256:f0cacc1a
Attaching clair-report-arm64.json to registry.io/repository/image@sha256:cc1af0ca
Attaching clair-report-ppc64le.json to registry.io/repository/image@sha256:f01acacc"
The contents of file "auth.json" should equal "selected auth"
The variable oras_args[@] should eq "$(oras_attach sha256:f0cacc1a clair-report-amd64.json) "\
"$(oras_attach sha256:cc1af0ca clair-report-arm64.json) "\
"$(oras_attach sha256:f01acacc clair-report-ppc64le.json)"
The contents of file "reports.json" should equal '{"sha256:f0cacc1a":"sha256:a1ccac0f","sha256:cc1af0ca":"sha256:ac0fa1cc","sha256:f01acacc":"sha256:ccaca10f"}'
End
End

0 comments on commit a87adda

Please sign in to comment.