feat: add support for "pinned" images #920

inFocus7 · 2023-11-29T23:29:54Z

TODO (very likely) - move logic from the scanner to the collector & remover

What this PR does / why we need it:

Add support for CRI pinned images. Pinned is a field images which should not be garbage collected have. When scanning images for removal we should .

We also make this configurable to allow users to decide whether or not they care about pinned images.

Which issue(s) this PR fixes:
Fixes #380

Special notes for your reviewer:
The configurability can be removed if we think that images being pinned is enough for us to assume they should never be deleted.

Notes for me (Fabian) so I ensure these behaviors occur...

deletePinnedImages: true

	isPinned	notPinned
isVulnerable	delete	delete
notVulnerable	keep	keep

deletePinnedImages: false (default)

	isPinned	notPinned
isVulnerable	keep	delete
notVulnerable	keep	keep

Do we care about scanning pinned images if deletePinnedImages: false? Since we'll be keeping them either way...
- As of now, I don't believe we have any reporting of CVEs, so scanning images we know we won't delete won't help with anything, but could keep us open to add to reports if we do so in the future (surface vulnerable images #356). I'll look into how/if we do special case handling for deleteEOL and other similar flags.

Signed-off-by: Fabian Gonzalez <[email protected]>

inFocus7 · 2023-12-02T13:41:28Z

api/v1/zz_generated.conversion.go

-// Convert_unversioned_Image_To_v1_Image is an autogenerated conversion function.
-func Convert_unversioned_Image_To_v1_Image(in *unversioned.Image, out *Image, s conversion.Scope) error {
-	return autoConvert_unversioned_Image_To_v1_Image(in, out, s)
-}
-


will look into why this got removed, i did hit some issues when I last generated code due to not being able to covert unversioned...Pinned to <version>...Pinned as the Pinned field did/does not exist in versioned generations.

inFocus7 · 2023-12-02T13:42:22Z

go.mod

@@ -151,6 +151,7 @@ replace (
 	k8s.io/component-helpers => k8s.io/component-helpers v0.26.11
 	k8s.io/controller-manager => k8s.io/controller-manager v0.26.11
 	k8s.io/csi-translation-lib => k8s.io/csi-translation-lib v0.26.11
+	k8s.io/dynamic-resource-allocation => k8s.io/dynamic-resource-allocation v0.26.11


hit issues locally due to the infamous

unknown revision <kubernetes-library> v0.0.0

requiring me to pin this.

inFocus7 · 2023-12-02T13:46:06Z

pkg/remover/helpers.go

@@ -75,6 +76,12 @@ func removeImages(c cri.Remover, targetImages []string) (int, error) {
 				continue
 			}

+			// TODO - figure out why is imgDigestOrTag used instead of imageID when it's called "idToImageMap" (copied usage from isExcluded).
+			if !removePinned && util.IsPinned(imageID, idToImageMap) {


The util.IsExcluded function does a idToImageMap[imgDigestOrTag], although I would expect it to use the imageId as the key..

The reason is a frustrating one: Docker's ImageID is different from Containerd's Digest. Newer versions of docker can use containerd to manage storage & metadata of images.

But without that feature, docker reports the sha256 digest of the Image Config as the ImageID. On the other hand, containerd's Image Digest is the sha256 digest of the image manifest. Of the two, the image digest is more stable, because the name of the image is included in the Image Config. Thus, a change to the name of an image (without any changes to the image itself) will result in a change to the ImageID.

Thus we use the digest as the hash key for the Set of images we build. Each distinct digest is a distinct image, full stop; the same is not true for ImageIDs: tag an existing image with a new name and you have one distinct image with two distinct ImageIDs.

The CRI is kind of in an in-between state. It was developed to provide an interface and had to work with older clusters using dockershim and newer clusters using containerd. As such, it takes the ImageID into account more than it should.

Using trivy to scan by ImageID doesn't work. Trivy scans the containerd image store by creating a containerd client (from the containerd library) and querying it directly. Since containerd doesn't manage images by docker's ImageID, it can't provide any image information to trivy for the scan if it's looking for it by ImageID.

We want to scan and remove by content as much as possible, not by name. We use the image name + tag as a backup in the event that the call to the CRI's ListImages returns an image with no digest information. This happens surprisingly frequently because of the ImageID cruft in the CRI implementation.

pmengelbert

Thank you so much for the draft PR, and apologies for taking so long to review. I've suggested some changes to the general approach, and I will be happy to clarify if anything isn't clear.

pmengelbert · 2024-01-17T14:24:09Z

api/unversioned/imagejob_types.go

@@ -25,6 +25,7 @@ type Image struct {
 	ImageID string   `json:"image_id"`
 	Names   []string `json:"names,omitempty"`
 	Digests []string `json:"digests,omitempty"`
+	Pinned  bool     `json:"pinned,omitempty"`


You will want to make a simultaneous identical change to v1 and unversioned. We keep unversioned synchronized with the latest api version for each type. Currently:

imagejob_types.go -> v1 == unversioned

imagelist_types -> v1 == unversioned

eraserconfig_type -> v1alpha3 == unversioned

pmengelbert · 2024-01-17T14:31:55Z

pkg/collector/collector.go

@@ -23,6 +23,7 @@ var (
 	enableProfile = flag.Bool("enable-pprof", false, "enable pprof profiling")
 	profilePort   = flag.Int("pprof-port", 6060, "port for pprof profiling. defaulted to 6060 if unspecified")
 	scanDisabled  = flag.Bool("scan-disabled", false, "boolean for if scanner container is disabled")
+	scanPinned    = flag.Bool("scan-pinned", false, "boolean for if scanner container should scan pinned images")


We are making an effort to stop using cli args to control the container applications. We are using the configmap instead. In the code that spawns this pod (imagecollector_controller.go or imagejob_controller.go), we can set an environment variable based on the configmap value.

For more information as to why we are moving away from cli args:
#446

pmengelbert · 2024-01-17T15:31:41Z

pkg/remover/helpers.go

@@ -75,6 +76,12 @@ func removeImages(c cri.Remover, targetImages []string) (int, error) {
 				continue
 			}

+			// TODO - figure out why is imgDigestOrTag used instead of imageID when it's called "idToImageMap" (copied usage from isExcluded).
+			if !removePinned && util.IsPinned(imageID, idToImageMap) {


The reason is a frustrating one: Docker's ImageID is different from Containerd's Digest. Newer versions of docker can use containerd to manage storage & metadata of images.

But without that feature, docker reports the sha256 digest of the Image Config as the ImageID. On the other hand, containerd's Image Digest is the sha256 digest of the image manifest. Of the two, the image digest is more stable, because the name of the image is included in the Image Config. Thus, a change to the name of an image (without any changes to the image itself) will result in a change to the ImageID.

Thus we use the digest as the hash key for the Set of images we build. Each distinct digest is a distinct image, full stop; the same is not true for ImageIDs: tag an existing image with a new name and you have one distinct image with two distinct ImageIDs.

The CRI is kind of in an in-between state. It was developed to provide an interface and had to work with older clusters using dockershim and newer clusters using containerd. As such, it takes the ImageID into account more than it should.

Using trivy to scan by ImageID doesn't work. Trivy scans the containerd image store by creating a containerd client (from the containerd library) and querying it directly. Since containerd doesn't manage images by docker's ImageID, it can't provide any image information to trivy for the scan if it's looking for it by ImageID.

We want to scan and remove by content as much as possible, not by name. We use the image name + tag as a backup in the event that the call to the CRI's ListImages returns an image with no digest information. This happens surprisingly frequently because of the ImageID cruft in the CRI implementation.

pmengelbert · 2024-01-17T15:34:52Z

pkg/remover/helpers.go

@@ -8,7 +8,7 @@ import (
 	util "github.com/eraser-dev/eraser/pkg/utils"
 )

-func removeImages(c cri.Remover, targetImages []string) (int, error) {
+func removeImages(c cri.Remover, removePinned bool, targetImages []string) (int, error) {


Instead of passing a boolean flag, just require that the caller remove any pinned images from targetimages before invoking this function. It won't require any changes here but will require them elsewhere.

If the collector is turned on, filter them out during the collector stage. If the collector (and therefore scanner) is turned off, filter them out just prior to removal.

inFocus7 added 3 commits September 3, 2023 12:49

include pinned image information

efcc951

Signed-off-by: Fabian Gonzalez <[email protected]>

Add extra comments on areas we can manage Pinned images

bdcd2e2

Signed-off-by: Fabian Gonzalez <[email protected]>

Merge branch 'main' into feat/add-cri-pinned-image-filter

c30da36

inFocus7 marked this pull request as draft November 29, 2023 23:42

inFocus7 added 2 commits December 2, 2023 08:39

Move Pinned check + removal to the collector and remover pods

7f95f58

Signed-off-by: Fabian Gonzalez <[email protected]>

remove Pinned config from scanner

d2d8b54

Signed-off-by: Fabian Gonzalez <[email protected]>

inFocus7 commented Dec 2, 2023

View reviewed changes

sozercan requested review from pmengelbert and ashnamehrotra December 19, 2023 18:46

pmengelbert requested changes Jan 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for "pinned" images #920

feat: add support for "pinned" images #920

inFocus7 commented Nov 29, 2023 •

edited

Loading

inFocus7 Dec 2, 2023

inFocus7 Dec 2, 2023

inFocus7 Dec 2, 2023

pmengelbert Jan 17, 2024

pmengelbert left a comment

pmengelbert Jan 17, 2024

pmengelbert Jan 17, 2024

pmengelbert Jan 17, 2024

pmengelbert Jan 17, 2024

feat: add support for "pinned" images #920

Are you sure you want to change the base?

feat: add support for "pinned" images #920

Conversation

inFocus7 commented Nov 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmengelbert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inFocus7 commented Nov 29, 2023 •

edited

Loading