falcoctl cosign signature verification

[maxgio92]

This issue tracks the feature proposal as for #132 to provide commands to:

• sign
• verify

in order to guarantee the authenticity and integrity of the Falco artifacts distributed by falcoctl.

Proposal

As falcoctl followed a design where OCI registries are used as storage, for general OCI artifacts, it can make sense to store every artifact, signatures included, as OCI artifacts in registry.

Additional context

• OCI artifacts
• OCI artifact manifest as part of the image specification
  • recent work
• ORAS artifact spec will soon support only OCI artifact
• Docker hub supports OCI artifact

[maxgio92]

Hi @falcosecurity/falcoctl-maintainers I've some points in mind and related questions:

• Compatibility: do we want to provide compatibility with current supply chain security tools (e.g. cosign)? Do we want instead to provide compatibility with only falcoctl verify feature? (either implicit or exposed via explicit command)

• Storage specification: do we want to stick to ORAS artifact? Do we want instead to leverage OCI artifact? (more here (of which manifest) and here)

[maxgio92]

Update: there's work in progress from ORAS and OCI to unify the artifact specs (OCI and ORAS) and consequent support from ORAS - of Go library and cli have already release candidates.

Sources:

• https://oras.land/blog/oras-0.15-a-fully-functional-registry-client/#whats-next-for-oras
• https://github.com/oras-project/oras-go/milestone/10

[FeynmanZhou]

@maxgio92 I like this proposal and I believe the next release of ORAS CLI and library could be the appropriate solution to support falcoctl sign/verify commands. The reason is that we are also working on similar things in Notary v2 (Notation) for signing and verifying OCI artifacts. Currently, Notation is using ORAS-go. The signatures are also persisted as ORAS artifact manifest.

[FeynmanZhou]

@maxgio92 BTW, Notation-go provides APIs for signing and verifying OCI artifacts. Currently, notation-go is also towards a more extensible framework for easy integration, and we are preparing for the API doc.
Maybe you can investigate if Falcoctl CLI can leverage the Notation API (Notation-go library) in the sign/verify commands. Let me know if you have any questions. Thanks.

[loresuso]

Hello there, and thank you for this discussion!
I am taking a deeper look at notation-go. It seems to me that it really could be the way to go!
I'll let you know once I will have a stronger opinion or further questions!

[FeynmanZhou]

Hello there, and thank you for this discussion! I am taking a deeper look at notation-go. It seems to me that it really could be the way to go! I'll let you know once I will have a stronger opinion or further questions!

Thank you.

[leogr]

cc @cpanato

[maxgio92]

Thank you so much @FeynmanZhou!

[loresuso]

Ok, my two cents on this. I know that for @FeynmanZhou this could be obvious, but I am new to these topics so I will write quite a bit, just to have a validation of what I have understood and find the right way to solve the problem.

We are in front of some alternatives, so let's start from the first one.

The first option we have is to put the signature inside one of the layers of the manifest. I believe that Helm did something similar in the past. I think we all agree that this approach is not optimal, since the signature is not part of the artifact and we may want to have a way to extend the artifact even after we pushed it (classic examples, SBOMs or scan results) and without modifying the digest, so this is the reason why I would not use this approach.

The second approach is cosign. Basically, cosign solves the problem of introducing the signature by creating a new artifact that is tagged with the hash of the artifact it is referring to plus the .sig suffix, as described here. For sure, this works, but to me it sounds a bit like a hack. We are free to do whatever we wants with tags, but I don't think they were meant to establish links between artifacts, and this is why I don't like this approach. Furthermore, the artifact info command of falcoctl will be polluted with all the signatures, and we have to write some more hacky code to filter them out.

Things get really better with notation-go and oras, especially in its latest rc-4 version. Basically, using the subject field of the artifact manifest, we are able to create links between artifact at the manifest level, without introducing weird tags to solve the problem. For who doesn't know, this is how notation store a signature referring to an image:

{
  "mediaType": "application/vnd.cncf.oras.artifact.manifest.v1+json",
  "artifactType": "application/vnd.cncf.notary.v2.signature",
  "blobs": [
    {
      "mediaType": "application/jose+json",
      "digest": "sha256:ec0ac89dde1eba1f73af138ca4819b5a28b7d36c99426d495f33e0a2328c8953",
      "size": 2117
    }
  ],
  "subject": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "digest": "sha256:d7eef7bbe23819225992666a81cd6134f5f80a490c9b78e435b94bd2ba968a4f",
    "size": 942
  },
  "annotations": {
    "io.cncf.oras.artifact.created": "2022-10-31T14:27:16Z"
  }
}

Plus, when the registry will be capable, the Referrers API can be used, that can enable us to discover and get a list of all the artifacts that points to a given one, so that the signature can be retrieved and verified. This for sure is the cleanest way to solve the problem (thank you ORAS authors for doing so!).

Now the problem is that our registry (GHCR in this case), does not yet implement the Referrers API as far as I know. So two questions:

• Do you know if it can be enabled somewhere, or is it really not implemented for the moment?
• if it's not implemented, how long do you think it will take to have the feature shipped? do you have some contacts with the people working on GHCR?

By the way, I have seen that we can also take advantage of oras-go v2.0.0-rc.4. One thing that I really like is that we can already start implementing this because that version of oras-go fallbacks to tag-based approach similar to what cosign was doing to link the signature to the artifact. Can you guys elaborate more on how we can do the transition from the tag-based approach to the subject + referrers approach? We will need to delete the image index containing all the manifests that are pointing the reference artifact, and I am a little bit concerned about this because from my experiments it seems that GitHub does not implement the delete API.

Lastly, it would be really nice to see a bump to oras-go v2.0.0-rc.4 in notation-go, just to try this out and see it in action before implementing it in falcoctl!

Hope the conclusions I made are right and that you can answer my questions, thanks!

[FeynmanZhou]

@leogr
You digged into Notation and ORAS deeply and your feedback is also valuable to us. I think we are on the same page.
Let me list your questions below and follow up:

Do you know if it can be enabled somewhere, or is it really not implemented for the moment?

As I know, the OCI Distribution-spec just cut the v1.1.0-rc1 in Sep. I am not sure if GHCR will adopt the rc.1 now.

if it's not implemented, how long do you think it will take to have the feature shipped? do you have some contacts with the people working on GHCR?

I can ask my colleagues to confirm.

Can you guys elaborate more on how we can do the transition from the tag-based approach to the subject + referrers approach?

@shizhMSFT @Wwwsylvia @qweeah Could you pls help to confirm if this transition is already supported by ORAS-go v2.0.0-rc.4?

Last, it would be really nice to see a bump to oras-go v2.0.0-rc.4 in notation-go.

Yes, we are toward the same goal! It was planned in notation-go and hopefully it will bump in Nov. See notaryproject/notation-go#136

cc @sajayantony @SteveLasker @toddysm

[toddysm]

I am trying to get a timeline from GHCR for when they will be able to support OCI 1.1 spec. Will update you if I learn something soon.

[shizhMSFT]

Can you guys elaborate more on how we can do the transition from the tag-based approach to the subject + referrers approach?

@shizhMSFT @Wwwsylvia @qweeah Could you pls help to confirm if this transition is already supported by ORAS-go v2.0.0-rc.4?

According to the upgrade procedures of the distribution-spec, the remote registry may do the transition at the server side. In the case of the remote server not doing the transition, the transition still can be done at the client side using oras-go v2.0.0-rc.4 or above.

[maxgio92]

Thanks @shizhMSFT @toddysm @FeynmanZhou @loresuso !!!

I think individual implementations of generic artifacts (signatures, SBOM, attestation, etc.) storage have been around before the introduction of what then OCI and ORAS communities just contributed and worked to unify into OCI as standard (artifact spec and reference API to link artifacts).

Now that registries are working too to stick to the fresh specification (e.g. Docker Hub) I believe - also becasue falcoctl is (re)fresh too - can make sense to follow OCI through the notation library. My 2 cents

[maxgio92]

An important point: do we want to include both the signin and verification features inside falcoctl?

Or, do we want to include verification only and instead delegate signin to external tools (e.g. supposed to run in CI alongside a falcoctl artifact push)?

In both cases, a decision on the API is needed IMO.

Moreover, as notaryproject/notation-go#136 will be completed I'd like to work on this and then propose it in draft if you agree.

[SteveLasker]

hi @maxgio92, you hit the primary point that we’re working towards; implementing standards (cose for singing), so any project, product or service can implement signing or verification across those standards.

[leogr]

Hey folks,

I've read this interesting discussion and some random thoughts came to my mind:

• IMO, users should be free to sign things using their preferred tool,
• so falcoctl should only support verification;
• moreover falcoctl may support more than one standard (why should we stick just with one?).

As of today, Cosign and Notation seem equivalent to me, and I guess they both will converge to the OCI distribution-spec (sigstore confirmed their intention here. We may think of supporting both, IMO.

Finally, I believe it is worth bringing this discussion up to the whole Falcosecurity organization since there may be other sub-projects that want to sign their artifacts (likely, we want to use the same standards/tools across the entire org). Here is an example of a sub-project using cosign 👉 falcosecurity/falcosidekick#302

cc @falcosecurity/core-maintainers

[leogr]

The Helm way 👉 falcosecurity/charts#427

[maxgio92]

Thanks @leogr. I see the point of the responsibility of verifying only the authenticity of the artifacts. And I agree as we discussed in the meantime.

I agree with the fact that all the implementations will converge to the OCI specification for artifacfs, for distribution. (as a detail, then we may want to discuss the encoding and signing algorithms to support, e.g. Notation supports JOSE and COSE).

I also agree with opening the discussion to the community in order to start talking about general signing at Falco organization-level, so that we make consistent choices on the architectures.

[maxgio92]

As of the discussion of the today's supply chain security working group [1], I think we are on the same page on the responsibilities of the tools for the Falco OCI artifact's signature and signature verification:

• signing: plugins and rules. The CI as code will implement generation and distribution of the signature of the artifact, as OCI artifact.
• signature verification: falcoctl. The tool will implement a command to verify the signature of the Falco artifact, which is distributed as OCI artifact.

---

[1] References:

• Slack channel
• Github project
• HackMD meeting notes

[LucaGuerra]

I believe we can use this issue for the verify feature of Falcoctl while keeping two separate ones for rules and plugins. WDYT?

[maxgio92]

I agree @LucaGuerra