Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] ec should fail if provided config references a non-existent rule or collection #1753

Open
ralphbean opened this issue Jul 8, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@ralphbean
Copy link
Contributor

Describe the Bug

I typo'd something in an EnterpriseContractPolicy but ec didn't fail, which incorrectly led me to believe that I had successfully created the policy I wanted.

Steps to Reproduce

Construct a EnterpriseContractResource with

apiVersion: appstudio.redhat.com/v1alpha1
kind: EnterpriseContractPolicy
spec:
  description: 'An example policy for this issue'
  publicKey: 'k8s://openshift-pipelines/public-key'
  sources:
    - name: Release Policies
      policy:
        - oci::quay.io/enterprise-contract/ec-release-policy:git-0cdcba3@sha256:febe9ada08701fad1cec392dfde8e66cc2408989d1396adb4073af6f646855d1
      config:
        include:
          - "@slsa3"
          - LOL not a rule

Use that policy to evaluate ~any artifact, and it will pass - even though "LOL not a rule" is not a rule

Expected Behavior

I expect ec to fail with "LOL not a rule" not found in any policy source among ... (and then, list the policy sources provided like oci::quay.io/enterprise-contract/ec-release-policy:git-0cdcba3@sha256:febe9ada08701fad1cec392dfde8e66cc2408989d1396adb4073af6f646855d1

Actual Behavior

ec exits successfully without even a warning. (IMO, a hard failure would be best in this situation, not just a warning.)

Screenshots or Terminal Output

If applicable, add screenshots or copy-paste the terminal output to help explain your problem.

Environment Details

❯ ec version                                                  
Version            v0.5.13
Source ID          0527fad71b065e9a0694a30ca70dbf04625ac811
Change date        2024-06-14 20:29:28 +0000 UTC (3 weeks ago)
ECC                v0.1.47
OPA                v0.65.0
Conftest           v0.53.0
Cosign             v2.2.4
Sigstore           v1.8.4
Rekor              v1.3.6
Tekton Pipeline    v0.54.0
Kubernetes Client  v0.29.5
@ralphbean ralphbean added bug Something isn't working triage labels Jul 8, 2024
@zregvart zregvart removed the triage label Jul 9, 2024
@lcarva
Copy link
Member

lcarva commented Jul 12, 2024

NOTE: There is a check to verify that at least one rule was executed. So if @slsa3 wasn't provided, EC would've raised an error.

@simonbaird
Copy link
Member

simonbaird commented Jul 16, 2024

Grooming discussion: We're leaning towards making this a warning rather than a violation since it's possible when preparing to include a new rule or rule collection that you would include it before it become available in the policy source. Also you might want to manage some adhoc style rules with an @adhoc collection.

Additional idea: ec validate policy could perhaps be extended to check whether all the includes actually match something in the data source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants