Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(decl): add test command support #239

Merged
merged 17 commits into from
Nov 14, 2024

Conversation

ekoops
Copy link
Contributor

@ekoops ekoops commented Nov 13, 2024

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind documentation

/kind tests

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area commands

/area pkg

/area events

What this PR does / why we need it:
This PR adds the test command, that allows to verify that Falco produces the expected alerts as a response to the executed tests. The command allows to connect to Falco via gRPC, through both Unix an AF_INET/AF_INET6 sockets.
The underlying infrastructure has been modified as follows:

  • the process chain must be composed at least by one process besides the root process; if no processes are provided in the process chain by the user, a default one is added to it
  • test IDs has been added in order to to enable unique identification of the process chain's leaf process. The test ID is stored in the environment of each process chain's process, except the root process. The root process generates a test ID for each test, in the form <ignorePrefix><testUID>. Each process in the chain but the one before the leaf passes the untouched test ID to the child. The one before the leaf passes the test ID without the <ignorePrefix> to the leaf. In this way, the leaf process is the only one having the test ID equals to <testUID>.

The following components have been added:

  • alert retriever - collects alerts from Falco (the only available implementation uses gRPC)
  • tester - collects alerts from the alert retriever and generates a report

A report is associated with a test (which in turn is associated to a single rule). It contains information regarding (1) the number of alerts successfully matching the expected outcome of the test, and (2) the generated warnings.
A successful match is determined when Falco produces an alert whose fields are equal to the one provided in the expected outcome.
A warning is associated to a single alert in the context of a rule report. A warning is generated if the alert matches the rule name but doesn't match other expected outcome's fields. In the context of a warning, for each expected outcome's field not matching the alert content, a field warning is generated in the report.
If no successful matches and warning are generated during the alerts inspection time interval for a given test, the test is considered failed.

The warning have been added to take into account possible event drops and consequent <NA> values in Falco alerts.

As last features:

  • the process chain executable path field is now optional and, if not provided, is automatically generated by the event-generator
  • the expected output is now called expected outcome (to distinguish them from Falco outputs)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

ekoops and others added 17 commits November 13, 2024 19:14
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Add test ID support to enable unique identification of the process
chain's leaf process. The test ID is stored in the environment of
each process chain's process, except the root process. The root
process generate a test ID in the form <ignorePrefix><testUID>.
Each process in the chain but the one before the leaf passes the
untouched test ID to the child. The one before the leaf passes
the test ID without the <ignorePrefix> to the leaf.

Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Make process context's executable path optional and default it to
a randomly generated path under tmp directory.

Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
The test command allows to run tests and verify that Falco generates
alerts matching the expected outcome. The
--skip-outcome-verification flag can be used to disable outcome
verification.

Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Fallback to test command with no outcome verification.

Signed-off-by: Leonardo Di Giovanna <[email protected]>
Co-authored-by: Aldo Lacuku <[email protected]>
Copy link
Contributor

@FedeDP FedeDP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@poiana
Copy link

poiana commented Nov 14, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ekoops, FedeDP

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@poiana
Copy link

poiana commented Nov 14, 2024

LGTM label has been added.

Git tree hash: 562f3715f80e08b07082e79b1f9fb1d9babd1aa8

@poiana poiana merged commit 80ea746 into falcosecurity:declarative-testing Nov 14, 2024
5 of 6 checks passed
@ekoops ekoops deleted the ekoops/falco-output branch November 15, 2024 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants