-
Notifications
You must be signed in to change notification settings - Fork 450
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce --sim.exclude
flag for excluding suites / tests
#737
Comments
Will take a look into adding this over the next couple days. I think should be simple enough:
|
hey @spencer-tb, any progress on this? |
Sorry for the delay here, I started on this locally but never quite finalized everything. Some updates nonetheless! I'm currently using this branch: Usage and semantics so far are below:
Note that exclusion is checked first. So if there is an overlap between inclusion and exclusion the test or suite will always be skipped. I have the functionality working for some simulators, i.e Nonetheless the first feature should work for the |
I will update you once I have finalized everything to work across all simulators. :) |
While
--sim.limit
is useful, we have been setting up some hive suites to run as github actions, and some test suites are not yet applicable, and should be excluded.For example,
snap
in thedevp2p
simulator is not yet applicable because we do not support thesnap
subprotocol yet. Thesim.limit
does accept a regular expression, but the golang regular expression library does not support negative lookaheads / lookbehinds. To exclude a suite, we have to explicitly include every other suite. To exclude a test, we have to explicitly include every other test.To exclude
snap
, we have to run--sim.limit
for each suite we want to include indevp2p
, which is luckily onlydiscv4
andeth
.I think it makes sense to allow both
--sim.limit
and--sim.exclude
for this use case, for example:Because reth does not process block hash announcements, this would exclude the right test, while allowing us to run all other tests in
eth
. Withoutsim.exclude
, we would have to enumerate all tests we want to include using thesim.limit
flag.Does this make sense?
The text was updated successfully, but these errors were encountered: