Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate all chains on PRs which modify the validation test code / expectations #689

Open
alfonso-op opened this issue Nov 18, 2024 · 1 comment
Labels
A-validation Area: validation

Comments

@alfonso-op
Copy link

alfonso-op commented Nov 18, 2024

We need to strengthen the validations that run on the superchain-registry to catch breaking changes such that the validate all chains CI job is triggered for all PR changes with the one exception being when a new chain is added.

See George's comment below for more details.

@geoknee
Copy link
Collaborator

geoknee commented Nov 18, 2024

The problem is rather that we aren't validating all chains in all the cases that we should be. We are being too eager to skip validation on all chains. On PRs we already skip validating any chain which didn't have its TOML files changed. But if the PR changes the standard config TOML files, or the Go files which compose the validation suite itself, this could change the validation status of any chain in the repo.

I think the rule should be that if you only add a chain, that is add the TOML files for a new chain, and don't touch anything else, then only your chain gets validated. Otherwise, all chains get validated. This preserves the nice property we already enjoy, that if a PR only adds a new chain, it won't get blocked on validation failures from other chains.

@geoknee geoknee changed the title Only validate all chains when needed Validate all chains on PRs which modify the validation test code / expectations Nov 18, 2024
@geoknee geoknee added the A-validation Area: validation label Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-validation Area: validation
Projects
None yet
Development

No branches or pull requests

2 participants