SJIP: Clarify Rules Enforcement

[Jammabeans]

Description

Clarify Rules Enforcement

Judging Guidelines PR

sherlock-protocol/sherlock-v2-docs#41

Rationale

Today, I would like to raise a discussion on how rules should be interpreted and whether there are any hidden requirements that are not clear to understand and can only be known in the mind of the judge.

I suggest there are 2 paths of logic, and only one of them can be correct.

1. Rules are meant to be clear and must be followed as written.
2. Rules are fluid, and judges can add or remove language as seen fit to reach the desired outcome.

As such I propose the following update me made to the judging rules:

"All issues must be judged on the plain English meaning as outlined in these rules. Judges do not have the authority to add or remove elements and must rule based on the published rules as stated during the contest period."

Relevant Issue Discussions

To analyze this topic in a real-world case involving duplication rules with a target and a duplicate issue:

sherlock-audit/2024-07-sense-points-marketplace-judging#147

Here we can see a case of the target issue being validated, but then a potential duplicate being rejected based on rules being fluid, with additional words being added to the rules to justify not making the duplicate issue a valid duplicate. The newly added words are "would this issue still be valid if the main one wasn't submitted?"

Although the defender of issue #52 makes a clear and concise argument on how the rules as published support making them duplicates, this is rejected by the judge based on a new requirement not listed in the rules.

This creates a situation where instead of asking if these issues are alike, we are now asking whether, even if they are very alike, there is any little thing to pick at to make it invalid.

This is a breaking point. If we follow the logic in Path 1, then we should be looking for likeness in duplicates. If we follow Path 2, then we can pick apart potential duplicates with an attack mindset.

As it stands, current enforcement of the rules is only making the rules more unclear, and we should address this topic as a community to achieve the outcome of having clear rules.

[IllIllI000]

The Judging Guidelines PR section should be the PR for the change you'd like to make to the rules, so we all have something specific to approve/disapprove.

This comment seems to be the reason for invalidation, and specifically calls out the fact that there aren't valid medium impacts. It reads to me as though the comment about "if the main one wasn't submitted", was to make clear that the medium impact was in fact meant to be a medium impact according to the Sherlock rules specifically, not some generic meaning of medium impact as appeared to be argued here. In other words, it doesn't appear to me that there are shadow rules being applied for this specific finding.

[Jammabeans]

Impacts are things like stuck tokens, stolen funds, locked contracts, loss of gas, no event. These are impacts. Whether or not a user mistake happens is not a factor of impact; it is a factor of attack path. In the target case, the judge is applying factors related to Attack Path as a requirement for Impact.

As for "The Judging Guidelines PR section should be the PR for the change you'd like to make to the rules, so we all have something specific to approve/disapprove."

I would suggest adding the following:

All issues must be judged on the plain English meaning as outlined in these rules. Judges do not have the authority to add or remove elements and must rule based on the published rules as stated during the contest period.

[IllIllI000]

The rule says Identify at least a Medium impact, so what are you saying is the plain English definition of Medium, since you say you listed impacts rather than Medium impacts?

[Jammabeans]

I'm saying that the phrase "Identify at least a Medium impact" intentionally does not use the word "valid," whereas the next requirement does use the word "valid." Plain English in this case would mean one has to be valid, the other does not, and may be out of scope.

Mostly, it's about misunderstanding what exactly the rules mean. If a plain English requirement is put in place, then any new rules will become much more detailed and easier to understand.

[IllIllI000]

If a plain English requirement is put in place, then any new rules will become much more detailed and easier to understand.

Unless you're willing to also update all the old rules to use whatever your definition of plain English is, I don't think your suggested change will result in better judgements.

VII. List of Issue categories that are not considered valid:
User input validation: User input validation to prevent user mistakes is not considered a valid issue.

Doesn't the above invalidate the finding, regardless of whether the word 'valid' exists in the impact clause?