Assume/Abort in sub Steps creates Passed Scenario in the report with odd count of Steps/Skipped

[lunmag]

Describe the bug

Hi!

Related to #299 in a way...
When an Assume/Abort - statement is made to stop the Scenario if needed, the Scenario-status in the @After-part (scenario.getStatus()) says SKIPPED.

However, I only get a Skipped Scenario in the report if there are no (zero) successful Steps completed.
If one or more Steps are completed before the Assume/Abort takes place, the Scenario is Passed in the Report.
Also, it does not calculate the skipped steps correctly, well , I added two snapshots for a look.

To Reproduce
Steps to reproduce the behavior:

Create a Feature with at least two Steps

Perform a
assumeTrue(false,"Oh No");
or
org.junit.jupiter.api.Assumptions.abort("stopping after a few steps");

at
Test 1/ the first Step of the Feature
Test 2/ the last Step of the Feature

1 => Skipped
2 => Passed

Expected behavior
I'd assume, :-) , that since the scenario.getStatus() reports SKIPPED, I'd get Skipped scenarios in both cases in the Report.

Attachments
2 Scenarios with Assume/Abort done in sub steps of their Features.

** Note **
Cucumber 7.11.1, JUnit 5.9.3

[bischoffdev]

Thank you for reporting, I will look into this.

[bischoffdev]

A scenario is considered skipped when all steps are skipped. When there are passing steps before it is considered passed. This is according to the official Cucumber guidelines.

(see issue #283)

Could you send me the JSON result file of your use case so I can check this better?

[lunmag]

Hi!
Thanks for quick handling!
Added .txt to JSON (It contains an image too, but should be ok)

Console-Out.txt
cucumber.json.txt

[bischoffdev]

Your example shows a passed step and then a skipped one. According to the Cucumber specs, this is considered passed.

[bischoffdev]

Also the step counts seem to be correct here.

[mpkorstanje]

@bischoffdev the state of a scenario is the most severe status of it's steps. But I didn't look at the json, maybe I'm reading your comment wrong.

https://github.com/cucumber/cucumber-jvm/blob/main/cucumber-core/src/main/java/io/cucumber/core/runner/TestCaseState.java#L54

[lunmag]

I just tried Allure-reporting and got this for an Assume()/Abort() operation:
(w/o any claims of what is wrong or right, just as info)

[bischoffdev]

@mpkorstanje this is interesting. What does "the most severe" status mean concretely?

I know the status constants are

PASSED,
SKIPPED,
PENDING,
UNDEFINED,
AMBIGUOUS,
FAILED,
UNUSED

Would that mean that any step status that is lower down the list than any other steps in the scenario determines the scenario status? If so, this is new info. I remember getting my initial info from the Cucumber Slack.

E.g.

Step 1: passed
Step 2: skipped
Step 3: skipped

=> Scenario is skipped?

According to my former info, this would be considered passed.

[mpkorstanje]

Yes, this specific example the scenario would be skipped.

Not sure what you got from Slack. I do remember this conversation:

https://cucumberbdd.slack.com/archives/C60TKS3SL/p1532961939000185?thread_ts=1532961939.000185&cid=C60TKS3SL

I think you interpreted the comment about scenario no. 3 as me saying the scenario passed. Rather the scenario is skipped, the execution of cucumber is passed (i.e. a zero exit code if this was the only scenario executed).

It is worth nothing that strict mode is default now and non-strict has been removed. So things are much simpler now.

[bischoffdev]

Ah cool! So I will refine the state handling again and adjust it to the official way. Thanks for your input!

[bischoffdev]

Ready to release in v3.3.0