Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Branch coverage gets ignored (Request: Support for Branch Coverage) #161

Open
draialexis opened this issue Sep 21, 2024 · 6 comments · May be fixed by #228
Open

Branch coverage gets ignored (Request: Support for Branch Coverage) #161

draialexis opened this issue Sep 21, 2024 · 6 comments · May be fixed by #228
Assignees
@EmbeddedDevops1
Labels
enhancement New feature or request

Comments

@draialexis
Copy link
Contributor

In a Java 21 / Spring Boot project using Gradle with JUnit and JaCoCo, I've noticed something.

New unit tests can get thrown away because "coverage did not increase", when in fact the test would increase branch coverage -- if not line coverage.

Here's an example:
(please do ignore the wonky casing and the dummy test that says 'assertTrue(true)')

image

In that example, the case where the collection is null is a different case than when the collection is empty. I believe the extra test is indeed valuable, and should be kept -- since it increases the branch coverage metric.

Besides, if a test were to increase complexity coverage, or method coverage, I would rather Cover Agent not throw it away either: users can then decide which tests they want to keep.

Looking at a JaCoCo report (in .csv here), we can see that those metrics are indeed available:

...CLASS,INSTRUCTION_MISSED,INSTRUCTION_COVERED,BRANCH_MISSED,BRANCH_COVERED,LINE_MISSED,LINE_COVERED,COMPLEXITY_MISSED,COMPLEXITY_COVERED,METHOD_MISSED,METHOD_COVERED
...AbstractService,131,69,11,3,44,25,25,9,18,9
...CrudDatableService,169,0,8,0,49,0,27,0,23,0
...AbstractEntity,13,25,3,3,1,7,4,2,1,2
...

I didn't see any issue templates here, so I wrote something basic. Please feel free to let me know if you do want me to adhere to specific guidelines, and I'll do what I can. Please let me know if you want any extra information. Thanks for putting this out there, I've envoyed testing it out so far
@EmbeddedDevops1
Copy link
Collaborator

@draialexis This is awesome. Thanks so much for opening up this issue. Branch coverage (and a myriad of other types of coverage) is something we absolutely want in Cover Agent.

We'd need to support it for the existing supported coverage report types (right now just Cobertura and JaCoCo) but it's definitely something we'd like to add.

If you'd like to take a stab at it you could create the parsing functions in cover_agent/CoverageProcessor.py and see how it fairs. After that I could integrate it into cover_agent/UnitTestGenerator.py or you could take try as well.

Regarding the template: This was great. Thanks again for contributing to this community's project!

@EmbeddedDevops1 EmbeddedDevops1 self-assigned this Sep 26, 2024
@EmbeddedDevops1 EmbeddedDevops1 added the enhancement New feature or request label Oct 6, 2024
@EmbeddedDevops1 EmbeddedDevops1 changed the title Branch coverage gets ignored [Request: Support for Branch Coverage] Branch coverage gets ignored Oct 8, 2024
@EmbeddedDevops1 EmbeddedDevops1 changed the title [Request: Support for Branch Coverage] Branch coverage gets ignored Branch coverage gets ignored (Request: Support for Branch Coverage) Oct 8, 2024
@draialexis
Copy link
Contributor Author

draialexis commented Oct 27, 2024 •
edited
Loading

Thanks for the response!

I'm not super proficient with Python and I don't have a lot of free time these days, so I don't think I'll be taking a stab at it myself.

Initially I was thinking about trying, because I thought I would be able to simply substitute line coverage for branch coverage, rename some stuff accordingly, and send my PR.

That's because branch coverage subsumes statement coverage (cf. ISTQB Certified Tester - Foundation Level Syllabus v4.0, Section 4.3.2. Branch Testing and Branch Coverage)

But looking more closely at examples from different coverage reports like JaCoCo and cobertura, it's possible to see 100% branch coverage and <100% line / statement / instruction coverage metrics. This indicates to me that they use specific approaches in the ways they calculate those metrics, making it so branch coverage no longer subsumes statement coverage.

If I'm correct, then that means the task at hand is no longer to substitute one metric for another, but to switch from a 1-metric paradigm (line coverage) to an n>=1-metric paradigm -- I haven't read all the code, but I'll bet that it's not going to be trivial

@EmbeddedDevops1 EmbeddedDevops1 linked a pull request Oct 28, 2024 that will close this issue
Support for Branch Coverage #196
Closed
@EmbeddedDevops1
Copy link
Collaborator

@draialexis Got a POC up for branch coverage. Want to give it a test? #196

@draialexis
Copy link
Contributor Author

@EmbeddedDevops1 thanks, I’d gladly give it a spin! I’ll try to fit that in tomorrow.

@draialexis
Copy link
Contributor Author

Just in case someone stumbles upon this issue and misses the PR (where I gave feedback), here it is: #196

@EmbeddedDevops1 EmbeddedDevops1 linked a pull request Nov 19, 2024 that will close this issue
Initial commit of rebased branch coverage. #228
Open
@EmbeddedDevops1
Copy link
Collaborator

Just in case someone stumbles upon this issue and misses the PR (where I gave feedback), here it is: #196

Heads up: I re-opened a new PR: #228

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EmbeddedDevops1 EmbeddedDevops1

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
3 participants
@draialexis @EmbeddedDevops1 and others