Misidentification of relevant coverage report causes tests to be thrown away

[j-ro]

There is a bug when matching a test run with the targeted file's coverage report where, if there is more than one file with the same name endings (different paths), the matched coverage report will not be the file we are targeting, causing an erroneous report that coverage did not increase, and thus causing the tests to be thrown out.

For example, my coverage.xml file might look like this (this is a Ruby/Rails project):

[snip]
        <class name="has_report_invite" filename="app/models/concerns/has_report_invite.rb" line-rate="0.32" branch-rate="0" complexity="0">
          <methods/>
          <lines>
            <line number="1" branch="false" hits="1"/>
            <line number="2" branch="false" hits="1"/>
            <line number="4" branch="false" hits="1"/>
            <line number="5" branch="false" hits="0"/>
            <line number="8" branch="false" hits="1"/>
            <line number="9" branch="false" hits="0"/>
            <line number="12" branch="false" hits="1"/>
            <line number="13" branch="false" hits="0"/>
            <line number="16" branch="false" hits="1"/>
            <line number="17" branch="false" hits="0"/>
            <line number="20" branch="false" hits="1"/>
            <line number="21" branch="false" hits="0"/>
            <line number="22" branch="false" hits="0"/>
            <line number="24" branch="false" hits="0"/>
            <line number="25" branch="false" hits="0"/>
            <line number="29" branch="false" hits="0"/>
            <line number="32" branch="false" hits="0"/>
            <line number="33" branch="false" hits="0"/>
            <line number="34" branch="false" hits="0"/>
            <line number="35" branch="false" hits="0"/>
            <line number="39" branch="false" hits="0"/>
            <line number="44" branch="false" hits="0"/>
          </lines>
        </class>
        <class name="report_invite" filename="app/models/report_invite.rb" line-rate="0.36" branch-rate="0" complexity="0">
          <methods/>
          <lines>
            <line number="1" branch="false" hits="1"/>
            <line number="2" branch="false" hits="1"/>
            <line number="4" branch="false" hits="1"/>
            <line number="5" branch="false" hits="0"/>
            <line number="8" branch="false" hits="1"/>
            <line number="9" branch="false" hits="0"/>
            <line number="10" branch="false" hits="0"/>
            <line number="11" branch="false" hits="0"/>
            <line number="13" branch="false" hits="0"/>
            <line number="14" branch="false" hits="0"/>
            <line number="15" branch="false" hits="0"/>
            <line number="17" branch="false" hits="0"/>
            <line number="20" branch="false" hits="0"/>
            <line number="23" branch="false" hits="1"/>
            <line number="24" branch="false" hits="0"/>
            <line number="25" branch="false" hits="0"/>
            <line number="26" branch="false" hits="0"/>
            <line number="27" branch="false" hits="0"/>
            <line number="28" branch="false" hits="0"/>
            <line number="29" branch="false" hits="0"/>
            <line number="34" branch="false" hits="1"/>
            <line number="35" branch="false" hits="0"/>
            <line number="38" branch="false" hits="1"/>
            <line number="39" branch="false" hits="1"/>
            <line number="42" branch="false" hits="1"/>
            <line number="43" branch="false" hits="0"/>
            <line number="46" branch="false" hits="1"/>
            <line number="47" branch="false" hits="0"/>
          </lines>
        </class>
[snip]

As you can see, there are two files that end in "report_invite" in this report. As they are related to each other, both are exercised when the one target test file (in this case, "spec/models/report_invite_spec.rb") is run. So they both show up in this report.

Here is the command I'm using:

cover-agent \
  --source-file-path "app/models/report_invite.rb" \
  --test-file-path "spec/models/report_invite_spec.rb" \
  --code-coverage-report-path "coverage/coverage.xml" \
  --test-command "docker-compose exec -e RAILS_ENV=test web rspec spec/models/report_invite_spec.rb" \
  --additional-instructions "" \
  --included-files spec/factories/report_invites.rb app/mailers/report_invite_mailer.rb lib/tasks/report_invites.rake app/models/concerns/has_report_invite.rb  \
 --desired-coverage 100 \
  --model "gpt-4o"

As you can see from the above, the target file is "app/models/report_invite.rb". And if you manually look at the coverage report, you can see that it is 36% covered right now.

However, cover-agent reports that it is only 32% covered:

Streaming results from LLM model...
yaml
language: ruby
testing_framework: rspec
number_of_tests: 1
test_headers_indentation: 4


Streaming results from LLM model...
yaml
language: ruby
testing_framework: rspec
number_of_tests: 1
relevant_line_number_to_insert_tests_after: 8
relevant_line_number_to_insert_imports_after: 1


2024-12-18 11:06:22,710 - cover_agent.UnitTestValidator - INFO - Running build/test command to generate coverage report: "docker-compose exec -e RAILS_ENV=test web rspec spec/models/report_invite_spec.rb"
2024-12-18 11:06:35,309 - cover_agent.UnitTestValidator - INFO - Initial coverage: 31.82%
2024-12-18 11:06:35,315 - cover_agent.CoverAgent - INFO - Current Coverage: 31.82%
2024-12-18 11:06:35,315 - cover_agent.CoverAgent - INFO - Desired Coverage: 100%

Digging in, you can see it is matching to the "app/models/concerns/has_report_invite.rb" file, not the actual target, which has that percentage covered.

Because cover-agent is reading the wrong file for its coverage report, it throws out tests it generates as not increasing coverage, causing a lot of wasted time and expensive tokens.

I'm guessing the issue is with this line: https://github.com/qodo-ai/qodo-cover/blob/main/cover_agent/CoverageProcessor.py#L131

The fix is probably to try and match on the path, rather than just the partial file name, as it assumes all file names in a coverage report are unique (or don't share endings), when that is only true in a given path.

If I get some time I'll try and make a PR here, but wanted to post this bug first to see if folks had feedback on how to approach it.