Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate rust coverage data #84

Merged
merged 2 commits into from
Oct 30, 2024
Merged

Generate rust coverage data #84

merged 2 commits into from
Oct 30, 2024

Conversation

torymur
Copy link
Contributor

@torymur torymur commented Oct 29, 2024

Closes #33

In order to retain our python coverage and diff-cover tool chain we're:

  • generating code coverage file in lcov format with cargo-tarpaulin
  • creating python coverage file in lcov instead of xml
  • combining these coverage artifacts with lcov into one
  • diff-cover seems to be able to handle combined lcov file

genhtml tool is used instead of coverage html to create html from combined file, which will create a bit different UI, but nothing major seems to change.

From something like this:
python code coverage report

To this:
lcov code coverage report

Also changed in PR:

  • actions-rs action is archived, dtolnay/rust-toolchain now used
  • improved caching:
    • for binaries
    • and in general Swatinem/rust-cache currently is the best way to handle rust cache workflows, it's a "smart caching for rust/cargo projects with sensible defaults"

Copy link
Member

@brandonwillard brandonwillard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds great! Let's add a temporary commit to make sure that it reports missing Rust and Python coverage.

@torymur torymur force-pushed the rust-coverage branch 3 times, most recently from 26a9441 to a500fb7 Compare October 30, 2024 13:01
@torymur torymur added the enhancement New feature or request label Oct 30, 2024
@torymur
Copy link
Contributor Author

torymur commented Oct 30, 2024

Alright, I see good cache hits & usage.

And testing commit failed as expected: https://github.com/dottxt-ai/outlines-core/actions/runs/11594946487/job/32282843486

Overall coverage rate:
  lines......: 35.3% (287 of 812 lines)
  functions..: 28.4% (33 of 116 functions)
Failure. Coverage is below 100%.
-------------
Diff Coverage
Diff: origin/main...HEAD, staged and unstaged changes
-------------
src/json_schema/helpers.rs (0.0%): Missing lines [43](https://github.com/dottxt-ai/outlines-core/actions/runs/11594946487/job/32282843486#step:9:44),47,49-53,55-58
-------------
Total:   11 lines
Missing: 11 lines
Coverage: 0%
-------------

Generated html & uploaded it:
Artifact download URL: https://github.com/dottxt-ai/outlines-core/actions/runs/11594946487/artifacts/2123365492

Seems, we're good to go 🚀

The only issue I see with this, is that we're adding a strong necessity to add potentially vain and unnecessary tests (which exist) just for the sake of coverage, which isn't always justified, considering huge compiler support in Rust, which will check so many things and complain on its own. At least tarpaulin allows to exclude some pieces explicitly. But, let's give it a try and see how it's going to be in action.

I would appreciate a review 🙏 @brandonwillard ?

@torymur torymur marked this pull request as ready for review October 30, 2024 14:34
Copy link
Member

@brandonwillard brandonwillard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! @torymur, feel free to merge it after removing the two test commits.

The only issue I see with this, is that we're adding a strong necessity to add potentially vain and unnecessary tests (which exist) just for the sake of coverage, which isn't always justified, considering huge compiler support in Rust, which will check so many things and complain on its own. At least tarpaulin allows to exclude some pieces explicitly. But, let's give it a try and see how it's going to be in action.

Yeah, as you said, we'll need to add some coverage ignore directives; otherwise, we'll leave it up to the admins to ignore those failures and merge anyway.

@torymur torymur merged commit 1cdf26a into main Oct 30, 2024
17 checks passed
@torymur torymur deleted the rust-coverage branch October 30, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI enhancement New feature or request testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check coverage on Rust code
2 participants