Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enabled sorting in compressed files #577

Merged
merged 8 commits into from
Dec 18, 2024
Merged

Conversation

pamburus
Copy link
Owner

@pamburus pamburus commented Dec 5, 2024

This allows sorting (--sort, -s) of messages in compressed files of the "gzip", "zstd", "bzip2" and "xz" formats.
The compression format is automatically detected by the file signature.

However, random access through the uncompressed stream in compressed files is not supported, so the compressed files are fully loaded into memory during processing with sorting. This can significantly increase memory consumption in some scenarios, and performance can be noticeably worse than performance with uncompressed files (typically up to 2x).

Copy link

codecov bot commented Dec 5, 2024

Codecov Report

Attention: Patch coverage is 88.97502% with 128 lines in your changes missing coverage. Please review.

Project coverage is 76.55%. Comparing base (edfcda4) to head (4ea7e45).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
src/vfs.rs 69.30% 66 Missing ⚠️
src/input.rs 93.14% 34 Missing ⚠️
src/index.rs 90.47% 16 Missing ⚠️
src/app.rs 30.00% 7 Missing ⚠️
src/main.rs 58.33% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #577      +/-   ##
==========================================
+ Coverage   72.83%   76.55%   +3.71%     
==========================================
  Files          40       41       +1     
  Lines       10567    11463     +896     
==========================================
+ Hits         7697     8775    +1078     
+ Misses       2870     2688     -182     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pamburus pamburus force-pushed the feature/compression branch 13 times, most recently from 0ad49ac to 158df81 Compare December 8, 2024 18:27
@pamburus pamburus force-pushed the feature/compression branch 8 times, most recently from 0e5e83b to 1ee16bc Compare December 17, 2024 23:50
@pamburus pamburus force-pushed the feature/compression branch from 1ee16bc to acc5ab0 Compare December 18, 2024 21:01
@pamburus pamburus marked this pull request as ready for review December 18, 2024 21:21
@pamburus pamburus merged commit 91bba5f into master Dec 18, 2024
4 checks passed
@pamburus pamburus deleted the feature/compression branch December 18, 2024 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant