-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add plugins for fsspec caching and cache cleaning #205
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #205 +/- ##
==========================================
+ Coverage 95.78% 95.87% +0.08%
==========================================
Files 12 12
Lines 2989 3052 +63
==========================================
+ Hits 2863 2926 +63
Misses 126 126
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
The failing test is due to a small change in Posttroll and the way we've implemented the tests (with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work! I'm wondering if this should really be its own plugin, or just an option to the reading plugin?
Co-authored-by: Martin Raspaud <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
About the _fs...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
This PR adds the possibility to do caching with
fsspec
. Mostly useful for remotely read data that has a complicated structure which slows down the reading such as FCI L1c in S3 storage. Additional plugin is available for cleaning the cache.flake8 trollflow2