Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow using local/cached CM2.6 dataset in data step #86

Open
raehik opened this issue Aug 23, 2023 · 2 comments
Open

Allow using local/cached CM2.6 dataset in data step #86

raehik opened this issue Aug 23, 2023 · 2 comments
Labels
enhancement New feature or request post-iccs-2023 Issues created during the ICCS 2023 project, but not completed.

Comments

@raehik
Copy link
Contributor

raehik commented Aug 23, 2023

The intake library appears not to have any caching. This means every time you run the data step, you re-download all parts of the CM2.6 dataset you require. This means more net bandwidth, and more Google Cloud charges.

Maybe we could add a CLI option for intake to try to load data from a given path (cache) first, and revert to the online data if not present (and update the cache...? appears to become more complex when we consider versioning)

@raehik
Copy link
Contributor Author

raehik commented Aug 25, 2023

We probably shouldn't tackle this until after #85 at least.

@arthurBarthe
Copy link
Collaborator

Do you mean to allow for the use of a local download of the cm2.6 simulations? One thing I had done in my implementation is that I implemented some caching: if the file had been downloaded previously, it was using that instead of the Google Bucket. However, to do this I had to modify the fsspec library at the time, which is an ugly solution. Might be worth checking if intake now allows for caching, I don't think it was the case back then.

@raehik raehik added this to the 2022-10 wrap-up: extension milestone Oct 26, 2023
@raehik raehik changed the title Data step: Allow using local data Allow using local/cached CM2.6 dataset in data step Oct 26, 2023
@raehik raehik added the enhancement New feature or request label Nov 9, 2023
@raehik raehik removed this from the 2023-10 wrap-up: extension milestone Nov 9, 2023
@raehik raehik added the post-iccs-2023 Issues created during the ICCS 2023 project, but not completed. label Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request post-iccs-2023 Issues created during the ICCS 2023 project, but not completed.
Projects
None yet
Development

No branches or pull requests

2 participants