Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add options for remote JobStore #88

Merged
merged 2 commits into from
Mar 19, 2024
Merged

Add options for remote JobStore #88

merged 2 commits into from
Mar 19, 2024

Conversation

gpetretto
Copy link
Contributor

Following #79, allow different options for the JobStore used remotely to dump the output.
Can be configured with the remote_jobstore option to test different possibilities. The default is left to None to avoid constraining the structure of the configuration, as it is not clear what would be the best option in the future and to avoid issues with old configuration files in the future. The default corresponds to the old option. a maggma JSONStore with json files not zipped. The additional options are the json format based on orjson and msgspec, or msgpack. Files can now optionally be zipped as well.

Copy link
Member

@ml-evs ml-evs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @gpetretto, the code here looks fine to me but I'm aware its not necessarily being hit by the tests (and right now I cannot easily test myself [train station wifi...]).

So I "approve" but would suggest that something is added to the docs about this, as it is not clear to me from the config description alone when I would want to configure a remote_jobstore (I think I understand it from the code itself but if this is meant to be user-facing then it needs more explanation).

@gpetretto gpetretto merged commit b691cd9 into develop Mar 19, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants