Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Atomate Drone Schema and Migrations #273

Open
mkhorton opened this issue Mar 12, 2019 · 0 comments
Open

Atomate Drone Schema and Migrations #273

mkhorton opened this issue Mar 12, 2019 · 0 comments
Labels
improvement reported issues that considered further improvement to atomate

Comments

@mkhorton
Copy link
Contributor

Atomate drone changes has caused issues for MP production, since it is not feasible to re-ingest whenever the drone changes, and performing aggregations is difficult if a key changes (most recent example of this was the delta_volume_as_percent key).

The atomate drone is quite stable now so hopefully this issue will be minimal going forwards, however I suggest two changes:

  1. The task doc needs a schema (jsonschema). We have tools to easily build up schemas that we've been using, so this will just need to be implemented, it will be an additional dependency however (the jsonschema package itself). JSON Schema is good because it is a standard and, also, can be easily attached to a Mongo collection to perform on-the-fly validation. It is also not necessarily strict, i.e. additional keys might be added and the schema will still validate, it will just ensure that a set of known keys are present. By adopting this standard schema, we can also remove the ad-hoc validation currently in the drone.

  2. We should have code for database migrations. Whenever the schema does change, even if it's just a trivial key rename, we could have a script eg migrate_tasks_0.65_to_0.66.py or similar, as a central source of authority for how to process old task docs.

@itsduowang itsduowang added the improvement reported issues that considered further improvement to atomate label Feb 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement reported issues that considered further improvement to atomate
Projects
None yet
Development

No branches or pull requests

2 participants