Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In ingest-pipeline, modify reporting pipeline for new report format #635

Open
mccalluc opened this issue Jul 22, 2022 · 0 comments
Open

Comments

@mccalluc
Copy link
Contributor

mccalluc commented Jul 22, 2022

Joel has a concern that the reporting scripts rely on particular patterns in the validation report: It would be better if the report could just be parsed as YAML. I don't want to change the validation report first, because that would break the downstream script, so a multi-step process instead:

  • In ingest-pipeline, change the script to just look for No errors anywhere in the first line.
  • When that's done, file a new issue in https://github.com/hubmapconsortium/ingest-validation-tools, asking that the report be changed, so that the first line would be No errors: True (ie, valid YAML)
  • Changes are made in ingest-validation-tools, and a follow-up issue is filed in ingest-pipeline
  • ingest-pipeline is updated to parse the report as JSON.

This is not a high-priority for me, but I think it would address Joel's concerns.

Context:

The software that generates the table of unpublished datasets looks for validation_report.txt in the top level directory, and declares the dataset to have been validated if that file is present and starts with 'No Errors'. This feature is pretty obsolete now that data is usually provided in the form of Uploads, so it could be disabled. If the desired solution is to modify that first line, an issue should be created in ingest-pipeline to disable this check.

Originally posted by @jswelling in hubmapconsortium/ingest-validation-tools#1099 (comment)

@mccalluc mccalluc transferred this issue from hubmapconsortium/ingest-validation-tools Jul 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant