Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture Missing Data (Result of daily pipeline failures) #37

Closed
franTarkenton opened this issue Mar 11, 2023 · 2 comments
Closed

Capture Missing Data (Result of daily pipeline failures) #37

franTarkenton opened this issue Mar 11, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@franTarkenton
Copy link
Member

The daily pipeline script runs as the title would suggest... daily. One of the inputs to the script is the current date. When the script fails for a particular day due to data not being available it creates holes in the historical information.

In order to avoid the holes in the data, this ticket would create a script that evaluates a time window for holes in data, and then runs the daily-pipeline for the missing days.

@franTarkenton
Copy link
Member Author

related: #44

Issue 44 would resolve this by running the pipeline for all available data

@franTarkenton franTarkenton added the enhancement New feature or request label Mar 17, 2023
@franTarkenton
Copy link
Member Author

This has been resolved with the matrix build that does the calculation at run time of data available vs data previously processed, and then processes the missing data.

@franTarkenton franTarkenton added zen_reorg temp label to help with migration to ghp and removed zen_reorg temp label to help with migration to ghp labels Dec 5, 2023
@franTarkenton franTarkenton moved this to Closed in RFC Backlog Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Closed
Development

No branches or pull requests

1 participant