Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Freshness checks are warning/failing #411

Open
ian-r-rose opened this issue Oct 3, 2024 · 8 comments · May be fixed by #436
Open

Freshness checks are warning/failing #411

ian-r-rose opened this issue Oct 3, 2024 · 8 comments · May be fixed by #436
Assignees
@pingpingxiu-DOT-ca-gov
Labels
unplanned Unplanned work added to current sprint, after sprint planning
Milestone
VDS Data Modeling...

Comments

@ian-r-rose
Copy link
Member

We've started to get some warnings (which will shortly be errors) in our source freshness checks: https://california.getdbt.com/deploy/6/projects/10/runs/5414

Unsure if the issue is with the checks or with the relay process right now, @pingpingxiu-DOT-ca-gov can you take a look?
@ian-r-rose ian-r-rose added the unplanned Unplanned work added to current sprint, after sprint planning label Oct 3, 2024
@ian-r-rose
Copy link
Member Author

ian-r-rose commented Oct 4, 2024 •
edited
Loading

@pingpingxiu-DOT-ca-gov the freshness checks are now failing, are you able to take another look?

https://california.getdbt.com/deploy/6/projects/10/runs/5449

@pingpingxiu-DOT-ca-gov
Copy link
Contributor

pingpingxiu-DOT-ca-gov commented Oct 4, 2024 via email

@pingpingxiu-DOT-ca-gov
Copy link
Contributor

pingpingxiu-DOT-ca-gov commented Oct 4, 2024 via email

@ian-r-rose
Copy link
Member Author

Interesting, so it's with the underlying system? That's a problem, but it's good that the freshness checks are working!

@pingpingxiu-DOT-ca-gov
Copy link
Contributor

pingpingxiu-DOT-ca-gov commented Oct 4, 2024 via email

@kengodleskidot
Copy link
Contributor

Good morning and I see that the last equipment set update occurred on 10/1/24 for D3. It looks like the D8 daily equipment feed update last occurred on 9/16/24. I will follow up with our consultant to see what may have changed to affect the D8 feed but since all other equipment feeds require manual updates they are not updated as frequently.
image

@jkarpen
Copy link

jkarpen commented Oct 10, 2024

Per @kengodleskidot the issue appears to be that the districts are not sending data updates. He will follow up with them to see if there is a failure on there side or if this is expected behavior.

If this is expected, @pingpingxiu-DOT-ca-gov says we can add a new timestamp which would detect if the data relay is the issue vs. the equipment data feed.

@pingpingxiu-DOT-ca-gov
Copy link
Contributor

I added #436 PR for making less sensitive on the detector config log freshness.

@jkarpen jkarpen linked a pull request Oct 24, 2024 that will close this issue
Make the detector_config_log freshness check less sensitive #436
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pingpingxiu-DOT-ca-gov pingpingxiu-DOT-ca-gov

Labels
unplanned Unplanned work added to current sprint, after sprint planning
Projects
None yet
Milestone
VDS Data Modeling: Stabilize Data Rel...
Development

Successfully merging a pull request may close this issue.

Make the detector_config_log freshness check less sensitive cagov/caldata-mdsa-caltrans-pems
4 participants
@ian-r-rose @jkarpen @kengodleskidot @pingpingxiu-DOT-ca-gov