You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We obviously face the problem of these external data sources changing schema from under our feet. A great way to handle this would be to add unit tests what would
1.) Assess whether a change has occurred
2.) Indicate which columns are different to simplify the research that needs to be done in order to get the scraping script back up.
We don't have to do any more engineering at that point, but it would set us up to have nightly builds and alerts if any of these things fail.
The text was updated successfully, but these errors were encountered:
We obviously face the problem of these external data sources changing schema from under our feet. A great way to handle this would be to add unit tests what would
1.) Assess whether a change has occurred
2.) Indicate which columns are different to simplify the research that needs to be done in order to get the scraping script back up.
We don't have to do any more engineering at that point, but it would set us up to have nightly builds and alerts if any of these things fail.
The text was updated successfully, but these errors were encountered: