You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the coursebook scraper uses calls to utils.RetryHTTP (defined in utils/methods.go) in order to handle the automatic retry process for requests. In these calls, it provides a callback which, after a certain number of retries, enters a "long delay" state where it simply waits for a long period of time (i.e. 5 minutes) before attempting to query Coursebook again.
This is not sufficient, however, as it appears oftentimes the scraper does not recover from this "long delay" state and is instead only able to make progress once restarted. This would imply a need to not only wait for a longer period of time, but also to create a new chromedp context in order to fix the issue.
As such, the following actions should be taken:
Develop a utility function which can establish a new chromedp context that picks up from where an old one left off
Add calls to this utility function when the scraper enters a long delay state
The text was updated successfully, but these errors were encountered:
jpahm
added
the
L2
A task suitable for someone who is comfortable helping with implementing features.
label
Oct 10, 2024
jpahm
changed the title
Refresh chromedpContext on long scraper delay
Refresh chromedp context on long scraper delay
Oct 10, 2024
Currently, the coursebook scraper uses calls to
utils.RetryHTTP
(defined inutils/methods.go
) in order to handle the automatic retry process for requests. In these calls, it provides a callback which, after a certain number of retries, enters a "long delay" state where it simply waits for a long period of time (i.e. 5 minutes) before attempting to query Coursebook again.This is not sufficient, however, as it appears oftentimes the scraper does not recover from this "long delay" state and is instead only able to make progress once restarted. This would imply a need to not only wait for a longer period of time, but also to create a new chromedp context in order to fix the issue.
As such, the following actions should be taken:
The text was updated successfully, but these errors were encountered: