Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refresh chromedp context on long scraper delay #33

Open
2 tasks
jpahm opened this issue Oct 10, 2024 · 2 comments
Open
2 tasks

Refresh chromedp context on long scraper delay #33

jpahm opened this issue Oct 10, 2024 · 2 comments
Assignees
Labels
L2 A task suitable for someone who is comfortable helping with implementing features.

Comments

@jpahm
Copy link
Contributor

jpahm commented Oct 10, 2024

Currently, the coursebook scraper uses calls to utils.RetryHTTP (defined in utils/methods.go) in order to handle the automatic retry process for requests. In these calls, it provides a callback which, after a certain number of retries, enters a "long delay" state where it simply waits for a long period of time (i.e. 5 minutes) before attempting to query Coursebook again.

This is not sufficient, however, as it appears oftentimes the scraper does not recover from this "long delay" state and is instead only able to make progress once restarted. This would imply a need to not only wait for a longer period of time, but also to create a new chromedp context in order to fix the issue.

As such, the following actions should be taken:

  • Develop a utility function which can establish a new chromedp context that picks up from where an old one left off
  • Add calls to this utility function when the scraper enters a long delay state
@jpahm jpahm added the L2 A task suitable for someone who is comfortable helping with implementing features. label Oct 10, 2024
@jpahm jpahm changed the title Refresh chromedpContext on long scraper delay Refresh chromedp context on long scraper delay Oct 10, 2024
@greeshiee
Copy link

I'd love to attempt tinkering with this if that's okay!

@jpahm
Copy link
Contributor Author

jpahm commented Oct 11, 2024

I'd love to attempt tinkering with this if that's okay!

Absolutely!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
L2 A task suitable for someone who is comfortable helping with implementing features.
Projects
None yet
Development

No branches or pull requests

2 participants