-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug fix and list replaced with deque #115
base: main
Are you sure you want to change the base?
Conversation
Someone is attempting to deploy a commit to a Personal Account owned by @neokd on Vercel. @neokd first needs to authorize it. |
Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.? |
Are you referring to a user-agent a browser would use? |
No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not. |
I see. I'll look into adding that option for the scraper. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@neokd thoughts on this?
@Bchass looks good I'll review it fully and merge it. Thinking to publish the scrapper as package on Pypi? |
Is this project being maintained anymore? |
@Bchass yeah the project is maintained. There was merge conflict so i didn't merge this PR. |
Description
Fixed a bug and switched over to a
deque
instead of a listRelated Issues
#112
Changes Made
load_proxy_list()
doesn't rely on an absolute path anymore, this wasn't working as intended.rotate_proxy()
Switched over to a deque for better efficiency when rotating proxiesScreenshots (if applicable)
N/A
Checklist
Please review and check the following before submitting your pull request:
Additional Notes
I would like to add test cases for this