Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more params to web/spider #174

Open
postmodern opened this issue Aug 31, 2024 · 0 comments
Open

Add more params to web/spider #174

postmodern opened this issue Aug 31, 2024 · 0 comments
Labels
Milestone

Comments

@postmodern
Copy link
Member

postmodern commented Aug 31, 2024

Add more params to the web/spider worker to control the spider.

  • limit (Integer)
  • max_depth (Integer)
  • delay (Integer)
  • open_timeout (Integer)
  • read_timeout (Integer)
  • ssl_timeout (Integer)
  • continue_timeout (Integer)
  • keep_alive_timeout (Integer)
  • proxy (URI) (eventually we will want to define a global HTTP proxy option for all web/* workers).
  • referer (String)
  • user_agent (String) (eventually we will want to define a global User-Agent option for all web/* workers).
  • robots (Boolean)

The web/spider worker should also have a agent_kwargs method which builds a Symbol Hash based on the params, which is then passed to Ronin::Web::Spider.site(base_uri,**agent_kwargs).

@postmodern postmodern changed the title Add more params to web/spider Add more params to web/spider Sep 7, 2024
@postmodern postmodern added this to the 0.2.0 milestone Sep 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant