Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refinement: Improve sending scan config to every crawler. #241

Open
Tracked by #153
sudiptob2 opened this issue Jul 9, 2023 · 2 comments
Open
Tracked by #153

Refinement: Improve sending scan config to every crawler. #241

sudiptob2 opened this issue Jul 9, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@sudiptob2
Copy link
Contributor

All crawlers do not need scan_config for their crawling mechanism. Hence it's redundant to pass the scan config into every crawler and needs refinement.

Ref:

crawler_config,

@aryanagrawal22
Copy link
Contributor

aryanagrawal22 commented Nov 22, 2023

Hi maintainers, I am starting to work on this issue. I plan to make a new staticmethod inside ICrawler base class which is base class of every crawler class.

The method will set if the crawler needs a config or not and then we can call it accordingly to know if we need to send a config or not. Currently, as to what I have seen only StorageBucketsCrawler uses the config file, so only it will return true when asked if cofig needed.

Sounds fair to solve the issue?

@aryanagrawal22
Copy link
Contributor

Hi folks, I have raised a PR which i feel will resolve this issue. PR: #304. Please let me know if any changes are required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants