-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added three new bad bots #10
base: master
Are you sure you want to change the base?
Conversation
This crawler is supposed to be respectful. Unfortunately it ties Magento's core_url_rewrite table in knots by hammering non-existent links with non-existant parameters. More to the point, this crawler exists only so that Majestic SEO can sell the data it generates. It'll never be of any use to anyone that doesn't hire them, so blocking it is fine.
I waited 3 weeks for AhrefsBot to honor a robots.txt. Gave up and blocked it. I removed the carat from MJ12Bot. I mean, who *else* would use that in their user agent string?
I'd really like to see this merged. |
The project appears to be dead, which is a shame. Need folk to decide which fork to choose to resume development and ask folk to resubmit pull requests to that fork. |
I dont mind creating a fork if people want to push there and we can resume ? |
That's fine with me @einfallstoll. If you sent me a pull request @kijin I'd be happy to merge your changes. |
Also an option, is add collaborators, so more people can merge pull-requests? |
@Firesphere Yes, that'd be possible as well, but it seems like @bluedragonz abandoned his account. @chrisfu No need for a PR. Just cherry pick / rebase / merge them. |
@einfallstoll Sounds good to me. @Firesphere I can add a couple of collaborators to my fork, no problem. Any takers? |
I'm okay with keeping an eye on pull requests and manage/merge them every once in a while. |
I could do this as well.
|
I've added you both as collaborators now. :) |
Yep, got the message. |
No description provided.