Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added three new bad bots #10

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

chrisfu
Copy link

@chrisfu chrisfu commented Jun 19, 2014

No description provided.

chrisfu added 3 commits June 19, 2014 11:44
This crawler is supposed to be respectful. Unfortunately it ties Magento's core_url_rewrite table in knots by hammering non-existent links with non-existant parameters. More to the point, this crawler exists only so that Majestic SEO can sell the data it generates. It'll never be of any use to anyone that doesn't hire them, so blocking it is fine.
I waited 3 weeks for AhrefsBot to honor a robots.txt. Gave up and blocked it.

I removed the carat from MJ12Bot. I mean, who *else* would use that in their user agent string?
@einfallstoll
Copy link

I'd really like to see this merged.

@chrisfu
Copy link
Author

chrisfu commented Jan 12, 2016

The project appears to be dead, which is a shame. Need folk to decide which fork to choose to resume development and ask folk to resubmit pull requests to that fork.

@stevetoza
Copy link

I dont mind creating a fork if people want to push there and we can resume ?

@einfallstoll
Copy link

I suggest to use the fork of @chrisfu and maybe merge the changes by @kijin

@chrisfu
Copy link
Author

chrisfu commented Jan 18, 2016

That's fine with me @einfallstoll. If you sent me a pull request @kijin I'd be happy to merge your changes.

@Firesphere
Copy link
Contributor

Also an option, is add collaborators, so more people can merge pull-requests?

@einfallstoll
Copy link

@Firesphere Yes, that'd be possible as well, but it seems like @bluedragonz abandoned his account.

@chrisfu No need for a PR. Just cherry pick / rebase / merge them.

@chrisfu
Copy link
Author

chrisfu commented Jan 27, 2016

@einfallstoll Sounds good to me.

@Firesphere I can add a couple of collaborators to my fork, no problem. Any takers?

@Firesphere
Copy link
Contributor

I'm okay with keeping an eye on pull requests and manage/merge them every once in a while.

@einfallstoll
Copy link

I could do this as well.

I'm okay with keeping an eye on pull requests and manage/merge them every once in a while.


Reply to this email directly or view it on GitHub.

@chrisfu
Copy link
Author

chrisfu commented Jan 31, 2016

I've added you both as collaborators now. :)

@Firesphere
Copy link
Contributor

Yep, got the message.
It seems you already included the pull-requests here. I'll keep an eye out and check if the bots are indeed malicious, and possibly update from here to your repo.
Thanks for the effort!
Right now, I'm quite short on time, since I'm migrating from NL to NZ (only one letter difference, and 40k km apart :P ) but when I arrive and settled, I'll be more active again :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants