News for hackers. Stuff that matters.
Re:Google's robots.txt by agwa on Sunday February 23, 2014 @05:32PM (#7287946)
The problem is that both Yandex and Baidu are rather poorly behaved - they hit your website way too fast, downloading large bandwidth files in quick succession. That's actually what led me to the bad-bot-blocker project in the first place. Baidu has also been accused of not respecting robots.txt though I have not personally observed that.

This is the reason they're blocked, not because they're new or non-English.

There are no comments on this comment yet.