Re:Google's robots.txt
by Cthulhu_ on Sunday February 23, 2014 @08:59AM
(#7285898)
I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
[ Parent
| on HN ]