News for hackers. Stuff that matters.
Re:Google's robots.txt by anonymfus on Sunday February 23, 2014 @03:48AM (#7285340)
Why you consider Yandex evil?
Re:Google's robots.txt by eksith on Sunday February 23, 2014 @05:05AM (#7285448)
I don't know if this is universal, but Yandex was quite bad in the past at stumbling around generated links (E.G. calendars that go to infinity) and it wasn't at all unusual to see 400+ links crawled in a day or so. Baidu was the same, but I think they're more well behaved these days.
Re:Google's robots.txt by Moru on Sunday February 23, 2014 @05:55PM (#7288064)
Lucky you with only 400+ per day. We had thousands per hour. Still the server survived but it wasn't fast...
Re:Google's robots.txt by Cthulhu_ on Sunday February 23, 2014 @08:59AM (#7285898)
I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
Re:Google's robots.txt by tbirdz on Sunday February 23, 2014 @01:14PM (#7286798)
Yandex is Russian, not Chinese.