Re:Google's robots.txt
by eksith on Sunday February 23, 2014 @05:05AM
(#7285448)
I don't know if this is universal, but Yandex was quite bad in the past at stumbling around generated links (E.G. calendars that go to infinity) and it wasn't at all unusual to see 400+ links crawled in a day or so. Baidu was the same, but I think they're more well behaved these days.
[ Parent
| on HN ]