News for hackers. Stuff that matters.
Re:Ask HN: Examples of interesting robots.txt by r4vik on Thursday August 11, 2011 @08:27AM (#2872494)
Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.

There are no comments on this comment yet.