News for hackers. Stuff that matters.

A comment on Ask HN: Examples of interesting robots.txt.

Re:Ask HN: Examples of interesting robots.txt by balducci on Thursday August 11, 2011 @06:48AM (#2872243)
How "innovative" do you expect robots.txt to be?
[ Parent | on HN ]
Re:Ask HN: Examples of interesting robots.txt by r4vik on Thursday August 11, 2011 @08:27AM (#2872494)
Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.
[ Parent | on HN ]