Re:Ask HN: Examples of interesting robots.txt
by r4vik on Thursday August 11, 2011 @08:27AM
(#2872494)
Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.
[ Parent
| on HN ]
There are no comments on this comment yet.