Re:Hacker News bans Google and all other search engines
by ks on Saturday March 20, 2010 @07:47PM
(#1207234)
Have you considered using last-modified/etag headers to control the crawlers? If you detect a crawler, and it requests a page that it has visited before, you could just return a 304 without doing anything else.
[ Parent
| on HN ]
There are no comments on this comment yet.