News for hackers. Stuff that matters.
Re:Hacker News bans Google and all other search engines by ks on Saturday March 20, 2010 @07:47PM (#1207234)
Have you considered using last-modified/etag headers to control the crawlers? If you detect a crawler, and it requests a page that it has visited before, you could just return a 304 without doing anything else.

There are no comments on this comment yet.