News for hackers. Stuff that matters.

A comment on Robots.txt is a suicide note. Parent comment

Re:Robots.txt is a suicide note by v21 on Tuesday May 10, 2011 @06:19AM (#2531982)
Or, if you do, hand over the data in a nicely archivable form, and prevent the need for crawlers.
[ Parent | on HN ]

There are no comments on this comment yet.