News for hackers. Stuff that matters.

A comment on Robots.txt is a suicide note. Parent comment

Re:Robots.txt is a suicide note by joelmichael on Tuesday May 10, 2011 @05:34AM (#2531922)
I agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.
[ Parent | on HN ]

There are no comments on this comment yet.