Robots.txt is a suicide note

Re:Robots.txt is a suicide note by chalst on Tuesday May 10, 2011 @04:09AM (#2531798)

that's what robots.txt is for and works well

Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Archiveteam.org have indicated that they wish to join the side of the spammers and incompetent spider authors.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by joelmichael on Tuesday May 10, 2011 @05:34AM (#2531922)

I agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by user24 on Tuesday May 10, 2011 @04:11AM (#2531802)

are you confusing archive.org and archiveteam?

[ Parent | on HN ]

Re:Robots.txt is a suicide note by chalst on Tuesday May 10, 2011 @04:58AM (#2531867)

Thanks, fixed. Actually I was not, that was a typo.

[ Parent | on HN ]