Robots.txt is a suicide note

Re:Robots.txt is a suicide note by yaix on Tuesday May 10, 2011 @03:52AM (#2531764)

robots.txt is simple and effective.

I do not want certain bots, especially so-called "archives" to automatically download all my content. And that's what robots.txt is for and works well.

The article is just stupid, sorry. There is not one real knowledgable argument.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by chalst on Tuesday May 10, 2011 @04:09AM (#2531798)

that's what robots.txt is for and works well

Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Archiveteam.org have indicated that they wish to join the side of the spammers and incompetent spider authors.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by joelmichael on Tuesday May 10, 2011 @05:34AM (#2531922)

I agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by user24 on Tuesday May 10, 2011 @04:11AM (#2531802)

are you confusing archive.org and archiveteam?

[ Parent | on HN ]

Re:Robots.txt is a suicide note by chalst on Tuesday May 10, 2011 @04:58AM (#2531867)

Thanks, fixed. Actually I was not, that was a typo.

[ Parent | on HN ]