News for hackers. Stuff that matters.

A comment on Robots.txt is a suicide note.

Re:Robots.txt is a suicide note by djmdjm on Tuesday May 10, 2011 @03:15AM (#2531689)
Dear "archiveteam", I pay by the MB. How do I opt out of your shit? KTHXBYE
[ Parent | on HN ]
Re:Robots.txt is a suicide note by sbierwagen on Tuesday May 10, 2011 @03:20AM (#2531700)
Don't accept user-generated content. Failing that, never go down, and never announce that you're shutting down.
[ Parent | on HN ]
Re:Robots.txt is a suicide note by v21 on Tuesday May 10, 2011 @06:19AM (#2531982)
Or, if you do, hand over the data in a nicely archivable form, and prevent the need for crawlers.
[ Parent | on HN ]