News for hackers. Stuff that matters.
Re:Archive.org: Only 12 days to reach the goal of $150,000 by drcube on Thursday December 20, 2012 @10:12AM (#4948062)
>The archive.org team does follow robots.txt and I believe they remove content retroactively meaning if you update your site with a robots.txt it will delete the old content (which I think sucks).

Every time the "Change Facebook back to the way it was!" brigade came out, I would link to the wayback machine's copy of facebook.com from 2005 and say "Is this what you want??". Now I can't do that anymore because of stupid robots.txt.

Re:Archive.org: Only 12 days to reach the goal of $150,000 by webreac on Friday December 21, 2012 @04:20AM (#4952073)
I hope they have backup of this old content. This robot.txt policy is crap. robot.txt should not be taken into account retroactively when the site owner has changed.