News for hackers. Stuff that matters.
Re:Screenshots Forever and Ever Until You Can’t Stand it by ggreer on Wednesday August 20, 2014 @04:28AM (#8201516)
This is a huge problem. Sites like NASA's NTRS are retroactively blocked.[1] It's not clear which user agents one must allow in robots.txt. NTRS allows archive.org_bot, but apparently ia_archiver is also needed. At some point the allow directive in the NTRS robots.txt[2] no longer matched, nuking all historical data.

1. See http://web.archive.org/web/20121029225832/http://ntrs.nasa.g... for an example.

2. http://ntrs.nasa.gov/robots.txt

Re:Screenshots Forever and Ever Until You Can’t Stand it by db48x on Wednesday August 20, 2014 @08:25PM (#8205385)
It's not been nuked, merely hidden.