Re:Screenshots Forever and Ever Until You Can’t Stand it
by ggreer on Wednesday August 20, 2014 @04:28AM
(#8201516)
This is a huge problem. Sites like NASA's NTRS are retroactively blocked.[1] It's not clear which user agents one must allow in robots.txt. NTRS allows archive.org_bot, but apparently ia_archiver is also needed. At some point the allow directive in the NTRS robots.txt[2] no longer matched, nuking all historical data.
1. See http://web.archive.org/web/20121029225832/http://ntrs.nasa.g... for an example.
2. http://ntrs.nasa.gov/robots.txt