News for hackers. Stuff that matters.
Re:Wayback machine gets a facelift, new features by memracom on Monday November 04, 2013 @08:20PM (#6672508)
That is NOT a technical reason.

Technically speaking a robots.txt that says

User-agent: * Disallow: /

means that you should not crawl the site today. It should have no effect whatsoever on displaying pages that WERE crawled before the timestamp on the robots.txt file.

Re:Wayback machine gets a facelift, new features by mmagin on Tuesday November 05, 2013 @04:45AM (#6674218)
Actually, if you want to interpret robots.txt that way, it raises the problem of "how long can I consider a robots.txt valid for?"