Re:Wayback machine gets a facelift, new features
by memracom on Monday November 04, 2013 @08:20PM
(#6672508)
That is NOT a technical reason.
Technically speaking a robots.txt that says
User-agent: *
Disallow: /
means that you should not crawl the site today. It should have no effect whatsoever on displaying pages that WERE crawled before the timestamp on the robots.txt file.
Technically speaking a robots.txt that says
User-agent: * Disallow: /
means that you should not crawl the site today. It should have no effect whatsoever on displaying pages that WERE crawled before the timestamp on the robots.txt file.