Re:Wayback machine gets a facelift, new features
by cmarschner on Monday November 04, 2013 @06:32PM
(#6671960)
There's a technical reason - the blocked pages might open infinite URL spaces or bring the site down (crawler hitting /cgi-bin).
[ Parent
| on HN ]
Re:Wayback machine gets a facelift, new features
by memracom on Monday November 04, 2013 @08:20PM
(#6672508)
That is NOT a technical reason.
Technically speaking a robots.txt that says
User-agent: * Disallow: /
means that you should not crawl the site today. It should have no effect whatsoever on displaying pages that WERE crawled before the timestamp on the robots.txt file.