Re:Wayback machine gets a facelift, new features
by mburns on Monday November 04, 2013 @08:19PM
(#6672503)
Those cornercases can exist on sites that don't have a robots.txt and still have to be crawled correctly.
[ Parent
| on HN ]
There are no comments on this comment yet.