Re:More than 9M broken links on Wikipedia are now rescued
by teddyh on Tuesday October 02, 2018 @11:57AM
(#18122830)
Really? Reference, please!
[ Parent
| on HN ]
Re:More than 9M broken links on Wikipedia are now rescued
by dredmorbius on Tuesday October 02, 2018 @03:59PM
(#18125176)
A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to [email protected]). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Re:More than 9M broken links on Wikipedia are now rescued
by duckington on Tuesday October 02, 2018 @01:11PM
(#18123583)
https://boingboing.net/2017/04/22/internet-archive-to-ignore...
They are gradually expanding the scope of sites where robots.txt is ignored, perhaps "testing the water".