News for hackers. Stuff that matters.
Re:More than 9M broken links on Wikipedia are now rescued by duckington on Tuesday October 02, 2018 @01:11PM (#18123583)
https://boingboing.net/2017/04/22/internet-archive-to-ignore...

They are gradually expanding the scope of sites where robots.txt is ignored, perhaps "testing the water".

There are no comments on this comment yet.