News for hackers. Stuff that matters.

A comment on More than 9M broken links on Wikipedia are now rescued.

Re:More than 9M broken links on Wikipedia are now rescued by LoSboccacc on Monday October 01, 2018 @06:34PM (#18117388)
And they are one robots away from cancellation. For all the good they do, retroactively applying robots exclusions to their crawler is a terrible thing. Luckily there are alternatives for going forward.
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by slededit on Monday October 01, 2018 @06:36PM (#18117404)
They keep the data and just don’t display it. The last thing they need is a court order demanding they delete it.

Sure the archive is useful today, but it’s primary purpose is retaining information for future generations. If that means placating copyright holders it’s worth the cost.

[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by dingaling on Tuesday October 02, 2018 @01:56AM (#18119120)
> If that means placating copyright holders it’s worth the cost.

In many cases it's nothing to do with the copyright holder. In fact, the complete opposite: someone who subsequently bought the domain and unwittingly stuck a generic robots.txt on the site.

If the IA hid content upon receiving a request from webmaster@domain then that'd make sense. But doing it automatically and retrospectively from robots.txt is bizarre and shows again the dangers of centralisation on the Internet.

I now block IA from my sites as a protest. We need more competition and fresh ideas in digital archiving.

[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by rovr138 on Monday October 01, 2018 @07:07PM (#18117560)
> They keep the data and just don’t display it.

I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?

[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by slededit on Monday October 01, 2018 @07:11PM (#18117581)
Sure see here: https://archive.org/post/133690/robotstxt-only-gives-tempora...
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by dredmorbius on Tuesday October 02, 2018 @03:24AM (#18119447)
IA are ignoring robots.txt as of earlier this year.
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by teddyh on Tuesday October 02, 2018 @11:57AM (#18122830)
Really? Reference, please!
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by dredmorbius on Tuesday October 02, 2018 @03:59PM (#18125176)
A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to [email protected]). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by duckington on Tuesday October 02, 2018 @01:11PM (#18123583)
https://boingboing.net/2017/04/22/internet-archive-to-ignore...

They are gradually expanding the scope of sites where robots.txt is ignored, perhaps "testing the water".

[ Parent | on HN ]