Re:More than 9M broken links on Wikipedia are now rescued
by dingaling on Tuesday October 02, 2018 @01:56AM
(#18119120)
> If that means placating copyright holders it’s worth the cost.
In many cases it's nothing to do with the copyright holder. In fact, the complete opposite: someone who subsequently bought the domain and unwittingly stuck a generic robots.txt on the site.
If the IA hid content upon receiving a request from webmaster@domain then that'd make sense. But doing it automatically and retrospectively from robots.txt is bizarre and shows again the dangers of centralisation on the Internet.
I now block IA from my sites as a protest. We need more competition and fresh ideas in digital archiving.
Re:More than 9M broken links on Wikipedia are now rescued
by rovr138 on Monday October 01, 2018 @07:07PM
(#18117560)
> They keep the data and just don’t display it.
I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?
Sure the archive is useful today, but it’s primary purpose is retaining information for future generations. If that means placating copyright holders it’s worth the cost.