News for hackers. Stuff that matters.

A comment on More than 9M broken links on Wikipedia are now rescued. Parent comment

Re:More than 9M broken links on Wikipedia are now rescued by rovr138 on Monday October 01, 2018 @07:07PM (#18117560)
> They keep the data and just don’t display it.

I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?

[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by slededit on Monday October 01, 2018 @07:11PM (#18117581)
Sure see here: https://archive.org/post/133690/robotstxt-only-gives-tempora...
[ Parent | on HN ]