More than 9M broken links on Wikipedia are now rescued

Re:More than 9M broken links on Wikipedia are now rescued by Hard_Space on Monday October 01, 2018 @07:17PM (#18117605)

It's an invaluable resource. In the past week I realized it is quite likely that a site I used to work at which hosts a lot of my portfolio might disappear or be heavily amended. So I located and indexed a complete list of my articles there -- and I was even able to click a button and create an archive for the few pages that IA hadn't bothered to index (I wish I had known about this before, since it was a shock to find that IA can be quite selective, and to find a page you were hoping was there simply isn't, and is now irretrievable).

But as one other commenter here has mentioned, you're only a robots.txt amend away from the oblivion that the entire IMDB comments section fell into [1], so a good archiving system is essential. I use (no affiliation)Save Page WE on Waterfox:

https://addons.mozilla.org/en-US/firefox/addon/save-page-we/

[1] https://news.ycombinator.com/item?id=13571893

[ Parent | on HN ]

Re:More than 9M broken links on Wikipedia are now rescued by zmw on Tuesday October 02, 2018 @01:51AM (#18119106)

Wayback Machine isn’t Googlebot, it doesn’t crawl the web, so there’s no such thing as “hadn’t bother to index”... Someone, be it a human or a bot, needs to submit a page for archival.

Programmatically submitting to Wayback Machine is trivial enough, so I have cron jobs backing up most of my static sites (in their entirety) periodically.

[ Parent | on HN ]

Re:More than 9M broken links on Wikipedia are now rescued by duckington on Tuesday October 02, 2018 @12:47PM (#18123350)

"bothered to index" and "IA can be quite selective" implies that there's a human overseer to the Wayback operation, not true!

Pages are archived automatically, and sites are crawled by robots, not humans.

[ Parent | on HN ]