Re:Link rot and redirect madness
by jerf on Thursday December 10, 2015 @04:22PM
(#10713575)
In my visualization of perfect blogging platform, it automatically periodically scrapes all your old links and looks for either redirections or wild size changes in the content. It also auto-archives linked content for you (which used to be easier, but can still be at least tried), and could offer these archives to either the blog owner or post reader. But among the many challenges implementing that provides is that it is hard to make it work at scale, because of the resulting legal issues.
[ Parent
| on HN ]
Re:Link rot and redirect madness
by toomuchtodo on Thursday December 10, 2015 @04:35PM
(#10713671)
Or the links are turned into archive.org links automatically at publish time? With any pages not captured immediately captured at that point?
EDIT: This is a hack until the content addressable web arrives.
I've recently started to consider tools like wkhtmltopdf [1] to easily dump a website I'd like to link to. Then I'd include both the original link and the pdf version on the blog page, in order to keep the linked structure and the archived version at the same spot.
I haven't done it yet, but would be interested in other's experiences. It would be interesting to have a crawler that does it automatically for each new post.
Re:Link rot and redirect madness
by dangrossman on Thursday December 10, 2015 @05:21PM
(#10713991)
This would be problematic legally, even when what you're linking to offers a permissive license of some sort.
I tend to keep local PDFs of everything that seems crucial. But linking to them seems ethically questionable. But maybe as a last resort, with an explanatory comment.
Yes, this would be awesome. Even just auto-detecting broken links and redirecting them to archive.org versions instead seems like it would be an easy win that would sidestep the legal problems of archiving them yourself.
Yes, archive.org is an amazing resource. But I'm wondering about the coverage, especially for dynamic and deep content. Maybe those are mainly issues on older snapshots.
Also, copyright issues aside, the prospect of a recursively archived Web is a little mind boggling. But storage is always cheaper, so hey.
Seems that would be better built as a stand-alone spider than a bolt-in to a single blogging platform.
It looks like there are some wordpress plugins that let you check and at least fix the links, but I don't see anything that takes the extra step of archiving.