Re:Archiving web sites
by fake-name on Thursday November 22, 2018 @11:26PM
(#18514348)
Whee, a topic close to my heart!
https://github.com/fake-name/ReadableWebProxy is a project of mine that started out as a simple rewriting proxy, but at this point is basically a self-contained archival system for entire websites, complete with preserved historical versions of scraped content. It has a distributed fetching frontend[1], uses chromium[2] to optionally deal with internet breaking bullshit (Helloooo cloudflare! Fuuuuucccckkkkk yyyyooouuuuuuu), supports multiple archival modes (raw, e.g. not rewritten and destyled, and a rewritten format which makes reading internet text content actually nice), and a bunch of other stuff. The links in fetched content are rewritten to point within the archiver, and if content is not already retreived, it's fetched on-the-fly as you browse.
It also has plugin-based content rewriting features, allowing the complete reformatting of content on-the-fly, and functions as a backend to a bunch of other projects (I run a translated light-novel/web-novel tracker site, and it also does the RSS parsing for that).
I've been occatonally meaning to add WARC forwarding to the frontend, and feed that into the internet archive, but the fetching frontend is old, creaky and brittle (it's some old code), and does a lot of esoteric stuff that would be hard to replicate.
https://github.com/fake-name/ReadableWebProxy is a project of mine that started out as a simple rewriting proxy, but at this point is basically a self-contained archival system for entire websites, complete with preserved historical versions of scraped content. It has a distributed fetching frontend[1], uses chromium[2] to optionally deal with internet breaking bullshit (Helloooo cloudflare! Fuuuuucccckkkkk yyyyooouuuuuuu), supports multiple archival modes (raw, e.g. not rewritten and destyled, and a rewritten format which makes reading internet text content actually nice), and a bunch of other stuff. The links in fetched content are rewritten to point within the archiver, and if content is not already retreived, it's fetched on-the-fly as you browse.
It also has plugin-based content rewriting features, allowing the complete reformatting of content on-the-fly, and functions as a backend to a bunch of other projects (I run a translated light-novel/web-novel tracker site, and it also does the RSS parsing for that).
I've been occatonally meaning to add WARC forwarding to the frontend, and feed that into the internet archive, but the fetching frontend is old, creaky and brittle (it's some old code), and does a lot of esoteric stuff that would be hard to replicate.
[1]: https://github.com/fake-name/AutoTriever [2]: https://github.com/fake-name/ChromeController