Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready.
It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1].
It does both rewritten archiving (basically overwriting the page style) and raw archiving.
I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet.
I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already.
Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready.
It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1].
It does both rewritten archiving (basically overwriting the page style) and raw archiving.
I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet.
I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already.
1: https://github.com/fake-name/AutoTriever 2: https://github.com/fake-name/ChromeController 3: https://github.com/fake-name/WebRequest