Re:ArchiveBox: Open-source self-hosted web archive
by machawk1 on Sunday March 10, 2019 @11:41PM
(#19356479)
When I initially coded up WARCreate, the webRequest API was still experimental. I believe there are more mature APIs that can be used from the extension context but some require DevTools to be visually open, which is not a common usage pattern of a typical web user.
Per the ticket, we have worked on a few other JavaScript-driven web preservation project like https://github.com/N0taN3rd/node-warc and https://github.com/N0taN3rd/Squidwarc, among others.
Web archiving from a browser extension is difficult but can be improved. I don't know of any other approaches at trying to do this via a browser extension beyond submitting a URI to an archive.
Re:ArchiveBox: Open-source self-hosted web archive
by nikisweeting on Tuesday March 12, 2019 @02:25PM
(#19371361)
Woah both these tools are awesome, thanks for sharing them!
I'm adding links to both from https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm...
There does not seem to be a tool to actually capture a warc directly in your own browser session. webrecorder (http://webrecorder.io/) is the only example I could find that comes close in terms of user experience but it still requires a third party and different browsing habits
- are there browser extensions that can save a warc while you browse?
- are there API limitations that require external browser control? something browser extensions can't be used for?
- or is it simply a question of use case. And crawlers are more popular (for archiving) than locally recorded browser history (for search/analytics)?
edit:
I have now found https://github.com/machawk1/warcreate, related discussions in issues #111 and #112 are quite interesting. Looks like there are some serious limitations for browser extensions. I will look deeper into how webrecorder works and how this could be combined