Re:ArchiveBox: Open-source self-hosted web archive
by progval on Sunday March 10, 2019 @03:44AM
(#19350871)
Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.
[ Parent
| on HN ]
Re:ArchiveBox: Open-source self-hosted web archive
by toomuchtodo on Sunday March 10, 2019 @12:05PM
(#19352521)
Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.
Re:ArchiveBox: Open-source self-hosted web archive
by nikisweeting on Monday March 11, 2019 @05:13PM
(#19362880)
We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space.
You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130