ArchiveBox: Open-source self-hosted web archive

Re:ArchiveBox: Open-source self-hosted web archive by toomuchtodo on Sunday March 10, 2019 @12:05PM (#19352521)

Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by nikisweeting on Monday March 11, 2019 @05:13PM (#19362880)

We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space.

You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130

[ Parent | on HN ]