ArchiveBox: Open-source self-hosted web archive

Re:ArchiveBox: Open-source self-hosted web archive by toomuchtodo on Saturday March 09, 2019 @05:59PM (#19349089)

When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard.

https://en.wikipedia.org/wiki/Web_ARChive

https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...

If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by progval on Sunday March 10, 2019 @03:44AM (#19350871)

Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by toomuchtodo on Sunday March 10, 2019 @12:05PM (#19352521)

Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by nikisweeting on Monday March 11, 2019 @05:13PM (#19362880)

We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space.

You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130

[ Parent | on HN ]