Re:ArchiveBox: Open-source self-hosted web archive
by j88439h84 on Saturday March 09, 2019 @05:46PM
(#19349023)
When I want to archive a site, don't I want the content, not just the headers? How do I save the content?
Re:ArchiveBox: Open-source self-hosted web archive
by toomuchtodo on Saturday March 09, 2019 @05:59PM
(#19349089)
When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard.
https://en.wikipedia.org/wiki/Web_ARChive
https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...
If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...
Re:ArchiveBox: Open-source self-hosted web archive
by progval on Sunday March 10, 2019 @03:44AM
(#19350871)
Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.
Re:ArchiveBox: Open-source self-hosted web archive
by toomuchtodo on Sunday March 10, 2019 @12:05PM
(#19352521)
Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.
Re:ArchiveBox: Open-source self-hosted web archive
by nikisweeting on Monday March 11, 2019 @05:13PM
(#19362880)
We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space.
You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130
I believe there is even a docker container for a quick pull and run.