ArchiveBox: Open-source self-hosted web archive

Re:ArchiveBox: Open-source self-hosted web archive by platz on Saturday March 09, 2019 @02:34PM (#19347981)

This looks very nice

As an aside,

   wget -r -k -np

works surprisingly well for my offline needs.

For permanent access I defer to archive.is

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by toomuchtodo on Saturday March 09, 2019 @05:06PM (#19348805)

Consider https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site.

I believe there is even a docker container for a quick pull and run.

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by j88439h84 on Saturday March 09, 2019 @05:46PM (#19349023)

When I want to archive a site, don't I want the content, not just the headers? How do I save the content?

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by toomuchtodo on Saturday March 09, 2019 @05:59PM (#19349089)

When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard.

https://en.wikipedia.org/wiki/Web_ARChive

https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...

If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by progval on Sunday March 10, 2019 @03:44AM (#19350871)

Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by toomuchtodo on Sunday March 10, 2019 @12:05PM (#19352521)

Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.

[ Parent | on HN ]

Re:ArchiveBox: Open-source self-hosted web archive by nikisweeting on Monday March 11, 2019 @05:13PM (#19362880)

We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space.

You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130

[ Parent | on HN ]