Re:Make Your Own Internet Archive with Archive Box
by evc on Tuesday January 19, 2021 @12:49PM
(#25836275)
You will need a lot of disk storage right?
[ Parent
| on HN ]
Re:Make Your Own Internet Archive with Archive Box
by LEARAX on Tuesday January 19, 2021 @09:45PM
(#25842013)
There are different extractors/services, and you can toggle them pretty easily. By default it screenshots everything, exports a PDF, saves like 4 different HTML copies and submits the link to the wayback machine. It also tries to extract important text, and stores that separately. You could easily configure it to only extract text, turn off some HTML extractors, or disable the PDF and screenshot captures if you want to prioritize disk space.
Re:Make Your Own Internet Archive with Archive Box
by flas9sd on Tuesday January 19, 2021 @05:56PM
(#25840080)
it doesn't show in the Screenshot in the article, but ArchiveBox in Aug 2020 implemented the "readability article text extractor", see description in the release notes: https://github.com/pirate/ArchiveBox/releases/tag/v0.4.14 and the module that does the work https://github.com/pirate/readability-extractor
By only extracting text and article images you could go deep into an archive. If you skip images, much more so
Re:Make Your Own Internet Archive with Archive Box
by Ace_Archer on Tuesday January 19, 2021 @03:17PM
(#25838312)
That probably depends on the scope of what you're looking to archive. If you're looking to make up local backup of your bookmarks folder (as one of the intentions seems to be), probably not an unreasonable amount of storage. Maybe a few GB at most(if you have a moderate to large bookmarks folder), depending on how many sites/heavy the sites are?
Re:Make Your Own Internet Archive with Archive Box
by reefab on Tuesday January 19, 2021 @08:31PM
(#25841485)
For reference, archivebox uses 250GB for 5000 links in my setup.
Re:Make Your Own Internet Archive with Archive Box
by mosselman on Wednesday January 20, 2021 @03:03AM
(#25843796)
That is an insane amount of storage for so few links. Is your setup somehow very greedy?
Saving article only view (images + text) should probably do better
I suspect your numbers come from JavaScript and css, etc? Is there a way for archivebox to not download react 5000 times but share source files? Most likely custom bundles that sites compile will not make this possible most of the time. Just thinking out loud here.
Re:Make Your Own Internet Archive with Archive Box
by nikisweeting on Wednesday January 20, 2021 @11:25AM
(#25848290)
It's recommended to run it on a compressed filesystem like ZFS. On mine it's using ~75GB for ~3000 URLs. It varies greatly depending on the content, usually the vast majority of storage is from video/audio ripped with youtube-dl.