DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System

Re:DwarFS – The Deduplicating Warp-Speed Advanced Read-Only ... by karteum on Friday April 12, 2024 @08:01AM (#40011571)

I guess it's without the pictures then ? Because if I compare with the zim file format (which is optimized for this use-case) https://kiwix.org/en/what-is-the-size-of-wikipedia/ I read "As of October 2022, the Full English Wikipedia (ca. 6.5 million articles), with images will use up 91GB of storage space (German and French, the second-largest: 36 GB). (...) If you can do without the images (what we call the nopic version), then you are down to 46 GB."

[ Parent | on HN ]

Re:DwarFS – The Deduplicating Warp-Speed Advanced Read-Only ... by mhx77 on Friday April 12, 2024 @10:03AM (#40013030)

Correct, there are no images in the data except for 68 PNGs. It's just HTML files.

[ Parent | on HN ]

Re:DwarFS – The Deduplicating Warp-Speed Advanced Read-Only ... by nextaccountic on Saturday April 13, 2024 @06:49PM (#40026900)

how it's possible that a bunch of html files would add up to 200gb? is it because of some kind of overhead?

would maybe a database dump be smaller?

[ Parent | on HN ]

Re:DwarFS – The Deduplicating Warp-Speed Advanced Read-Only ... by hifromwork on Monday April 15, 2024 @08:37AM (#40039778)

Well, "a bunch" is an understatement, I bet they have a bit more than just a bunch! It does pass a sniff test, since from https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia:

>As of May 2015, the current version of the English Wikipedia article / template / redirect text was about 51 GB uncompressed in XML format.

Compressed data at the same time was 11.5 GB. And that's data from 9 years ago, and just English Wikipedia.

For comparison, I collect leaked password dumps and they (combined, after deduplication) go into hundreds of GBs too. And that's for just username:password lines, not even text.

[ Parent | on HN ]