Re:How to Download All of Wikipedia onto a USB Flash Drive
by bscphil on Thursday October 06, 2022 @06:24PM
(#33114903)
I was curious how they achieve this. It looks like the underlying file format uses LZMA, or optionally Zstd, compression. Both achieve pretty high compression ratios against plain text and markup.
> Its file compression uses LZMA2, as implemented by the xz-utils library, and, more recently, Zstandard. The openZIM project is sponsored by Wikimedia CH, and supported by the Wikimedia Foundation.
Re:How to Download All of Wikipedia onto a USB Flash Drive
by kragen on Thursday October 06, 2022 @07:11PM
(#33115354)
The more important thing is that they aggressively downsize the images and omit the history and talk pages. Even if they were using LZW it would probably only triple the filesize.
> Its file compression uses LZMA2, as implemented by the xz-utils library, and, more recently, Zstandard. The openZIM project is sponsored by Wikimedia CH, and supported by the Wikimedia Foundation.
https://en.wikipedia.org/wiki/ZIM_(file_format)