News for hackers. Stuff that matters.
Re:How to Download All of Wikipedia onto a USB Flash Drive by orliesaurus on Thursday October 06, 2022 @05:38PM (#33114429)
Oh wow, I thought this was gonna be a REALLY large file, but only 95GB not bad, some worthless videogames are larger haha
Re:How to Download All of Wikipedia onto a USB Flash Drive by bscphil on Thursday October 06, 2022 @06:24PM (#33114903)
I was curious how they achieve this. It looks like the underlying file format uses LZMA, or optionally Zstd, compression. Both achieve pretty high compression ratios against plain text and markup.

> Its file compression uses LZMA2, as implemented by the xz-utils library, and, more recently, Zstandard. The openZIM project is sponsored by Wikimedia CH, and supported by the Wikimedia Foundation.

https://en.wikipedia.org/wiki/ZIM_(file_format)

Re:How to Download All of Wikipedia onto a USB Flash Drive by kragen on Thursday October 06, 2022 @07:11PM (#33115354)
The more important thing is that they aggressively downsize the images and omit the history and talk pages. Even if they were using LZW it would probably only triple the filesize.
Re:How to Download All of Wikipedia onto a USB Flash Drive by aendruk on Thursday October 06, 2022 @06:06PM (#33114716)
Circa 2003 I carried around a pared down copy on a Pocket PC. Dropping a few chosen categories (who needs Sports?) allowed it to barely fit on a 1-GB SD card.
Re:How to Download All of Wikipedia onto a USB Flash Drive by FeistySkink on Thursday October 06, 2022 @06:13PM (#33114795)
People going back in time need sports. An almonac of some kind.
Re:How to Download All of Wikipedia onto a USB Flash Drive by II2II on Thursday October 06, 2022 @08:47PM (#33116066)
While handy, it would be a bit too conspicuous. At least one could claim that an almanac is a novelty print.
Re:How to Download All of Wikipedia onto a USB Flash Drive by 988747 on Thursday October 06, 2022 @07:02PM (#33115274)
BTW: what's the difference between 95.2 GB file and 45 GB one? There is no info on download page.
Re:How to Download All of Wikipedia onto a USB Flash Drive by lavezza on Thursday October 06, 2022 @07:21PM (#33115462)
95.2 is the "maxi" file. 49.48 is the "nopic" file. 13.39 is the "mini".

From https://www.kiwix.org/en/documentation/

File size is always an issue when downloading such big content, so we always produce each Wikipedia file in three flavours:

Mini: only the introduction of each article, plus the infobox. Saves about 95% of space vs. the full version. nopic: full articles, but no images. About 75% smaller than the full version Maxi: the default full version.

Re:How to Download All of Wikipedia onto a USB Flash Drive by keepquestioning on Thursday October 06, 2022 @06:57PM (#33115224)
I remember the era of stupidly large games.
Re:How to Download All of Wikipedia onto a USB Flash Drive by orliesaurus on Friday October 07, 2022 @12:36PM (#33123628)
you mean today!