I did something similar 3 years ago. My goals were minimal file size, as well as fast read times.

I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.

The start of the files had a header with pointers to the manifest.

The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.

Edit: If there's some interest, I can try to convert the dataset to this format.

See my sibling comment, I did something similar but I used sqlite to store the manifest.

ZIP is seekable -- you can get a list of files and/or read individual files out of the archive without decompressing the whole thing. TAR+(stream compressor) isn't; the only way to get a list of files or to decompress individual files is to read through the whole thing.

The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.

7z's PPMd should do better than zip on compressing text files/speed (7z's default LZMA method is rather slow).

I built an offline wikipedia reader that heavily relied on compression while giving nearly instantaneous access to an arbitrary article (< 100ms). It uses sqlite FTS for searching article titles. I did not enable FTS over the body of the articles.

Outline of the technique is here https://news.ycombinator.com/item?id=28013347

Yes, there's a lot better compression available these days, but not everyone has the software installed to unpack them

> not everyone has the software installed to unpack them

People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone".

> Yes, there's a lot better compression available these days

Tell me more!

zstd: https://github.com/facebook/zstd

Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)

We practically need libgen-crystal to use this anyway, so just link zstd to libgen-crystal and now it's available.

* Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.