Re:Libgen-text: complete Library Genesis, re-packaged as tex...
by torginus on Sunday August 21, 2022 @05:40PM
(#32544736)
I did something similar 3 years ago. My goals were minimal file size, as well as fast read times.
I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially.
At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.
The start of the files had a header with pointers to the manifest.
The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.
Edit: If there's some interest, I can try to convert the dataset to this format.
I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.
The start of the files had a header with pointers to the manifest.
The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.
Edit: If there's some interest, I can try to convert the dataset to this format.