Re:Sorted Integer Compression
by atombender on Friday June 18, 2021 @09:12AM
(#27549914)
That book is great, and relies on a combination of Golomb-Rice, delta coding, and arithmetic coding. I wonder how this compares with the OP's compression.
... it doesn't just "rely on" that stuff, but it explains the fundamentals of them really well!
That book is really dated (1994), but unfortunately there aren't much better options.
It is unfortunate because the understanding of the field has improved dramatically over the past decades, and explanations from a more modern perspective would benefit even novices.
https://www.amazon.com/Managing-Gigabytes-Compressing-Multim...
For instance you might be keep track of facts like
and you code the gaps; the word "the" appears in every document and the gap is always 1, but the gap for "john" is 11. With a variable-sized encoding you use fewer bits for smaller gaps -- with that kind of encoding you don't have to make "the" be a stopword because you can afford to encode all the postings.