Re:Dense bitpacking
by hywel on Saturday August 08, 2015 @02:09PM
(#10028136)
It doesn't actually use bitfields, it describes an alternative to them that was a better choice in this specific instance.
Huffman would have been much slower to access, which would have been unacceptable for the use case (iterating 100 million data points multiple times a second).
Re:Dense bitpacking
by uxcn on Saturday August 08, 2015 @02:45PM
(#10028256)
I'm not familiar with the dataset you were working with, but I would be willing to bet certain values occur a lot more frequently than others. In which case, you could work with the compressed values, and only uncompress them for the results.
Huffman would have been much slower to access, which would have been unacceptable for the use case (iterating 100 million data points multiple times a second).