News for hackers. Stuff that matters.
Re:Bitpacking and Compression of Sparse Datasets by derefr on Friday January 06, 2017 @09:53PM (#13342145)
Regarding the benefits of knowing your data on compression: rearranging the dimensions of a dataset can also yield interesting increases in compressibility.

For example, you can store your position records arr[T][X,Y] as arr[X,Y][T]: that is, sequences of bits representing how each point on the grid evolves over time, rather than sequences of bits representing boards. These single-position time-series should both be more internally predictable (lower entropy), and more isotropic (having similarity with other features, allowing secondary compressibility) than the original ones, and you should be able to repopulate the in-memory 3D array from the serialized representations in roughly the same time (ignoring caching effects, because a 19x19x28 array is a very small array.)

Of course, this doesn't always work: the same magic could be applied to videos... but decompressed videos do not fit easily into memory, let alone are they small enough to ignore cache-coherency during reads/writes, so actually doing the dimensional transforms is a bit implausible. (But if we really needed a video squeezed 10x more than we do today, and were willing to spend hours on both ends doing so, it'd certainly be possible.)

There are no comments on this comment yet.