Re:Sorted Integer Compression
by NohatCoder on Friday June 18, 2021 @02:48PM
(#27553549)
If you know the specifics of your data, maybe. But try to err on the side of simpler models, every detail you add slows your code down, and the difference after compression is often small.
Or simply a delta off the last delta (or mean of last n deltas, I guess), so you can do the compression in a stream.
Re:Sorted Integer Compression
by wallacoloo on Friday June 18, 2021 @01:03PM
(#27552463)
If I understand you & GP, these are both just particular instantiations of Linear Predictive Coding [1]. For both of these approaches, you’d probably get similar or better results just throwing your data into an off-the-shelf LPC, and with less effort.
[1] https://en.m.wikipedia.org/wiki/Linear_predictive_coding
That might also be equivalent to one of the ARMA models, although I'd have to do the math to prove that (and to derive which one corresponds to what you're describing).
Edit: It seems that if you detrend the data, the "delta off the last delta", if I understand it correctly, is an MA(1) model. I think. Maybe. My math is wobbly.
Calculating a trend line and storing deltas off of that trend line might be mildly better for some series. Having an AR model and storing errors of the model might work even better, although the obvious (and common) problem is which AR model of the many possible ones you go for.