News for hackers. Stuff that matters.
Re:Sorted Integer Compression by jhgb on Friday June 18, 2021 @12:20PM (#27551949)
> For this case that might mean convert the data into deltas, then write those deltas in a byte-oriented variable length integer fashion.

Calculating a trend line and storing deltas off of that trend line might be mildly better for some series. Having an AR model and storing errors of the model might work even better, although the obvious (and common) problem is which AR model of the many possible ones you go for.

Re:Sorted Integer Compression by NohatCoder on Friday June 18, 2021 @02:48PM (#27553549)
If you know the specifics of your data, maybe. But try to err on the side of simpler models, every detail you add slows your code down, and the difference after compression is often small.
Re:Sorted Integer Compression by ballenf on Friday June 18, 2021 @12:51PM (#27552332)
Or simply a delta off the last delta (or mean of last n deltas, I guess), so you can do the compression in a stream.
Re:Sorted Integer Compression by wallacoloo on Friday June 18, 2021 @01:03PM (#27552463)
If I understand you & GP, these are both just particular instantiations of Linear Predictive Coding [1]. For both of these approaches, you’d probably get similar or better results just throwing your data into an off-the-shelf LPC, and with less effort.

[1] https://en.m.wikipedia.org/wiki/Linear_predictive_coding

Re:Sorted Integer Compression by jhgb on Friday June 18, 2021 @01:01PM (#27552436)
That might also be equivalent to one of the ARMA models, although I'd have to do the math to prove that (and to derive which one corresponds to what you're describing).

Edit: It seems that if you detrend the data, the "delta off the last delta", if I understand it correctly, is an MA(1) model. I think. Maybe. My math is wobbly.