News for hackers. Stuff that matters.
Re:Beringei: A high-performance time series storage engine by crescentfresh on Saturday February 04, 2017 @03:47AM (#13566265)
Ok so it's an inmemory product, sharded no less.

They speak about compressing the data before "storing" it.

I don't have a lot of experience with inmemory anything, but are we talking about retaining the compressed format in server memory here? Ie, RAM is your datastore.

Then, at some point, to serve requests/queries for the data don't you have to get it "out of" RAM and uncompress it, also an inmemory operation?

Did I get this right?

Re:Beringei: A high-performance time series storage engine by storyinmemo on Saturday February 04, 2017 @04:03AM (#13566314)
The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf

Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz

A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

Re:Beringei: A high-performance time series storage engine by jdonaldson on Saturday February 04, 2017 @04:17AM (#13566345)
They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.
Re:Beringei: A high-performance time series storage engine by Scaevolus on Saturday February 04, 2017 @06:22AM (#13566609)
Not quite. Delta of deltas means that periodic timestamps turn into a long run of zeros (since the difference between each point is constant), which is very easy to compress. In the Gorilla paper, 96% of timestamps compress down to zero, which is encoded by a single zero bit.

For series without periodic sampling, delta-of-deltas performs similarly to normal deltas.

Re:Beringei: A high-performance time series storage engine by rakoo on Saturday February 04, 2017 @07:33AM (#13566757)
"delta of delta" is only used for timestamps, not for values. In any standard monitoring setup, the acceleration of timestamps will be 0 for a big portion of measurements, and very small for a big portion of the rest, so it makes compression extremely easy and efficient.
Re:Beringei: A high-performance time series storage engine by throwaway2016a on Saturday February 04, 2017 @11:36AM (#13567566)
> I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

I thought they were talking about healthcare data too when I first read it but seeing as this is Facebook I realized they are talking about server health not human health.

Re:Beringei: A high-performance time series storage engine by machineman44 on Saturday February 04, 2017 @05:31PM (#13569496)
Haha, I thought jdonaldson was making a joke :P