News for hackers. Stuff that matters.

A comment on InfluxDB v0.10 GA with hundreds of thousands writes/sec, 98% better compression.

Re:InfluxDB v0.10 GA with hundreds of thousands writes/sec, ... by chaotic-good on Friday February 05, 2016 @03:54AM (#11040348)
It depends greatly on the shape of your data, but with regularly spaced timestamps at second level precision and float64s, we’ve seen compression reduce each point down to around 2.2 bytes per point.

AFAIK you're using float64 compression scheme from "Gorilla" paper. 2.2 bytes per data point is possible with it but only on data that doesn't utilize full double precision (example: small integers converted to float64). You should compare compression algorithm used by TSM storage engine with zlib or any other general purpose compression algorithm, otherwise this number will be meaningless.

[ Parent | on HN ]
Re:InfluxDB v0.10 GA with hundreds of thousands writes/sec, ... by Dylan16807 on Friday February 05, 2016 @05:03AM (#11040530)
Honestly if you have a series of increasing unix timestamps you should be able to do much better than 2.2 bytes each.
[ Parent | on HN ]
Re:InfluxDB v0.10 GA with hundreds of thousands writes/sec, ... by chaotic-good on Friday February 05, 2016 @05:38AM (#11040629)
Even simple delta+RLE can squeeze increasing unix timestamps up to few bytes. I'm talking about floating point data compression. Even the specialized state of the art algorithms can't guarantee you 2.2 bytes per float in general.
[ Parent | on HN ]