Re:Interplanetary telemetry compression
by rklaehn on Friday June 22, 2018 @03:46AM
(#17372231)
Sure, I looked at parquet and HDF5. The compression is much better with iptm for the kind of data I am interested in, and the algorithm is also much simpler. It's less than 500 LOC in total.
The reason I wrote this in typescript is that it is nice for prototyping when working with JSON data. If it ever turns out too slow, I would write a rust version. Should not take more than a few days.
I have not profiled it, but I would guess that the most time is spent in the zlib deflate implementation, which is of course already a C library, so I would not expect miracles from a rewrite in rust.
The reason I wrote this in typescript is that it is nice for prototyping when working with JSON data. If it ever turns out too slow, I would write a rust version. Should not take more than a few days.
I have not profiled it, but I would guess that the most time is spent in the zlib deflate implementation, which is of course already a C library, so I would not expect miracles from a rewrite in rust.