I developed an archive system for our spacecraft TM/TC using HDF5 with blosc-lz compression for speed reasons (and I plan to move to Zstd in the future). One of the main issue I have seen in the industry is the difficulty to upgrade the hardware so we have to design something working great on regular hard drives and with minimal RAM requirements.
While HDF5 is good when you manage the whole system, it is always tricky for sharing data with other people so we are also using SQLite. We lose the compression but it is very easy to share and people are more familiar with it.
For the export at GSOC we had a streaming REST interface based on akka-streams. It is used to get the data into other systems based on spark for more complex analysis.
Of course you also lose the compression, but the target system is typically only interested in a small subset of the data.
Sure, I looked at parquet and HDF5. The compression is much better with iptm for the kind of data I am interested in, and the algorithm is also much simpler. It's less than 500 LOC in total.
The reason I wrote this in typescript is that it is nice for prototyping when working with JSON data. If it ever turns out too slow, I would write a rust version. Should not take more than a few days.
I have not profiled it, but I would guess that the most time is spent in the zlib deflate implementation, which is of course already a C library, so I would not expect miracles from a rewrite in rust.
Their main advantage is that they have good, mature implementations in a variety of languages, which would be handy if you ever find Javascript to be too slow.