InfluxDB is betting on Rust and Apache Arrow for next-gen data store

Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by pauldix on Tuesday November 10, 2020 @04:07PM (#25051929)

Yeah, Parquet is awesome. One of the things we really want to do here is to push DataFusion (the Rust based SQL execution engine) to work on Parquet files, but to push down predicates and other things and operate on the data while it's compressed.

You pay such a high overhead marshalling that data into an Arrow RecordBatch. Best thing ever is to work with the Parquet file and not even decompress the chunks that you don't need. Of course, this assumes that you're writing summary statistics as part of the metadata, which we plan to do.

[ Parent | on HN ]

Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by nevi-me on Tuesday November 10, 2020 @05:21PM (#25052860)

There's rudimentary statistics support, but I've found that it accounts for a lot of the write time (I wrote some tests last weekend, I can ping your team when I put them on GH).

Improving our stats writing could yield a lot of benefits. I'll open JIRAs for this in the next few days.

[ Parent | on HN ]