News for hackers. Stuff that matters.

A comment on InfluxDB is betting on Rust and Apache Arrow for next-gen data store. Parent comment

Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by seg_lol on Tuesday November 10, 2020 @07:16PM (#25053963)
I am confused about Parquet wrt to the rest of your stack. Is it just that Parquet happens to be the Redshift export format? Or are you actually using Arrow and Parquet at the same time in some manner?
[ Parent | on HN ]
Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by mhall119 on Tuesday November 10, 2020 @07:59PM (#25054327)
Parquet is the persistent storage format for when the data is written to disk or object storage. Arrow is the in-memory format and a set of tools built around working with it.

So InfluxDB IOx will use both, Arrow in-memory for fast access, and Parquet on storage for persistence.

[ Parent | on HN ]
Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by seg_lol on Thursday November 12, 2020 @06:56AM (#25068851)
Ok, I didn't realize that Arrow was in-memory and not also on-disk. As serializing between the two didn't make sense to me. I would have thought that Arrow would also be an on-disk format (mmap'd) so that there would be little to no conversion losses.

Why convert to an on-disk format (Parquet) and not save the in-memory representation to storage directly?

[ Parent | on HN ]