News for hackers. Stuff that matters.

A comment on InfluxDB is betting on Rust and Apache Arrow for next-gen data store. Parent comment

Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by gizmodo59 on Tuesday November 10, 2020 @04:51PM (#25052481)
Parquet + Arrow reminds me of a fast SQL engine on data lake called Dremio.

https://www.dremio.com/webinars/apache-arrow-calcite-parquet...

They also have an OSS version in GitHub.

[ Parent | on HN ]
Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by pauldix on Tuesday November 10, 2020 @04:55PM (#25052520)
They're heavily into Arrow. A few years ago they contributed Gandiva, an LLVM expression compiler for super fast processing. https://arrow.apache.org/blog/2018/12/05/gandiva-donation/

It's one of the reasons I like being all in on Arrow. Why do everything ourselves when a ton of other smart people are working on this too?

[ Parent | on HN ]
Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by nevi-me on Tuesday November 10, 2020 @05:20PM (#25052835)
Talking about Gandiva, something that's open for taking: https://issues.apache.org/jira/browse/ARROW-5315 (creating Gandiva bindings for Rust).

I think DataFusion is mature enough that we could plug in Gandiva into it.

Disclaimer: I work on the Arrow Rust implementation

[ Parent | on HN ]
Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by pauldix on Tuesday November 10, 2020 @05:30PM (#25052948)
Ah yes, of course, hi Nevi :). Thank you again for all your work on the Rust implementation. We're obviously big fans.

Gandiva bindings is definitely something we should look into, but I'm guessing there's much lower hanging fruit within DataFusion in terms of optimizing, particularly for our use case.

[ Parent | on HN ]
Re:InfluxDB is betting on Rust and Apache Arrow for next-gen... by nevi-me on Tuesday November 10, 2020 @05:42PM (#25053064)
Thanks Paul :)

I think with compute functions/kernels, we're sitting under a grapevine, so we'll be able to add a lot to Arrow without yet needing Gandiva bindings. The Rust stdsimd [0] work will also enable us to better use SIMD in ~ a year from now (I hope)

[0] https://github.com/rust-lang/stdsimd

[ Parent | on HN ]