News for hackers. Stuff that matters.

A comment on Apache Arrow Datafusion 5.0.0 release. Parent comment

Re:Apache Arrow Datafusion 5.0.0 release by hashjoiner on Tuesday August 24, 2021 @03:22PM (#28293598)
There is a PR from me (Daniël, committer) with for db-benchmark. For the group by benchmarks, on my machine, it is currently somewhat slower than the fastest (Polars).

https://github.com/h2oai/db-benchmark/pull/182

Also we do support running TPC-H benchmarks. For the queries we can run, those are already finishing faster than Spark. We are planning to do more benchmarking and optimizations in the future.

[ Parent | on HN ]
Re:Apache Arrow Datafusion 5.0.0 release by tomnipotent on Tuesday August 24, 2021 @10:34PM (#28297456)
> it is currently somewhat slower than the fastest (Polars).

That's really promising, considering how fast Polars is. Both are written in Rust and use Apache Arrow, so they can even co-exist in the same context.

[ Parent | on HN ]
Re:Apache Arrow Datafusion 5.0.0 release by hashjoiner on Wednesday August 25, 2021 @04:00AM (#28299073)
Yes, that's pretty exciting! There is even support in Polars to execute the dataframe compute in DataFusion (as it can handle larger than memory datasets).
[ Parent | on HN ]