News for hackers. Stuff that matters.
Re:Apache DataFusion by netcraft on Wednesday January 15, 2025 @08:55PM (#42720035)
Why would this be useful over of DuckDb? (earnest question)
Re:Apache DataFusion by chatmasta on Wednesday January 15, 2025 @09:26PM (#42720282)
They’re similar, but DuckDb is more of a batteries-included database whereas DataFusion is an embeddable query engine. You can use DuckDb in embedded-ish scenarios, but it’s not primarily targeting that use case. To put it another way, DataFusion is sometimes described as “the LLVM of databases.”

Another difference is that DuckDb is written in C++ whereas DataFusion is in Rust, so all the usual memory-safety and performance arguments apply. In fact DataFusion has recently overtaken DuckDb in Clickbench results after a community push last year to optimize its performance.

Re:Apache DataFusion by jitl on Wednesday January 15, 2025 @11:36PM (#42721284)
We tried both about 8 months ago, at the time DuckDB’s Node driver leaked memory and segfaulted, DataFusion was missing some features we wanted. But they are both improving rapidly.
Re:Apache DataFusion by geysersam on Thursday January 16, 2025 @12:49AM (#42721766)
> DataFusion has recently overtaken DuckDb in Clickbench results after a community push last year

Really? I don't see it near the top.

[CH benchmarks](https://benchmark.clickhouse.com/#eyjzexn0zw0ionsiqwxsb3leqi...)

Re:Apache DataFusion by kalendos on Thursday January 16, 2025 @03:43AM (#42722955)
You might need to adjust filters to do an apple to apple comparison.

https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

Re:Apache DataFusion by riku_iki on Thursday January 16, 2025 @04:03PM (#42730875)
Not clear why someone need to give up on native duckdb format if it is much faster.
Re:Apache DataFusion by chatmasta on Thursday January 16, 2025 @04:57PM (#42731447)
Because it means you need to keep another copy of your data in a special format just for DuckDb. The point of Parquet is that it’s an open format queryable by multiple tools. You don’t need to wait to load every table into a new format, you don’t need to retain multiple copies, and you don’t need to keep them in sync.

If DuckDb is the only query engine in your analytics stack, then it makes sense to use its specialized format. But that’s not the typical Lakehouse use case.

Re:Apache DataFusion by riku_iki on Thursday January 16, 2025 @05:07PM (#42731518)
> But that’s not the typical Lakehouse use case.

that benchmark is also not typical lakehouse use case, since all data is hosted locally, so they don't test significant component of the stack.

Re:Apache DataFusion by chatmasta on Thursday January 16, 2025 @05:16PM (#42731617)
Yeah, that’s one of many issues with Clickbench. It’s also one table so it can’t test joins.

TPC-H is okay but not Lakehouse specific. I’m not aware of any benchmarks that specifically test performance of engines under common setups like external storage or scalable compute. It would be hard to design one that’s easily reproducible. (And in fairness to Clickbench, it’s intentionally simple for that exact reason - to generate a baseline score for any query engine that can query tabular data).

Re:Apache DataFusion by alamb on Thursday January 16, 2025 @12:24PM (#42728160)
Specifically, DataFusion is faster when querying parquet directly.

Most of the leaderboard of ClickBench is for database specific file formats (that you first have to load the data into)

Re:Apache DataFusion by alamb on Thursday January 16, 2025 @12:23PM (#42728151)
I think you would pick DataFusion over DuckDB if you want to customize it substantially. Not just with user defined functions (which are quite easy to write in DataFusion and are very fast), but things like * custom file formats (e.g. Spiral or Lance) * custom query languages / sql dialects * custom catalogs (e.g. other than a local file or prebuilt duckdb connectors) * custom indexes (read only parts of parquet files based on extra information you store) * etc.

If you are looking for the nicest "run SQL on local files" experience, DuckDB is pretty hard to beat

Disclaimer: I am the PMC chair of DataFusion

There are some other interesting FAQs here too: https://datafusion.apache.org/user-guide/faq.html