News for hackers. Stuff that matters.
Re:Airport for DuckDB by mrbungie on Thursday May 22, 2025 @09:34PM (#44068994)
I was almost going to build a lakehouse* with DuckDB because I low-key love it, easiest and strongest analytical engine I've found yet: scale from laptops to big metal, while being mostly out-of-core when doing sane stuff, and avoiding distributed computing for SQL in the process (looking at you Spark).

That is until I found out it does not support Iceberg writes[1], big nono as I would need another engine for inserts, and I want a simple stack :(. What a bummer.

[1] https://github.com/duckdb/duckdb_iceberg/issues/37

*that is what they are called now aren't they? I just can't follow the terms anymore haha.

Re:Airport for DuckDB by mritchie712 on Thursday May 22, 2025 @09:52PM (#44069098)
it's coming. they already have hive style parquet writes. Iceberg is more complicated than that, but it's certainly doable.
Re:Airport for DuckDB by mrbungie on Thursday May 22, 2025 @09:56PM (#44069130)
Yeah, it just would be great if it already did so and I hope it supports Iceberg soon, as it would enable me to change expensive (and bad) engines like AWS Athena for something more manageable.

Don't get me wrong, I'm just being a tongue-in-check egotistical bastard data engineer from hell. DuckDB is a fine piece of software as it is, and those mantainers deserve heaven.

Re:Airport for DuckDB by jeadie on Thursday May 22, 2025 @10:15PM (#44069226)
This is one of the ideas behind using DuckDB in github.com/spiceai/spiceai
Re:Airport for DuckDB by mrbungie on Thursday May 22, 2025 @10:37PM (#44069356)
Looks very cool! I will take a look, tysm!
Re:Airport for DuckDB by anentropic on Friday May 23, 2025 @07:41AM (#44071930)
That looks like an amazing "swiss army knife"...!
Re:Airport for DuckDB by nicornk on Friday May 23, 2025 @02:03AM (#44070295)
Fivetran tried to upstream write support but it was not accepted https://github.com/duckdb/duckdb-iceberg/pull/95
Re:Airport for DuckDB by shakna on Friday May 23, 2025 @06:42AM (#44071622)
That sounds less "not accepted" and more "will implement, rewrite required". It was only a couple months ago.
Re:Airport for DuckDB by sukhavati on Friday May 23, 2025 @07:43AM (#44071946)
same here man, ended up going with trino explicitly for writing and data management and using chdb/duckdb to process data for front-ends etc (mostly ethereum data so chdb "support" for ui256 is quite important)
Re:Airport for DuckDB by buremba on Friday May 23, 2025 @07:48AM (#44071973)
Not just for building a new one, it can also complement existing data-warehouse/lakehouses: https://github.com/buremba/universql

The flight extension is excellent as it removes the need to write C++ extensions and lets you use your favorite language to develop native DuckDB catalogs. It's straightforward to build data lake connectors and plug them in as a flight catalog, thanks to Airport!

Re:Airport for DuckDB by benrutter on Friday May 23, 2025 @02:34PM (#44075265)
I'm curious, did you consider delta tables? Pretty sure duckdb supports them nicely. If you did, how come you chose not to go with them?
Re:Airport for DuckDB by mrbungie on Saturday May 24, 2025 @09:35AM (#44081014)
Afair (I might be wrong) AWS and a big chunk of the industry is promoting Iceberg over Delta. Delta is mostly backed up by Databricks.