Re:Airport for DuckDB
by mritchie712 on Thursday May 22, 2025 @09:52PM
(#44069098)
it's coming. they already have hive style parquet writes. Iceberg is more complicated than that, but it's certainly doable.
Yeah, it just would be great if it already did so and I hope it supports Iceberg soon, as it would enable me to change expensive (and bad) engines like AWS Athena for something more manageable.
Don't get me wrong, I'm just being a tongue-in-check egotistical bastard data engineer from hell. DuckDB is a fine piece of software as it is, and those mantainers deserve heaven.
This is one of the ideas behind using DuckDB in github.com/spiceai/spiceai
Looks very cool! I will take a look, tysm!
Re:Airport for DuckDB
by anentropic on Friday May 23, 2025 @07:41AM
(#44071930)
That looks like an amazing "swiss army knife"...!
Fivetran tried to upstream write support but it was not accepted
https://github.com/duckdb/duckdb-iceberg/pull/95
That sounds less "not accepted" and more "will implement, rewrite required". It was only a couple months ago.
same here man, ended up going with trino explicitly for writing and data management and using chdb/duckdb to process data for front-ends etc (mostly ethereum data so chdb "support" for ui256 is quite important)
Not just for building a new one, it can also complement existing data-warehouse/lakehouses: https://github.com/buremba/universql
The flight extension is excellent as it removes the need to write C++ extensions and lets you use your favorite language to develop native DuckDB catalogs. It's straightforward to build data lake connectors and plug them in as a flight catalog, thanks to Airport!
I'm curious, did you consider delta tables? Pretty sure duckdb supports them nicely. If you did, how come you chose not to go with them?
That is until I found out it does not support Iceberg writes[1], big nono as I would need another engine for inserts, and I want a simple stack :(. What a bummer.
[1] https://github.com/duckdb/duckdb_iceberg/issues/37
*that is what they are called now aren't they? I just can't follow the terms anymore haha.