News for hackers. Stuff that matters.

A comment on Pg_lakehouse: Query Any Data Lake from Postgres. Parent comment

Re:Pg_lakehouse: Query Any Data Lake from Postgres by philippemnoel on Tuesday May 14, 2024 @12:21AM (#40351585)
Are you using the iceberg-rust crate for Rust? It's a rather young project, have you found it sufficient for your needs (if using)?
[ Parent | on HN ]
Re:Pg_lakehouse: Query Any Data Lake from Postgres by sammysidhu on Tuesday May 14, 2024 @05:03PM (#40360147)
We're actually using pyiceberg to retrieve metadata! All our IO and decoding happens in the rust side once the data has been passthrough.

We expose something called a ScanOperator which allows integration into various catalogs through a thin layer that exposes ScanTasks.

Iceberg's impl: https://github.com/Eventual-Inc/Daft/blob/416009138359a9d410...

[ Parent | on HN ]