Pg_lakehouse: Query Any Data Lake from Postgres

Re:Pg_lakehouse: Query Any Data Lake from Postgres by nathanwallace on Monday May 13, 2024 @04:57PM (#40348331)

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.)

1 - https://github.com/turbot/steampipe

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by snthpy on Tuesday May 14, 2024 @06:27AM (#40353609)

I like steampipe but found the use of postgres a bit heavy for my use cases.

Could you make it run with pg_lite in wasm or DuckDB?

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by nathanwallace on Tuesday May 14, 2024 @06:34AM (#40353652)

In addition to Postgres FDWs, Steampipe plugins are also available as a SQLite extension [1] or a CLI export tool [2] for lighter weight use cases. (Although a local Postgres has a surprisingly small footprint!) Building plugins as DuckDB extensions would be cool too, but we haven't done that yet.

1 - https://steampipe.io/blog/2023-12-sqlite-extensions 2 - https://steampipe.io/blog/2023-12-steampipe-export

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by snthpy on Tuesday May 14, 2024 @05:09PM (#40360195)

This is awesome!

I last played with it over a year ago and I see that blog post is from Dec 2023. Looking forward to trying out the sqlite extension.

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by snthpy on Tuesday May 14, 2024 @05:11PM (#40360207)

DuckDB would be great or accessible through DataFusion!

Looking forward to see what you will build next!

[ Parent | on HN ]