Pg_lakehouse: Query Any Data Lake from Postgres

Re:Pg_lakehouse: Query Any Data Lake from Postgres by samber on Monday May 13, 2024 @04:22PM (#40347948)

It seems very promising!

2 questions:

- do you distribute query processing over multiple pg nodes ?

- do you store the metadata in PG, instead of a traditional metastore?

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by retakeming on Monday May 13, 2024 @04:37PM (#40348110)

Thanks!

1. It's single node, but DataFusion parallelizes query execution across multiple cores. We do have plans for a distributed architecture, but we've found that you can get ~very~ far just by scaling up a single Postgres node.

2. The only information stored in Postgres are the options passed into the foreign data wrapper and the schema of the foreign table (this is standard for all Postgres foreign data wrappers).

[ Parent | on HN ]