Pg_lakehouse: Query Any Data Lake from Postgres

Re:Pg_lakehouse: Query Any Data Lake from Postgres by nikita on Monday May 13, 2024 @09:16PM (#40350542)

I have another question. So far on the clickbench leaderboard it's 15x slower than baseline. The number 1 place is 1.67 slower the baseline.

I assume that's DataFusion speed. What's the plan to improve upon it?

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by retakeming on Tuesday May 14, 2024 @10:04AM (#40355354)

Could you clarify which result you're referring to as the baseline and "number 1 place?"

I should clarify that our published Clickbench results are from our pg_analytics extension. New results with pg_lakehouse will be released. They're going to beat the old benchmarks because 1. No overhead from Postgres transactions/MVCC, since pg_analytics used the table access method whereas pg_lakehouse is just a foreign data wrapper 2. Uses the latest release of DataFusion.

The performance differences that exist between DataFusion and other OLAP engine are rapidly becoming commoditized. DataFusion is already a world-class query engine and will only improve. pg_lakehouse absorbs all those improvements into Postgres.

[ Parent | on HN ]

Re:Pg_lakehouse: Query Any Data Lake from Postgres by riku_iki on Tuesday May 14, 2024 @03:10PM (#40358932)

Would be great to also see new pg_lakehouse and datafusion benchmark results here: https://duckdblabs.github.io/db-benchmark/

Currently Datafusion is much slower than duckdb or OOMing.

[ Parent | on HN ]