Transforming Postgres into a Fast OLAP Database

Re:Transforming Postgres into a Fast OLAP Database by retakeming on Wednesday February 07, 2024 @02:34PM (#39293153)

Datafusion's SQL dialect has some slight quirks that pertain to Datafusion-specific functionality. For instance, the ability to create an external table.

With regards to ParadeDB, we rely on the Datafusion SQL parser, which can transform the Postgres SQL dialect into a Datafusion logical plan that can be executed by Datafusion. We actually have an open PR that adds support for user-defined functions...it will likely get merged within a few days.

[ Parent | on HN ]

Re:Transforming Postgres into a Fast OLAP Database by DrDroop on Wednesday February 07, 2024 @02:58PM (#39293456)

thx for your reply, so postgres is taking care of the parquet files in the background in some way i guess, would it be possible to combine this with something like neon?

[ Parent | on HN ]

Re:Transforming Postgres into a Fast OLAP Database by philippemnoel on Wednesday February 07, 2024 @04:20PM (#39294466)

We haven't tested. Neon has a pretty unique storage architecture, so I suspect it wouldn't work out-of-the-box, but could with some modifications on our end. We're open to this type of partnership eventually.

If you decide to try it, I believe they now have a way to load in arbitrary Postgres extensions!

[ Parent | on HN ]

Re:Transforming Postgres into a Fast OLAP Database by nikita on Wednesday February 07, 2024 @04:24PM (#39294497)

We are happy to try for sure. Analytics workloads really benefit from separation of storage and compute!

[ Parent | on HN ]