Re:Transforming Postgres into a Fast OLAP Database
by DrDroop on Wednesday February 07, 2024 @02:27PM
(#39293040)
Interesting stuff keeps happening in PostgresLand. This uses Apache DataFusion, first think i hear about it, but from reading the docs it seems like it already offers build in SQL interface atop of Apache parquet and Apache Arrow. What is the difference between this two versions of SQL. Can I still use my favourite pl/sql function together with a deltalake table and does this play nice with other kind of tables?
[ Parent
| on HN ]
Re:Transforming Postgres into a Fast OLAP Database
by retakeming on Wednesday February 07, 2024 @02:34PM
(#39293153)
Datafusion's SQL dialect has some slight quirks that pertain to Datafusion-specific functionality. For instance, the ability to create an external table.
With regards to ParadeDB, we rely on the Datafusion SQL parser, which can transform the Postgres SQL dialect into a Datafusion logical plan that can be executed by Datafusion. We actually have an open PR that adds support for user-defined functions...it will likely get merged within a few days.
Re:Transforming Postgres into a Fast OLAP Database
by DrDroop on Wednesday February 07, 2024 @02:58PM
(#39293456)
thx for your reply, so postgres is taking care of the parquet files in the background in some way i guess, would it be possible to combine this with something like neon?
Re:Transforming Postgres into a Fast OLAP Database
by philippemnoel on Wednesday February 07, 2024 @04:20PM
(#39294466)
We haven't tested. Neon has a pretty unique storage architecture, so I suspect it wouldn't work out-of-the-box, but could with some modifications on our end. We're open to this type of partnership eventually.
If you decide to try it, I believe they now have a way to load in arbitrary Postgres extensions!