DuckDB quacks Arrow: A zero-copy data integration between Arrow and DuckDB

Re:DuckDB quacks Arrow: A zero-copy data integration between... by 1egg0myegg0 on Friday December 03, 2021 @05:42PM (#29436551)

I am not very familiar with the Postgres Parquet FDW, but here is an educated guess!

Postgres is a row store engine rather than a column store, so I believe there will need to be quite a lot of translation for Postgres to be able to process parquet data (DuckDB and parquet are both columnar). My hypothesis is that DuckDB would be significantly faster! However, feel free to benchmark things!

[ Parent | on HN ]

Re:DuckDB quacks Arrow: A zero-copy data integration between... by 1egg0myegg0 on Friday December 03, 2021 @06:49PM (#29437115)

I should also add that there is a duckdb fdw, so you could have DuckDB read from your parquet files and do faster transformations before you pull your data into Postgres!

https://github.com/alitrack/duckdb_fdw

[ Parent | on HN ]

Re:DuckDB quacks Arrow: A zero-copy data integration between... by polskibus on Monday December 06, 2021 @04:30PM (#29465969)

This is crazy, have you measured that parquet + duckdb + postgresql is faster than parquet + postgresql?

[ Parent | on HN ]