DuckDB quacks Arrow: A zero-copy data integration between Arrow and DuckDB

Re:DuckDB quacks Arrow: A zero-copy data integration between... by ekzhu on Friday December 03, 2021 @06:12PM (#29436806)

TLDR: Arrow got an SQL interface provided by DuckDB.

So you have a new way to run SQL on Parquet et al through DuckDB -> Arrow -> Parquet. Of course, you still need to watch out for memory usage of your SQL query if it contains JOINs or Window functions because the integration is designed for streaming rows.

[ Parent | on HN ]

Re:DuckDB quacks Arrow: A zero-copy data integration between... by TheGuyWhoCodes on Friday December 03, 2021 @07:04PM (#29437221)

You could already run SQL on Parquet with DuckDB (even on Java). I believe it already used Arrow under the hood to read the Parquet files, could be wrong tho, but this is a more memory performant integration which is great.

[ Parent | on HN ]

Re:DuckDB quacks Arrow: A zero-copy data integration between... by hfmuehleisen on Saturday December 04, 2021 @12:41AM (#29438987)

DuckDB has its own Parquet reader

[ Parent | on HN ]

Re:DuckDB quacks Arrow: A zero-copy data integration between... by infinite8s on Monday December 06, 2021 @12:43PM (#29462965)

DuckDB can read parquet directly - however the interesting bit is that the results of SQL queries can be returned as arrow objects into python for further processing (by pyarrow/pandas).

[ Parent | on HN ]