Re:Fast analysis with DuckDB and Pyarrow
by 1egg0myegg0 on Sunday May 01, 2022 @12:17AM
(#31222143)
DuckDB is not only an in memory DB - it can persist data and handle larger than RAM workloads. We are expanding the number of operators that can handle larger than memory data, but it is a key design goal!
Re:Fast analysis with DuckDB and Pyarrow
by lmeyerov on Saturday April 30, 2022 @11:06PM
(#31221814)
Yep, dask_sql is probably a better comparison here (and similarly arrow-friendly). We've been having pretty good experiences here, though clearly young. For in-process, we're sticking with pandas/cudf (GPU dataframes), and likewise still Arrow for IO interop, so easy hand-off.
Isn't the whole purpose of Athena to scale to large amounts of data that don't fit into memory? How does duckdb fit in here? I thought it's an in-memory database?