Re:DuckDB: Querying JSON files as if they were tables
by cweill on Friday March 03, 2023 @09:41PM
(#35017596)
If you ever need to join two large dataframes, but are OOMing on the join, write them to disk as parquet files then use DuckDB to do the join. It's amazing what you can do on one machine thanks to DuckDB.
[ Parent
| on HN ]
Re:DuckDB: Querying JSON files as if they were tables
by qolop on Friday March 03, 2023 @09:53PM
(#35017670)
This isn't unique to duckdb. Almost all databases allow for sorting and joins of large tables that don't fit into memory.