News for hackers. Stuff that matters.
Re:DuckDB: Querying JSON files as if they were tables by cweill on Friday March 03, 2023 @09:41PM (#35017596)
If you ever need to join two large dataframes, but are OOMing on the join, write them to disk as parquet files then use DuckDB to do the join. It's amazing what you can do on one machine thanks to DuckDB.
Re:DuckDB: Querying JSON files as if they were tables by qolop on Friday March 03, 2023 @09:53PM (#35017670)
This isn't unique to duckdb. Almost all databases allow for sorting and joins of large tables that don't fit into memory.
Re:DuckDB: Querying JSON files as if they were tables by cweill on Saturday March 04, 2023 @12:14AM (#35018399)
Yes but if you're in a Jupyter notebook, you may not be directly connected to a DB. If you're using pandas, this unlocks some scalability before needing dask and a cluster.