Re:DuckDB performance improvements with the latest release
by timost on Monday November 06, 2023 @12:42PM
(#38165846)
Interestingly, Dask runs out of memory on many of the tasks of the benchmark.
[ Parent
| on HN ]
Re:DuckDB performance improvements with the latest release
by pgorczak on Monday November 06, 2023 @01:33PM
(#38166697)
I had similar issues with the Dask scheduler a few months ago. The docs say it encourages depth first behavior in the computation graph but in my case it kept running out of memory on a large ETL task by first trying to load all the input files into memory before moving on to the next stage.