News for hackers. Stuff that matters.
Re:Farewell Pandas, and thanks for all the fish by datadrivenangel on Thursday August 29, 2024 @02:08PM (#41393649)
Dask is designed for scaling to multiple compute nodes, and is often slower than vanilla pandas on a single machine because of the overhead.

If your data is larger than memory though, at a certain point Dask is faster than pandas (which will often just fail), and DuckDB (Which is super fast for in-memory data, and still quite fast for out of memory data).

Rule of thumb these days though is that if your data isn't in the terabytes per run, a single processing node is probably best.

There are no comments on this comment yet.