Re:Dask – A flexible library for parallel computing in Python
by gsteinb88 on Thursday November 18, 2021 @12:51AM
(#29262366)
Just FYI, Amazon and most other providers offer instances that go up to many TB of ram — the largest listed on EC2 right now is 24TB.
Depending on your workload, you may get far better throughput not worrying about distributing the work and data
Benefits of dask.distributed: easy to get up and running, and has support for spinning up clusters on lots of different computing platforms (local machines, HPC cluster, k8s, etc.)
One difficulty is optimizing performance - there are so many configuration details (job size, number of workers, worker resources, etc. etc.) that it's been hard to know what is best.
[1] https://github.com/Dana-Farber-AIOS/pathml