News for hackers. Stuff that matters.
Re:Dask – A flexible library for parallel computing in Python by rosej on Thursday November 18, 2021 @12:28AM (#29262257)
We have been using dask to support our computational pathology workflows [1], where the images are so big that they cannot be loaded in memory, let alone analyzed (standard pathology whole slide images are ~1GB; some microscopy techniques generate images >1TB). We divide each image into a bunch of smaller tiles and process each tile independently. The dask.distributed scheduler lets us scale up by distributing the tile processing across a cluster.

Benefits of dask.distributed: easy to get up and running, and has support for spinning up clusters on lots of different computing platforms (local machines, HPC cluster, k8s, etc.)

One difficulty is optimizing performance - there are so many configuration details (job size, number of workers, worker resources, etc. etc.) that it's been hard to know what is best.

[1] https://github.com/Dana-Farber-AIOS/pathml

Re:Dask – A flexible library for parallel computing in Python by gsteinb88 on Thursday November 18, 2021 @12:51AM (#29262366)
Just FYI, Amazon and most other providers offer instances that go up to many TB of ram — the largest listed on EC2 right now is 24TB.

Depending on your workload, you may get far better throughput not worrying about distributing the work and data