News for hackers. Stuff that matters.

A comment on Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5.

Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by wdobbels on Wednesday February 26, 2020 @12:22PM (#22425047)
No mention of dask? So far I haven't really had a need for larger-than-memory arrays, but dask would be my go-to choice.
[ Parent | on HN ]
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by itamarst on Wednesday February 26, 2020 @01:03PM (#22425498)
Dask, while great, doesn't solve this problem at all. You still need some way to load the data.

And for that Dask supports Zarr, and HDF5, and pre-existing arrays (where mmap() comes in), and a few other formats: https://docs.dask.org/en/latest/array-creation.html#

[ Parent | on HN ]