Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by mborg on Thursday February 27, 2020 @09:14AM
(#22433385)
The article leaves out a pretty important question you should ask yourself: "What fraction of the data will I need to access?"
If the answer is "much less than the whole file", accessing the data over the network makes sense.
Otherwise, you should almost certainly bring it to a local disk, where mmap shines.
If the answer is "much less than the whole file", accessing the data over the network makes sense. Otherwise, you should almost certainly bring it to a local disk, where mmap shines.