News for hackers. Stuff that matters.
Re:Ask HN: Disk-backed NumPy for data analysis? by yorwba on Thursday February 16, 2023 @02:15PM (#34824157)
Use mmap to load only the region of the file that corresponds to row #2, interpret it as an array using np.frombuffer, proceed as with normal in-memory data.

This works as long as you control the file format (so you can ensure that rows are stored contiguously) and as long as you don't need too many rows in memory at the same time.

There are no comments on this comment yet.