Re:I wrote one of the fastest DataFrame libraries
by dTal on Sunday March 14, 2021 @08:33AM
(#26454787)
Memory mapping lazily loads the file, completes immediately, and scales arbitrarily; using disk-backed virtual memory would require the entire file to be read from disk, written back out to disk, and then read in from disk again on access; it would also require swap to be set up at the OS level, and the amount of swap set up puts a hard limit on the size of the file.
[ Parent
| on HN ]
Re:I wrote one of the fastest DataFrame libraries
by wtallis on Sunday March 14, 2021 @08:55AM
(#26454981)
You've described the difference between using mmap vs relying on the operating system's swap mechanism. But neither of those is quite the same as having an application that's aware of its memory usage and explicitly manages what it keeps in RAM. Using mmap may be useful for achieving that, but mmap on its own still leaves most of the management up to the OS.
Re:I wrote one of the fastest DataFrame libraries
by hermitcrab on Sunday March 14, 2021 @10:56AM
(#26455925)
Thanks for clearing that up!