News for hackers. Stuff that matters.
Re:When your data doesn’t fit in memory: the basic techniques by zneveu on Monday November 11, 2019 @05:00PM (#21509354)
Another easy solution I didn't see mentioned is to create swap space on Linux. This obviously isn't the fastest solution, but setting up 128GB of swap space let's me mindlessly load most datasets into memory without a single code change.
Re:When your data doesn’t fit in memory: the basic techniques by olavgg on Monday November 11, 2019 @05:24PM (#21509557)
Adding a 280GB Optane as swap is very efficient and cheap. It is still a lot slower than RAM though. But much much faster than NVME ssd's
Re:When your data doesn’t fit in memory: the basic techniques by p1esk on Tuesday November 12, 2019 @02:07AM (#21512121)
Are you talking about Optane NVDIMM or NVME?
Re:When your data doesn’t fit in memory: the basic techniques by yummypaint on Monday November 11, 2019 @05:06PM (#21509409)
Do you have a sense of how the performance would compare to chunking? My naive expectation is that loading data from disk to swapped memory involves writing that data to disk (even if it will only be read).
Re:When your data doesn’t fit in memory: the basic techniques by wongarsu on Monday November 11, 2019 @05:37PM (#21509672)
Chunking can speed up processing even if the dataset fits into memory because you are interleaving disk reads and computation and the OS is likely to prefetch the next chunk into the read cache while you're still busy computing.

On other problems chunking doesn't work at all and just mmaping or dedicating giant amounts of swap are better strategies. It depends on the problem at hand

Re:When your data doesn’t fit in memory: the basic techniques by edoo on Monday November 11, 2019 @10:18PM (#21511254)
Using mmap with the right flags should let you load and process huge files as well as if they were in ram.
Re:When your data doesn’t fit in memory: the basic techniques by Rafuino on Monday November 11, 2019 @06:25PM (#21510027)
Plus, swap performance is being improved bit by bit, so it's not as much of a dirty word as it was before.

https://lwn.net/Articles/717707/

Re:When your data doesn’t fit in memory: the basic techniques by oxplot on Monday November 11, 2019 @07:14PM (#21510297)
This! It's simple, performant depending on application and costs zero in extra development.