Re:When your data doesn’t fit in memory: the basic techniques
by nine_k on Tuesday November 12, 2019 @12:01AM
(#21511653)
One important application of chunking is efficient I/O.
Mass storage is most efficient when doing large sequential reads and writes, so you normally feed your constant-space streaming algorithms from buffers with a large number of input records.
Sometimes you can just tell the OS do efficient chunking prefetch for you.
Re:When your data doesn’t fit in memory: the basic techniques
by hofstee on Tuesday November 12, 2019 @05:45AM
(#21513106)
If you're streaming in a language like Python, its IO will be doing some degree of chunking behind the scenes. It might be beneficial to do more manually.
Mass storage is most efficient when doing large sequential reads and writes, so you normally feed your constant-space streaming algorithms from buffers with a large number of input records.
Sometimes you can just tell the OS do efficient chunking prefetch for you.