When your data doesn’t fit in memory: the basic techniques

Re:When your data doesn’t fit in memory: the basic techniques by dmurray on Monday November 11, 2019 @09:35PM (#21511038)

Chunking and streaming are different things to me. Chunking means you get to process multiple rows of data at the same time, usually useful to take advantage of SIMD. Streaming means that the data is accessed in a single pass: once you compute the effect of a given row on your statistics, you never have to rewind to see it again.

Many modern performant solutions will use both, but they're not the same thing.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by nine_k on Tuesday November 12, 2019 @12:01AM (#21511653)

One important application of chunking is efficient I/O.

Mass storage is most efficient when doing large sequential reads and writes, so you normally feed your constant-space streaming algorithms from buffers with a large number of input records.

Sometimes you can just tell the OS do efficient chunking prefetch for you.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by hofstee on Tuesday November 12, 2019 @05:45AM (#21513106)

If you're streaming in a language like Python, its IO will be doing some degree of chunking behind the scenes. It might be beneficial to do more manually.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by Sean1708 on Tuesday November 12, 2019 @08:38AM (#21513951)

Often called buffering when applied to IO.

[ Parent | on HN ]