Re:Parallel streaming in Haskell: Part 1 – Fast, efficient, ...
by unhammer on Thursday January 05, 2023 @05:00AM
(#34257875)
> makes me think that there aren't any conduits that have more data than can fit in memory on a single machine
Just because you can dedup a conduit doesn't mean you have to. We use conduits for streaming gigabytes-large files while staying within megabytes of memory use – ensuring that is one of the main selling points of libraries like conduit: https://github.com/snoyberg/conduit#readme
Just because you can dedup a conduit doesn't mean you have to. We use conduits for streaming gigabytes-large files while staying within megabytes of memory use – ensuring that is one of the main selling points of libraries like conduit: https://github.com/snoyberg/conduit#readme