Re:Parallel streaming in Haskell: Part 1 – Fast, efficient, ...
by still_grokking on Thursday January 05, 2023 @09:09AM
(#34259725)
> We'll show this in the third blog post of the series.
Thanks for the pointer! Will have a look.
For parallel processing the backpressure becomes a bit more complicated, in particular if you end with a sequential consumer. If you're just doing your parallel tasks when a sequential consumer demands it, you'll find yourself still doing one task at a time. The straightforward solution is to work ahead a bit, spawning just enough parallel work to fill a small buffer that the consumer can read from. You get backpressure this way, but may have done a little bit too much if the consumer doesn't want the rest of the data. We'll show this in the third blog post of the series.