Re:Go Python, Go: Stream Processing for Python
by ericfrederich on Thursday October 12, 2017 @10:10AM
(#15458014)
Thanks, I think it does. I did a quick search to see if or how Dask handles streams. It seems trivial to add queues to dask.
https://www.youtube.com/watch?v=hfmwXSeM8pk
I've never actually played with Dask at all for even batch processing let alone stream. If I ever get some free time I'll try to implement Word Count in Dask and see how the two codes compare.
Re:Go Python, Go: Stream Processing for Python
by happy-go-lucky on Thursday October 12, 2017 @10:20AM
(#15458105)
> Stream processing and batch processing are complimentary
Must be complementary, I think.
Re:Go Python, Go: Stream Processing for Python
by spooneybarger on Thursday October 12, 2017 @10:30AM
(#15458194)
Indeed. I regularly mix the two up. Thanks. Editing to fix.
Dask is very "batch" oriented, which as I said above, is something we are in the process of adding to Wallaroo. Wallaroo is very stream processing oriented. Wallaroo's strength are working with stateful, event-by-event applications.
If you were to take word count as an example. Dask would be great if you had a body of text in files or whatnot that you needed to count. There's a beginning and end to that task. Count the words in this text. Wallaroo would shine if you had a stream of never ending text, like twitter's trending topics.
That's a very coarse outline of a couple of differences. While we are working with clients to help them move off of Dask by adding that functionality to Dask, I also think that if you wanted to, you could use Wallaroo along side a more batch oriented system like Dask. Stream processing and batch processing are complementary. A number of technologies (us included) are looking to unify them. Why? Well, there's a lot of operational overhead to running a batch system and a streaming system. A lot of folks would like to run a single system that works well for both.
I hope that answers your question.