Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

Re:Show HN: Riko – A Python stream processing engine modeled... by reubano on Thursday July 21, 2016 @12:09PM (#12137787)

Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4].

But since generators can only be "pulled" into one destination, you have to copy a stream (subsequently converting it into a list) if you want more than one destination [5]. This works fine if the data can fit in memory, but if it can't then you're out of luck!

[1] http://www.dabeaz.com/coroutines/copipe.py

[2] http://www.dabeaz.com/coroutines/

[3] http://www.dabeaz.com/generators/retuple.py

[4] http://www.dabeaz.com/generators

[5] https://github.com/nerevu/riko/blob/master/riko/modules/spli...

[ Parent | on HN ]

Re:Show HN: Riko – A Python stream processing engine modeled... by olviko on Thursday July 21, 2016 @12:56PM (#12138198)

Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time…

But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine, in C# it can be nicely abstracted with “async/await”, not sure how it is handled in Python, I stopped using it around 2.5

[ Parent | on HN ]

Re:Show HN: Riko – A Python stream processing engine modeled... by greenshackle on Thursday July 21, 2016 @02:35PM (#12138972)

Python 3.5 introduced the async/await syntax, I don't do C# but at a glance it's the same.

I've been working on a similar project and I've also found the push model easier.

[ Parent | on HN ]

Re:Show HN: Riko – A Python stream processing engine modeled... by reubano on Thursday July 21, 2016 @02:57PM (#12139154)

Care to share your code? I'd like to see a proper push implementation.

[ Parent | on HN ]

Re:Show HN: Riko – A Python stream processing engine modeled... by infinite8s on Friday July 22, 2016 @10:27AM (#12143698)

Another benefit of the push model is that back pressure becomes much easier to implement.

[ Parent | on HN ]

Re:Show HN: Riko – A Python stream processing engine modeled... by reubano on Friday July 22, 2016 @02:53PM (#12145820)

Any code to share? I've yet to see how to properly do push.

[ Parent | on HN ]