News for hackers. Stuff that matters.
Re:Pypeline: A Python library for creating concurrent data p... by cgarciae on Monday September 24, 2018 @03:26PM (#18060621)
Hey, thanks for all the feedback. I will change the naming since its something most of you have agreed is a good change.

The goal I have for Pypeline is much simpler: let you easily setup data pipelines where you leverage processes, threads and asyncio where they are good at. So in my mind a killer app would be a pipeline that maybe starts with an asyncio stage for e.g. downloading images, maybe then a multiprocess stage for e.g. doing image processing, and finally a threading stage for e.g. interacting with the OS.

Right now I see Pypeline more as an easy to use single machine tool instead of a higher level distributed abstraction like Celery. Maybe other framework could leverage Pypeline to ease their work.

Re:Pypeline: A Python library for creating concurrent data p... by bb88 on Saturday September 29, 2018 @09:02PM (#18103921)
So then I would want to have one more stage... a celery stage for when you want to cluster work across multiple machines. :)