News for hackers. Stuff that matters.
Re:Pypeline: A Python library for creating concurrent data p... by adamcharnock on Monday September 24, 2018 @05:12AM (#18056110)

    Pypeline was designed to solve simple medium 
    data tasks that require concurrency 
    and parallelism but where using frameworks 
    like Spark or Dask feel exaggerated or unnatural.
This is exactly what I was looking for very recently. Thank you for writing this, I'll certainly look into it.
Re:Pypeline: A Python library for creating concurrent data p... by marcyb5st on Monday September 24, 2018 @06:34AM (#18056493)
What about Apache Beam? Getting started with the Python SDK has been very easy IMHO. Also, you are future proof as you can easily switch runner from Local to Dataflow/Flink/...
Re:Pypeline: A Python library for creating concurrent data p... by cgarciae on Monday September 24, 2018 @01:24PM (#18059523)
I use BEAM for my Dataflow jobs. But their local "DirectRunner" is just for testing purposes. As with Spark, BEAM is a huge beast, Pypeline was created with simplicity in mind, its a pure python library, no dependencies.
Re:Pypeline: A Python library for creating concurrent data p... by philote on Monday September 24, 2018 @09:05AM (#18057313)
But not future proof in having to use python 2.7 unfortunately.