News for hackers. Stuff that matters.
Re:Pypeline: A Python library for creating concurrent data p... by marcyb5st on Monday September 24, 2018 @06:34AM (#18056493)
What about Apache Beam? Getting started with the Python SDK has been very easy IMHO. Also, you are future proof as you can easily switch runner from Local to Dataflow/Flink/...
Re:Pypeline: A Python library for creating concurrent data p... by cgarciae on Monday September 24, 2018 @01:24PM (#18059523)
I use BEAM for my Dataflow jobs. But their local "DirectRunner" is just for testing purposes. As with Spark, BEAM is a huge beast, Pypeline was created with simplicity in mind, its a pure python library, no dependencies.
Re:Pypeline: A Python library for creating concurrent data p... by philote on Monday September 24, 2018 @09:05AM (#18057313)
But not future proof in having to use python 2.7 unfortunately.