News for hackers. Stuff that matters.
Re:Dataflow/Beam and Spark: A Programming Model Comparison by andrioni on Thursday February 04, 2016 @05:19AM (#11033031)
What do you think about Cloudera's Spark backend to Google Cloud Dataflow pipelines[1]?

[1]: https://github.com/cloudera/spark-dataflow

Re:Dataflow/Beam and Spark: A Programming Model Comparison by jkff on Friday February 05, 2016 @11:05AM (#11042268)
Hi, I'm also an engineer on the Dataflow team. This is a very exciting project! However it is still constrained by Spark's limitations outlined in this post in terms of what Dataflow pipelines it can run. For example, currently it doesn't support session windows, it windows based on processing time instead of event time, and it doesn't seem to support triggers (the word "trigger" doesn't appear in the code base). AFAIK Spark has plans to add support for event time, and then it will be possible to make this runner run more pipelines consistently with the semantics of the Dataflow model. It is also likely that the capabilities of this runner will expand greatly as https://wiki.apache.org/incubator/BeamProposal progresses.