News for hackers. Stuff that matters.
Re:Stream Processing for Go by joshlemer on Thursday January 18, 2018 @03:30PM (#16181079)
Does this support advanced windowing and watermarking strategies like are found in Flink?
Re:Stream Processing for Go by spooneybarger on Thursday January 18, 2018 @03:36PM (#16181134)
You can do event based windowing now, but you need to build it up from primitives. Time based windows aren't supported yet, they are however on our roadmap. We plan on adding windowing APIs to make it very easy to do a variety of windowing.

You can learn more about the existing event based windowing approach that is available now here: https://blog.wallaroolabs.com/2017/11/non-native-event-drive...

---

Can you elaborate on what specifically you are referring to as "watermarking strategies" in Flink? We are familiar with Flink's use of the Chandy Lamport algorithm but I'm not sure if that is what you are referring to.

Re:Stream Processing for Go by joshlemer on Thursday January 18, 2018 @04:12PM (#16181501)
So in flink you can use a few different bundled in strategies for emitting watermarks, or you can define your own. Here's the documentation on that https://ci.apache.org/projects/flink/flink-docs-release-1.4/...

For example, you can use the bundled MaxOutOfOrdernessTimestampExtractor to emit watermarks that lag behind the maximum timestamp seen so far by some constant amount (say, 50 ms). You could customize this behaviour by implementing the required interface, to emit a watermark, say, whenever we pass an other calendar day. You can also assign strictly-ascending watermarks which you could say is equivalent to maxOutOfOrderness = 0.

You can also choose to emit watermarks based on the content of the stream (see Punctuated Watermarks https://ci.apache.org/projects/flink/flink-docs-release-1.4/...)

Re:Stream Processing for Go by spooneybarger on Thursday January 18, 2018 @04:20PM (#16181575)
Our time based windowing has been implemented yet, so the answer would be: not at this time. It is however on our roadmap.