News for hackers. Stuff that matters.
Re:Announcing Apache Flink 1.0.0 by faizshah on Wednesday March 09, 2016 @11:19AM (#11253801)
I'm pretty sure that this is the algorithm flink uses: http://arxiv.org/pdf/1506.08603.pdf

EDIT: fyi sewen is one of the authors of the paper and a founder of flink

Re:Announcing Apache Flink 1.0.0 by sewen on Wednesday March 09, 2016 @11:25AM (#11253838)
You do not need physical exactly once. If you can "roll back" changes, than physical "at least once" is enough to realize logical "exactly once".
Re:Announcing Apache Flink 1.0.0 by falcolas on Wednesday March 09, 2016 @12:09PM (#11254131)
Rolling back changes means that at some point in time, you have "more than once". It also means you can only ever implement logic which can be rolled back. In a pipeline scenario, that is challenging to do without some sort of external locking system.
Re:Announcing Apache Flink 1.0.0 by sewen on Thursday March 10, 2016 @05:13AM (#11258275)
True. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: http://data-artisans.com/extending-the-yahoo-streaming-bench...

For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.