News for hackers. Stuff that matters.

ptgoetz

Re:Announcing Apache Flink 1.0.0 on Wednesday March 09, 2016 @08:23PM (#11256870)
Exactly-once in a distributed system doesn't exist, and we need to get past that. It can be approximated with replay (at-least-once) and de-dupe/idempotency.

Trying to ensure true exactly-once is a fools errand. In a distributed system it required guarantees (down to the magnetic material level) that are very hard to get right. If any of those guarantees fail, you don't have it. What you have is a close approximation.

Real world applications usually involve a lot more than counting words in memory.

At-least-once is relatively easy. Combine it with idempotent operations and your work is done.

Storm is fairly explicit in documenting what it takes to achieve this, but it's not trivial, and every system involved has to support certain guarantees.

Spark (streaming) made some pretty big claims about exactly-once guarantees, but it turned out that claim was riddled with holes.

In my opinion, "exactly-once" doesn't imply that there are exceptions to that rule.

Guard against dupes and you'll be fine (easier said than done, obviously), but also know the limitations of the systems and frameworks you are working with.

EDIT: Disclaimer: I am an Apache Storm PMC Member

Re:Announcing Apache Flink 1.0.0 on Wednesday March 09, 2016 @11:33AM (#11253906)
The Storm compatibility layer may work for word count, but not for anything non-trivial.

Here's the fine print (no fault tolerance, etc.): https://github.com/apache/flink/blob/master/flink-contrib/fl...

Re:Flying faster with Twitter Heron on Tuesday June 02, 2015 @09:35PM (#9650102)
(DISCLAIMER: I am an Apache Storm PMC Member)

This could be very good thing for Apache Storm depending on how Twitter handles it.

Just to clarify, the Storm version mentioned in the paper and blog post is not an official Apache release and doesn't include many performance improvements included in the newer releases of Apache Storm. There are a lot, and many more on the horizon.

That being said, the performance numbers look impressive, even though there is no way confirm those since no code or benchmarks have been published. IMHO, until that happens, there's not much to see here (not that I doubt it -- I'd just like to see proof/code).

My hope is that Twitter is dedicated to the projects it has open-sourced, and this is not a case of NIH, but rather an honest effort on Twitter's part to contribute back to the open source community.

@haberman:

Storm implements exactly-once processing through a higher-level API called Trident, that I like to call Storm's "Streams API" since it's not unlike Java 8's Streams API (and largely inspired by Cascading). Trident processes data in configurable micro-batches, as opposed to one-at-a-time, which gives it an advantage in terms of throughput, but at the cost of latency. Trident topologies "compile" down to Core Spout/Bolt topologies (The Trident API has a planner implementation that figures that out -- not unlike an SQL query planner).

The Storm Core API provides at-least-once semantics through an acking mechanism described here [1]. The Trident API builds on top of that to support exactly-once semantics by essentially doing a de-dupe [2].

I'm not sure exactly why they don't claim to support this, since Trident is build on top of Storm's Core API.

[1] https://storm.apache.org/documentation/Acking-framework-impl... [2] https://storm.apache.org/documentation/Trident-state.html

@filereaper:

Assuming you are referring to Spark streaming, forget about any benchmarks you may have seen. Either can be faster than the other depending on how you configure it, and what your use case is. See my presentation on the subject here [3]. With either, you can configure yourself into a corner and screw your performance.

Performance tuning distributed systems is a mysterious art. As is benchmarking. Unfortunately, that fact is frequently exploited for "benchmarketing" purposes. Don't trust any benchmark but your own unless it is fully open-sourced (including configuration).

[3] http://www.slideshare.net/ptgoetz/apache-storm-vs-spark-stre...

@vicaya

Version numbers don't necessarily equate to code quality, performance, or stability. I've seen many projects bump to 1.0 only for marketing purposes.