News for hackers. Stuff that matters.
Re:Clojure on Hadoop: A New Hope by samstokes on Thursday November 03, 2011 @07:41PM (#3194217)
partitioned state that the runtime then handles for the user

Yes, when "wiring up" a bolt to a spout or another bolt, you can say that it should receive tuples grouped by one or more of the tuple fields. (e.g. tuples [user, num_events] grouped by "user".) Then Storm takes care of the consistent hashing needed to ensure that each instance of the bolt receives all of the tuples in a given group (e.g. instance 1 gets all events for user 42, instance 2 gets all events for user 53).

I had assumed there was no arbitrary state because of the replay semantics.

Storm itself doesn't implement replay; it relies on the ability of an external event source (e.g. a message queue) to replay tuples if needed. It just provides the ability to notify the source of whether replay is needed (by tracking the tuples as they flow through the topology). So whether or not you replay tuples is an application choice.

But yes, if you have stateful bolts and replay then you do need to make sure that processing a given tuple is idempotent.

There are no comments on this comment yet.