NSQ: Realtime distributed message processing at scale (in Go)

Re:NSQ: Realtime distributed message processing at scale (in... by jehiah on Tuesday October 09, 2012 @01:26PM (#4632820)

Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq

In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.

The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.

If there are any specific numbers you are curious about, please ask.

[ Parent | on HN ]

Re:NSQ: Realtime distributed message processing at scale (in... by old_sound on Wednesday October 10, 2012 @08:35AM (#4636020)

When people say "billions of messages per day" do they realize that 1 billion messages =~ 11500 msgs/sec ? RabbitMQ as well as any self respecting message broker is able to handle that on a single machine

[ Parent | on HN ]