News for hackers. Stuff that matters.

A comment on NSQ: Realtime distributed message processing at scale (in Go).

Re:NSQ: Realtime distributed message processing at scale (in... by bgentry on Tuesday October 09, 2012 @12:59PM (#4632653)
This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost.

I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies.

One solution is to stand up redundant nsqd pairs (on separate hosts) that receive copies of the same portion of messages.

OK, delivery is only guaranteed if I run multiple independent sets of NSQd and write messages to both.

Regardless, it looks like an interesting project.

[ Parent | on HN ]
Re:NSQ: Realtime distributed message processing at scale (in... by jlgreco on Tuesday October 09, 2012 @01:02PM (#4632670)
How would you design a system that didn't have this problem at the individual node level?
[ Parent | on HN ]
Re:NSQ: Realtime distributed message processing at scale (in... by philjr on Tuesday October 09, 2012 @01:46PM (#4632938)
A similar design to TCP/IP - sequence numbers, acknowledgements & resending capability on sender side
[ Parent | on HN ]
Re:NSQ: Realtime distributed message processing at scale (in... by dclusin on Tuesday October 09, 2012 @01:14PM (#4632737)
Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.
[ Parent | on HN ]
Re:NSQ: Realtime distributed message processing at scale (in... by jlgreco on Tuesday October 09, 2012 @01:50PM (#4632965)
So the sender just keeps trying until it gets an ACK. Fair enough, though I imagine you'd want to run multiple nodes even with this.
[ Parent | on HN ]
Re:NSQ: Realtime distributed message processing at scale (in... by dclusin on Tuesday October 09, 2012 @03:48PM (#4633579)
Correct. Reliability and Availability are two separate but related goals.
[ Parent | on HN ]
Re:NSQ: Realtime distributed message processing at scale (in... by onetwothreefour on Tuesday October 09, 2012 @03:43PM (#4633552)
This is a solved problem in most messaging protocols.
[ Parent | on HN ]