News for hackers. Stuff that matters.
Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by knodi on Thursday April 11, 2013 @10:34AM (#5532178)
Anyone here use NSQ in production (https://github.com/bitly/nsq). If so what are your thoughts in comparison to the article?
Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by mh- on Friday April 12, 2013 @12:12AM (#5536577)
we do; rather happy with it. do you have any specific questions about it? the article's approach is.. strange, so I'm not sure how I can meaningfully contrast from a high-level.

(disclosure: i've contributed to nsq, after we began using it in production)

Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by daemon13 on Friday April 12, 2013 @02:09AM (#5536919)
Any gotchas moving from RabbitMQ to nsq?

Also, are you missing any features in nsq?

Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by mh- on Friday April 12, 2013 @09:09PM (#5542093)
didn't run into any gotchas, but I tested things very thoroughly before going into production.

to the contrary- deploying static binaries (instead of the erlang environment) simplified things nicely, in my opinion. to be fair, i (personally) don't have the requisite experience tuning BEAM which probably biases my preference.

while relatively high volume, our usage of RabbitMQ was straightforward and covered by the functionality offered in NSQ.

like most, we were using AMQP.. so the switch to NSQ's concise wire protocol (and the associated reduction in pkt/s) saved us a lot of pain given the highly-variable performance we see in the EC2 network.

Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by matticakes on Friday April 12, 2013 @12:11AM (#5536571)
there are a few (growing) number of production installations of NSQ...

our (bitly's) cluster spans a a few datacenters and hits peaks of 80k messages/second.

I can answer any questions you have (one of the authors)

Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by daemon13 on Friday April 12, 2013 @02:17AM (#5536941)
Any advice on dealing with de-duping?

Also, how was performance using JSON data format compared with ProtoBuffers & MsgPack?

Re:A quick message queue benchmark: ActiveMQ, RabbitMQ, Horn... by matticakes on Friday April 12, 2013 @09:20AM (#5538264)
NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously.

re: de-duping - there are lots of things to consider, I highly recommend reading through http://cs.brown.edu/courses/csci2270/archives/2012/papers/we..., it's a fantastic paper. At a high level the answer is idempotency.

What sort of use case are you thinking of? (context would help answer your de-dupe question)