News for hackers. Stuff that matters.
Re:From Kafka to ZeroMQ for real-time log aggregation by smetj on Saturday September 17, 2016 @03:29PM (#12522013)
100K msg/s going through RabbitMQ ... Would you mind commenting on how your Rabbitmq is setup? Is it a cluster? Distributed queue(s)? Synced queues? What kind of exchange? How many queues your messages end up in? (because 1 queue is bound to 1 core), persistent queue? lazy queue? What is the "Consumer utilisation" value when doing 100K msg/s?

I'd be really interested to hear how you can achieve such a thoughput with rabbitmq

Re:From Kafka to ZeroMQ for real-time log aggregation by agentgt on Sunday September 18, 2016 @09:19AM (#12525167)
I'm not sure how much I can help because I didn't setup the RMQ cluster so I don't know the configuration details but I know it is fairly powerful (its also partly why I can't entirely be critical of Auth0 because ZMQ is probably far less expensive infrastructure wise).

I do know we use multiple queues and even exchanges (and I did not know about the one core to queue).

A simple googling shows though folks have achieved far greater throughput[1] than 100k (and by the way this wasn't sustained.. it was spikes).

[1]: https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-m...