News for hackers. Stuff that matters.
Re:An introduction to RabbitMQ by vel0city on Thursday May 21, 2020 @11:24AM (#23259506)
I've been running RabbitMQ on pretty small VMs for a long time. RabbitMQ doesn't need a lot of resources per message, even with very small VMs (512MB RAM, single CPU) I've seen it handle peaks of many thousands of messages a second without running into problems. Give it a bit of beefy hardware and it'll probably handle whatever load you were thinking, unless you're saturating 10gig links with messages or something.

RabbitMQ and Kafka are very different struggles when thinking of scaling and performance. Kafka is almost a database itself of messages which have routed through the system. In many configurations clients can come back and demand to replay the message stream from almost any point in time. This means you need to handle _a lot_ of disk and memory access. With RabbitMQ, messages are traditionally very ephemeral. Once a message has been ack'd, its gone. Poof. Not in memory. Not on disk. Nobody is going to come back asking for that message. This leads to a lot more efficiency in handling things per message, but at the cost of not being able to remember the messages that went through the system a few milliseconds ago.

Re:An introduction to RabbitMQ by aeyes on Thursday May 21, 2020 @11:39AM (#23259725)
CPU usage highly depends on the number of connected clients, not that much on message throughput. You can experiment with the excellent rabbitmq-perf-test tool to get some ballpark numbers.

I have a system that only pushes 5k messages per second but it needs 32 cores.

Re:An introduction to RabbitMQ by nacs on Thursday May 21, 2020 @11:55AM (#23259972)
What’s the amount of connected clients around for that 32 core setup?
Re:An introduction to RabbitMQ by aeyes on Thursday May 21, 2020 @11:57AM (#23260007)
around 3000 connections, 2000 queues, 1K message size
Re:An introduction to RabbitMQ by airfreak on Thursday May 21, 2020 @12:06PM (#23260154)
Yeah that sounds about right. Of course if you had 200 connections and 50 queues you'd more likely be seeing 100000 msg/s. The number of connections and queues has a big effect on total throughput.