News for hackers. Stuff that matters.
Re:Queues for Kafka by throw1234651234 on Wednesday September 27, 2023 @01:37PM (#37678182)
This is asking for a huge favor, but can anyone explain to me what Kafka does that RabbitMQ doesn't? Is it just scale?
Re:Queues for Kafka by wmal on Wednesday September 27, 2023 @01:42PM (#37678260)
They are conceptually different. Kafka optimizes throughput of data.

I wouldn't use Kafka for a job queue, and wouldn't use RabbitMQ for streaming data when ordering would be important.

Re:Queues for Kafka by leetharris on Wednesday September 27, 2023 @01:45PM (#37678303)
They are very different, but some people use them for similar things.

Kafka is a stream, RabbitMQ is a queue. Without getting into the details, RabbitMQ is designed to add things to a stack and pop them off when consumed. Kafka is designed to stream everything to a continuous log and anywhere can tune in when appropriate.

Re:Queues for Kafka by flashgordon on Wednesday September 27, 2023 @01:48PM (#37678351)
One way to think about it is who does the routing. My understanding of rabbitmq from 10 years ago was that RMQ pushes to connected consumers and to only 1 at a time? You'd need a fanout consumer that does more of the work. And lower throughput overall.

With Kafka it "just" keeps appending to a dumb (but huge) circular buffer. But you can have multiple consumers read off this buffer and can start any point. Downside is customers have to maintain their own offsets (in some storage) but there is now a big decoupling between producer and consumer. This contributes a large part to high throughput too (and consumers can go at their own pace -ofcourse if they are too slow they can fall off the log).

Re:Queues for Kafka by teraflop on Wednesday September 27, 2023 @02:07PM (#37678675)
> Downside is customers have to maintain their own offsets (in some storage)

Minor correction: you can maintain the offsets yourself if you want, but usually it's not necessary because Kafka can do it for you.

The abstraction Kafka provides is that for each consumer group and for each (topic, partition) tuple, your consumer object that is guaranteed not to receive messages before the last offset at which you called commit(). Internally, the committed offsets are stored in a special Kafka topic of their own.

Re:Queues for Kafka by flashgordon on Wednesday September 27, 2023 @06:39PM (#37682390)
Ah is this offset maintained by the Kafka client? (I included that as "client" as well). I thought the Kafka topic itself did not maintian any client specific offsets (how could it)? Unless this was added in recent years. Interesting tho!
Re:Queues for Kafka by facorreia on Wednesday September 27, 2023 @01:52PM (#37678417)
This blog post does a good job of explaining how they compare on different use cases:

https://eranstiller.com/rabbitmq-vs-kafka-an-architects-dile...

Re:Queues for Kafka by RomanPushkin on Wednesday September 27, 2023 @03:13PM (#37679747)
Kafka events are replayable, and represent the final state we're aiming for. You might have two exactly the same events that tell you how the state should look like. And there are batches of events. It's good for batch processing - "we're getting new portion of data to train AI model".

RabbitMQ messages are supposed to be processed/consumed/acked only once. Your app most probably won't ever get two exactly the same messages, unless you misconfigured/misued RabbitMQ. It's good for classic message processing - "used clicked something, run the job of informing subscribers that new post has been created" (because you can't send 1000 messages from a web worker thread).

Re:Queues for Kafka by bikezen on Wednesday September 27, 2023 @04:07PM (#37680534)
This hasnt been true for awhile, RabbitMQ added streams back in 2021.