> Downside is customers have to maintain their own offsets (in some storage)
Minor correction: you can maintain the offsets yourself if you want, but usually it's not necessary because Kafka can do it for you.
The abstraction Kafka provides is that for each consumer group and for each (topic, partition) tuple, your consumer object that is guaranteed not to receive messages before the last offset at which you called commit(). Internally, the committed offsets are stored in a special Kafka topic of their own.
Re:Queues for Kafka
by flashgordon on Wednesday September 27, 2023 @06:39PM
(#37682390)
Ah is this offset maintained by the Kafka client? (I included that as "client" as well). I thought the Kafka topic itself did not maintian any client specific offsets (how could it)? Unless this was added in recent years. Interesting tho!
With Kafka it "just" keeps appending to a dumb (but huge) circular buffer. But you can have multiple consumers read off this buffer and can start any point. Downside is customers have to maintain their own offsets (in some storage) but there is now a big decoupling between producer and consumer. This contributes a large part to high throughput too (and consumers can go at their own pace -ofcourse if they are too slow they can fall off the log).