Re:Queues for Kafka
by flashgordon on Wednesday September 27, 2023 @06:39PM
(#37682390)
Ah is this offset maintained by the Kafka client? (I included that as "client" as well). I thought the Kafka topic itself did not maintian any client specific offsets (how could it)? Unless this was added in recent years. Interesting tho!
Minor correction: you can maintain the offsets yourself if you want, but usually it's not necessary because Kafka can do it for you.
The abstraction Kafka provides is that for each consumer group and for each (topic, partition) tuple, your consumer object that is guaranteed not to receive messages before the last offset at which you called commit(). Internally, the committed offsets are stored in a special Kafka topic of their own.