News for hackers. Stuff that matters.
Re:You Cannot Have Exactly-Once Delivery by EdwardDiego on Wednesday March 01, 2023 @07:14PM (#34990233)
Yeah, non trivial is correct.

A common Kafka approach is to partition by key, so that a given UUID will only be placed on one partition, and we're guaranteed that any further messages with that key will also be placed on that partition, so handled by the same consumer.

Then create a change-log topic that's co-partitioned by key with the input topic. And then funk around with partition assignment strategies so if consumer X is assigned partition 10 of the input topic, then it's also assigned partition 10 of the change-log topic.

Then add RocksDB as a fast state store on a persistent volume claim, as restoring state fresh from your change-log topic turns out to take about 7 minutes.

And then realise you've just reimplemented bits of Kafka Streams poorly.

There are no comments on this comment yet.