News for hackers. Stuff that matters.
Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by Svenskunganka on Monday October 23, 2023 @02:36AM (#37982258)
I'm not the one who wrote the original comment, so I can't modify it. But one should still commit offsets because it is the happy-path; DB transaction successful? Commit offset. If the latter fails due to e.g application crash and you seek at startup to the partition offset stored in the DB + 1, you get exactly-once semantics. There's some more details, e.g you'd have to do the same during consumer group rebalance, and topic configuration also plays a role, for example if the topic is a compacted topic or not, and if you write tombstones, what its retention policy is.
Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by mirekrusin on Monday October 23, 2023 @03:20AM (#37982487)
Yes, it's good option but it requires serialised processing in partition scope, which may or may not be desirable.

If one-and-only-one semantics are needed and processing should be parallel, other methods have to be used.

Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by robertlagrant on Monday October 23, 2023 @05:12AM (#37983283)
Presumably every exactly once processing scenario needs you to squeeze things through a serial pipe at some point, or you could have 2 messages with the same ID come in and be processed in parallel?
Re:OpenTelemetry at Scale: Using Kafka to handle bursty traffic by mirekrusin on Monday October 23, 2023 @05:45AM (#37983482)
Yes but scope/blocking/serialisation can be narrow or wide - ie. it can be per message id (highly parallel) or one for all messages of certain type/partition (not parallel).