cmatta

This is a pretty self-selecting group, so I'm not surprised that most people reading this don't have a problem with after-hours coms. If you've ever worked in hospitality or retail, you'll know that managers will call/contact you at all hours to make sure they have coverage. It's irritating.

Kafka does give you a counter, it's in Neha's blog post. Kafka's idempotent producer registers itself with the brokers with a unique producer id and includes a sequence number with each message it sends. The brokers simply keep track of producer id + the highest sequence number they've seen, and return an ack, if the ack is lost the producer retries and the broker knows to deduplicate the message, and send the ack. In the event counter example this would ensure that each event would be incremented once, even in the face of multiple failures.

Relying on another distributed system to ensure the first distributed system isn't duplicating messages sounds like a headache I wouldn't wish on anyone. (I am a Confluent employee).

Kafka has "exactly once" delivery guarantees since 0.11, I believe this will be the default in Kafka 1.0 (I work for Confluent).