News for hackers. Stuff that matters.
Re:Reliable event dispatching using a transactional outbox by ryanjshaw on Saturday February 04, 2023 @04:22PM (#34658427)
Maybe you are using different terminology, because IMO you don't need "cross-service transactions to publish the event only-once"

"Exactly-once" message processing is what you typically want in business operations, and that is straightforward to achieve with "at-least-once" message delivery:

  1. send(message with unique .id):
    a. insert into DB queue where not exists message.id

  2. dispatcher:
    a. fetch messages from DB queue where unpublished = 1
    b. publish message to integration queue [at-least-once-delivery]
    c. update message set unpublished = 0 (and ideally record some audit information e.g. the sent timestamp and on Kafka save the message offset!)

  3. receive(message):
    a. check if message.id has been processed [exactly-once-processing]
    b. perform operation, including linking it back to message.id, in a single DB transaction
If failure occurs:

  1. between 2.a and 2.b, you fetch messages again and nothing is lost

  2. between 2.b and 2.c, you republish the same message; 3.a ensures exactly-once *processing*

  3. between 3.a and 3.b, you retry the message (with IBM MQ, you use transactions or peek then read;  with Kafka, you don't update offsets until 3.b completes)

  4. on the integration queue: set unpublished = 1 on all messages published sent in the timeframe where integration queue lost data
In practice: SQL DBs will surprise you with lock and concurrency semantics; I suggest using something battle tested under high load, or an event store database

There are no comments on this comment yet.