Re:Reliable event dispatching using a transactional outbox
by ryanjshaw on Saturday February 04, 2023 @04:34PM
(#34658567)
Nice list, brings back fond memories.
On MS SQL you have a similar scenario IIRC - you can READ COMMITTED to block the query when there are dirty writes in the range of rows, but you're also encouraged in general to actually use READ COMMITTED SNAPSHOT, which won't block...
And then you get the problem where a different thread is inserting a message into the queue table, but it's taking a while, and the dispatcher is reading the queue table, and misses the uncommitted message... and never sees it again because you're using "last processed offset" as your "resume point", and you've moved past that point by the time the concurrent message insert completes. That's why I use "IsUnpublished = 1" as my flag, which also makes replay very simple to do (just create a filtered index on that column for performance reasons).
When I first started building applications like this, I eventually learned that very little emphasis was placed on distinguishing between "application events" and "integration events" -- if you are struggling with too many events, maybe this is where your issue is.
A final point - your event handler shouldn't send mail, because that is an integration action! It should queue up an email, just like you queue up an integration message. Unfortunately emails are received by humans, so they get annoyed by "at-least-once" delivery ;)
On MS SQL you have a similar scenario IIRC - you can READ COMMITTED to block the query when there are dirty writes in the range of rows, but you're also encouraged in general to actually use READ COMMITTED SNAPSHOT, which won't block...
And then you get the problem where a different thread is inserting a message into the queue table, but it's taking a while, and the dispatcher is reading the queue table, and misses the uncommitted message... and never sees it again because you're using "last processed offset" as your "resume point", and you've moved past that point by the time the concurrent message insert completes. That's why I use "IsUnpublished = 1" as my flag, which also makes replay very simple to do (just create a filtered index on that column for performance reasons).
When I first started building applications like this, I eventually learned that very little emphasis was placed on distinguishing between "application events" and "integration events" -- if you are struggling with too many events, maybe this is where your issue is.
A final point - your event handler shouldn't send mail, because that is an integration action! It should queue up an email, just like you queue up an integration message. Unfortunately emails are received by humans, so they get annoyed by "at-least-once" delivery ;)