> queue sending to dead endpoints
> circular queue problem
This is a subcategory of "app fails post-queue." Suppose you build a happy working queue, and someday somebody releases a new version of the queue consumer that isn't backward compatible and therefore is broken. How many messages will be lost on production before this caught? Do you have a way to recover those messages?
> circular queue problem
This is the queue version of an infinite loop. Suppose you have an infinite loop in your code, you'll crash the app but catch it very quickly.
Suppose queue message X calls a function which generates ANOTHER queue message X (infinite loop). This will be VERY HARD to catch and slow down the queue system progressively until its overwhelmed (likely only caught on prod).
Interesting point about things getting caught on prod, no amount of stress testing can sometimes reproduce these finicky bugs :)
Great explanations by the way but I try to avoid having too much logic spread out over different events.
Every time I work somewhere I have to play shepherd and ask the very basics:
- Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments
- Have you thought about all the new problems that might happen: queue sending to dead endpoints, circular queue problem, queue being restarted somehow (e.g. deploys) and losing messages?
- If the app fails post-queue, not surfacing the message to the user, do you have a plan to ensure somebody in engineering sees and fixes that error? And then goes back and remediates the broken request(s)?
- Have you prepared code/logs to do distributed tracing?
- If there's a dispute a week from now whether Joe didn't get an email because of a problem BEFORE or AFTER the queue, will you be able to tell from the logs?
Many powerful engineering abstractions (threads, async, services) require one notch higher of engineering talent and allows for all sorts of new failure paths. The tradeoff must be taken very seriously. Most places I have worked at adopted complexity too soon.