Re:Level 3 Global Outage
by hinkley on Sunday August 30, 2020 @01:25PM
(#24324282)
I think we used to call that a poison pill message (still bring it up routinely when we talk about load balancing and why infinite retries are a very, very bad idea).
[ Parent
| on HN ]
Some queue processing systems I've seen have infinite retries.
At least they have exponential backoff I guess.
But your queue will grow and grow and the fraction of time you spend servicing old messages grows and grows.
Not a terribly big fan of these queueing systems. People always seem to bung things up in ways they are not quite equipped to fix (in the “you are not smart enough to debug the code you wrote” sense).
Last time I had to help someone with such a situation, we discovered that the duplicate processing problem had existed for >3 months prior to the crisis event, and had been consuming 10% of the system capacity, which was just low enough that nobody noticed.