News for hackers. Stuff that matters.
Re:How far can you go without a message queue? by XorNot on Thursday August 25, 2022 @08:12AM (#32592628)
The problem is people leap to message queuing to make things fast, but don't build their recovery system first...but the recovery system is exactly what they need to build first. Whatever you're doing with queuing, you need to have a fallback so your service can "catch up" by pulling down what it thinks it missed.

And generally this system should not use the message queue to just get things resent - because most likely in this situation you still need to be mostly handling new messages rather then old ones when you're stuck trying to catch up.

All of which tends to be a product of just utterly magical thinking from managers and solutions architects as to how anything works, and how likely an outage is (basically guaranteed no matter what anyone says about their product).

Re:How far can you go without a message queue? by trombone5000 on Thursday August 25, 2022 @08:32AM (#32592841)
> ...but the recovery system is exactly what they need to build first.

I can't find the reference, but consistent with that, I remember reading someplace that the number of items in your queue should typically be zero, since queues can't actually make the rest of the system any faster. All they can really do is buffer temporary bursts, or buffer while the rest of the system recovers. And for that you'll definitely need your recovery system working as a prerequisite.