Re:Job queues are deceptively tricky
by groundzeros2015 on Thursday July 16, 2026 @05:41AM
(#48932323)
> capacity provisioning according to the max will be very costly
I’m skeptical. You can support a pretty massive messaging system with one box.
What did you try? What went wrong?
Re:Job queues are deceptively tricky
by nachexnachex on Thursday July 16, 2026 @03:54PM
(#48939429)
Similar to what gp mentioned, max latency. You can tune your consumers to burn through the queue and push messages, but when you have latency constraints e.g. a very tight deadline the first few messages in the queue might achieve it but not the 20000th.
When you want to switch from no processing to say 2M messages than need to be sent within 5 minutes it's difficult to appropriately provision without fast scale out.
Indeed you need backpressure but the traditional methods (CPU usage or similar metrics) are difficult because many consumers aren't high on those metrics --imagine a messaging plaform, pure IO. Also you'd have to tailor to the consumer itself and that's difficult, which is what you mention on the next-to-last paragraph.
In the end I helped solving it by scaling based on queue size and input/output rates, agnostic to the consumer itself, but with the hypothesis that you can scale consumers linearly (or at least monotonically, some sublinearity is allowed). The queue scaler watches for incoming and outgoing traffic on the queue, plus items on the queue itself, and it can scale from 0 to 11 in seconds for gusts, then shutting everything down.
It's a satisfying problem to work on, but its proper solution demanded quite the investigation. Now every queue we've got in the system is managed by this autoscaler -- except when we can't ensure linearity.