This philosophy, all queues have explicit tuned limits with a drop log and increment a metric on queue full, was used thoroughly at AOL in the 90s. It worked very well, and let us use hardware at much higher loadings than is popular currently. Our goal was ~90% CPU at peak (systems I have worked with more recently start to die around fifty percent). Also of course there was a meeting each Friday to look at all queue depths and queue latencies and to see where we needed more capacity coming up. We did have so many subscribers that traffic was fairly smooth over all.

> I like the author's solution (always use bounded queues) because it usually forces you to confront back-pressure up front.

I work on a system that can process a million financial transactions a second, and this principle of "always use bounded queues" has definitely made an incredible impact on the design.

We use bounded queues everywhere — and everything is also static allocation: memory, network resources, disk too, even all the messages that might possibly be needed to execute different variants of the distributed consensus protocol. It's really nice coding on it, because all the limits are clearly defined.

It forces a little more upfront thinking. This thinking can be a little difficult for a day or two, but the net result is a better design.

I love Little's law.

> and thus your system must handle the possibility that an event fails to be processed and must be retried/dropped

Isn’t a retry just extending the queue to the caller?

That's why I said "retry/drop". Handling back-pressure means you must propagate it backwards to a point in your program where there is enough context to enable decision to be made on how to handle the failure. Also, a retry can wait some amount of time until the system is less busy (and this can be enforced by some other part if needed). Further, you may have many callers and one central thing processing requests so propagating back-pressure to callers also allows you to push the problem back to a point where there are potentially more resources. In a silly example, if someone spamming your queue starts to experience a sluggish UI or run out of RAM because back-pressure has forced the queue to grow on their machine an not your infrastructure, perhaps they'll get the message and cool it (ideally the program handles this gracefully too and tells the user a maximum has been hit, but even without that back-pressure is useful).

So, yes. But that's the point.

That's the best scenario; regardless of whether the event reaches the destination or not, in most cases (except continuous monitoring of some parameter which admits losses) the source still will keep the information or the way to generate that information, so the amount of state that the queue at the source requires is still being used. But moreover, the source can throttle the generation of events, or it can just decide that that event is stale and can be dropped, or can pack that event with the next events, or can signal other processes about what is happening, or it can choose other path, or maybe it has some out-of-band signaling capability, or....

Even worse, it multiplies the amount of requests hitting the overloaded processor. You can create a "metastable failure" situation that will prevent an overloaded service from ever recovering.

It helps if your client retries have backoff (but that's not always possible), but you do need to have a way to drop requests much faster than you typically process them and it's nice to have a sensible way to drop requests that are unlikely to be successful. Sometimes you can drop based on time in queue, sometimes you can drop based on percentage, sometimes you can track backend success and drop requests to a failing backend, sometimes you have to just flush the whole queue periodically if overloaded. Facebook has published something about switching from a queue to a stack during heavy load, which can work, although that stack needs bounds, too.

> Even worse, it multiplies the amount of requests hitting the overloaded processor.

How is that? The queue goes between the "client" and the "server". The client thread does the work of trying to enqueue something, while the server continues to churn through existing requests completely independently.

"backpressure" is an unfortunate term because it's not something the queue does. "back-off" would probably be a better term, since it's the behavior of the producer. The producer has to be aware of it.

"backpressure" quite literally refers to a queue blocking when full. Exerting backpressure on a system is absolutely something a queue does. Backing off or "loadshedding" is something the producer does in response to backpressure from a queue.

The point I am trying to make is that backpressure is a contract between producer and consumer. It requires a shared understanding by the both sides.

When you put a queue in your HTTP server, you are implementing both the producer (the code accepting connections) and the consumer (the code handling requests).

TCP itself also works on a shared understanding that the producer will not send more data until it was ACK-ed. If a sender kept sending packets as fast as it can, that would be a DDOS attack, and you can't stop a network level DDOS attack by adding a queue, you can only be fast enough at discarding malicious packets. So back-pressure absolutely requires the producer to behave.

You don't even need a queue for back-pressure. Java reactive streams create back-pressure without queues by making the consumers "request" data. In fact TCP would work "fine" without a queue (i.e. with a window size of 1) it would just be slow. In a sense queues are actually used to avoid back-pressure, not to create it.

Backpressure is not a contract. A protocol is a contract and there are plenty of examples of algorithms deployed that account for the existence of backpressure. But you do not need the consumer to do anything special for a system to have backpressure.

I still don't understand your point about queues not creating backpressure. If we're getting abstract about it, everything has a queue size of 1. In the article and in this thread we're talking about infinite vs bounded queues. An ideal infinite queue has no backpressure because anything can be immediately queued. However, it comes at the unfortunate expense of infinite processing time. The difference between an infinite queue and a bounded queue is that a bounded queue, when it fills up, propagates backpressure. If you choose a queue size of 1 then whatever, it doesn't change anything conseptually. You're just trying to say a queue size of 1 is not a queue and you add a queue when you go from size 1 to > 1. And now we're just arguing semantics.

Making a bounded queue bigger adds slack and relieves pressure caused by unpredictable producers. But it does not change the maximum throughput of your system. The only thing that can do that is adding more consumers or making the existing ones faster (either by improving their speed or allowing more sophistication in the handling of requests in the queue, e.g. batch processing). If you system is already at peak load, adding a queue does not relieve pressure.

I agree with everything you just said, it's just that in my (maybe twisted) mind the principle of back-pressure can be implemented in a number of ways and adding a blocking queue in front of a system that is falling over is just one of those ways. But yeah, I guess at some level everything is a queue (or has a queue in it) so you win.

I mean it’s not about winning I think we’re mostly just saying the same thing differently at this point XD

Yes but that queue is bounded too. And if you follow the systems backward eventually you end up with a real person. That is then forced to prioritize away less important tasks.

> It's annoying but also understandable how often people just dump stuff into an unbounded queue and punt on making sure things work until the system is falling down.

It's annoying if it is done by the infrastructure team (I mean, they should know the details of the queue they are managing). It's understandable if it is done by product developers (they are more into "inheritance vs composition" kind of things).

I've seen plenty of product engineers not understand this fundamental aspect of queues and just add them because it "felt right" or "for scale" or something silly...

There's a lot of "Let's add a queue to deal with scaling issues" type thoughts which don't really work in practice. Like 95% of the time the system works better without one.

I think this is insightful thanks. We did this recently, and combined with the OP article, I'm really reminded of how fundamental and ubiquitous queues are. They aren't always obvious, or even intentional. I generally don't set out to design a queue. It just sort of happens while I solve a problem.

So yes, adding an MQ specifically just embeds another queue in your original queue. If your scaling problem is an unbounded, maybe unintentional queue, then the MQ can provide just the throttling you need to keep from overloading your consumer end.

Yep, the system just got more complicated, but now it's also more manageable, because we hacked a governor into our unregulated queue.

As discussed elsewhere, you still have to deal with the back-pressure on the producers' side.

Aren’t queues simply a requirement to do asynchronous processing? And MQs are a way to do it while keeping your application stateless, and with features to make it easier to recover from failure (e.g. persistence, DLQs).

I love discovering simpler solutions to problems! Could you explain this a bit more - how could you design things that seemingly need a queue, without a queue?

Abstractly, everything has a queue size of 1. Synchronous vs asynchronous just refers to what the producer does while its message is being processed. In synchronous programming, the producer blocks and waits until their message is processed to proceed. In async programming, the producer does other things and optionally receives a notification from the consumer once the task is complete.

How does this apply to not needing queues? I suppose you can rely on your language runtime to juggle the various jobs and concurrent threads (analogous to workers), but then you lose a lot of the benefits of having an explicit MQ system. If your system goes down, for example, you’ll lose all the in-progress work.

Actually, is that the point I was missing? That the benefits of an explicit MQ system are not always required, so it can be simpler to just rely on the async primitives of your language?

The joke I like is:

You have a problem so you implement a queue. Now you have two problems.

It succiently illustrates the problem because you should build your application to account for the queue being down. So you still have your original problem of what do I do if I can't process everything I need to.

> you should build your application to account for the queue being down

Maybe. but the whole idea is that the queuing infrastructure is intrinsically more reliable than the rest of the system. So while you may design for it, you can do so with different thresholds for severity of what the conseqeunces might be.

I've never met an application developer who was unaware of what a "queue" was or the problems they purport to solve. Pretty sure stacks/queues are among the first "data structures" that students create, which inevitably leads to the question of, "what to do when full?" i.e. circular queues, double ended queues, priority queues. I know that the enterprise grade queuing systems we're talking about are a lot more involved than that, but to suggest that developers don't grok queues is pretty disingenuous. And the implications of rate in > rate out is pretty obvious for anyone that's ever had a clogged sink.

I didn't mean queues in general, my bad. I meant, as you pointed out, enterprise grade queuing systems: there are lot of stuff going on there that are not exactly the stuff one learns in the Data Structures 101 course.

> And the implications of rate in > rate out is pretty obvious for anyone that's ever had a clogged sink.

Well, many developers I know are in their early 20s. I'm not sure they ever had to deal with clogged sinks :)

> Queues only serve to mitigate bursts.

Well, they also serve to hold your state while you add processing capacity.

If you look at a market queues, when they start to grow, management deviates people from non time sensitive tasks into cashiers. The queues make keeps things working during the change.

Equivalent situations happen on software too.

It's two sides of the same coin. In your example management only adds more cashiers if the queues have become unacceptably long (they've tripped a soft limit). The queues are serving to accommodate irregular shopper purchasing patterns because shoppers don't arrive consistently throughout the day. So really they are serving to smooth over bursts. Adding more cashiers is simply a response to queues hitting an unacceptably long length. If management had infinite cashiers and you could add them instantly then you wouldn't need queues. But, and this is especially obvious if you consider the scenario where all registers are staffed, then you still need queues to handle bursts. Costco is a good place to experience this.

I wonder which combination of HTTP reverse proxy (e.g. nginx, cloud load balancer) and application server (e.g. Python WSGI server) is best at using bounded queues and signaling backpressure (e.g. HTTP 503) rather than just letting requests pile up.

I guess that basically everything you'd consider for production use allows you to configure queue sizes and related behaviour however you fancy.