I'm not sure what you mean. Are you talking about regulating messages going from clients to the broker? I was thinking about regulating messages from the broker to the workers. I haven't seen a broker which will let you regulate the flow on a per-consumer basis like that.
> I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.
It's one less hassle in the control plane, and one more in the worker. I would prefer to have this stuff in the control plane rather than the worker, since it's not something directly concerned with the work being done. Separation of concerns and all that.
> With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken
Yes. Exactly. I would like the system to do something sane in the face of broken workers.
> why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?
That's quite a bit more work than just taking it out of the pool. And killing the server isn't necessarily the right answer - if the instance is locking up because it's going a full garbage collection, for instance, the best thing is probably to let it finish and then put it back in the pool.
> What is the difference between monitoring for 500 errors and monitoring for errors from an MQ?
500 errors are thrown straight into the client's face. The client itself can log that something went wrong. The closest analogue with an MQ would be for the broker to log when a consumer rejects a message. Can you configure them to do that? I honestly don't know. You'll struggle to have the logs tell a story which makes as much sense, though, because the MQ doesn't have the context that exists in the sending app.
Don't get me wrong, i don't think message queues are worthless. There are problems for which they are indisputably the right solution. But i think people should be aware of the operational challenges in using them. I have seen a lot of naive enthusiasm for queues which doesn't seem to take this into account.
But if you genuinely have more messages than the workers can process, pretending you don't is not solving a problem. With the broker you have the choice to decide what size of queue is acceptable without building your own queue logic into the client. The only case where HTTP is generally preferable there is the case where a queue of any size is always unacceptable: If you need the message processed right now, or not at all.
> Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself.
I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.
> Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick.
With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken - why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?
> Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem.
What is the difference between monitoring for 500 errors and monitoring for errors from an MQ? Something needs to generate those errors. Either you are monitoring the logs, or your app is sending responses directly. If you need/can use synchronous responses, either you are abusing a message broker for something it shouldn't be, or your app should be sending reply messages that the client needs to wait for (which may still be preferable if the processes are potentially very long running). There's no difference here.
> Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load.
With workers pulling work out of a message broker, they self-regulate: The ones on the highest loaded machines will pull messages slower. Load is usually the wrong thing to worry about from the app perspective - processing rate is much more relevant, and a metric that's trivial to make available from your app. Latency is easily handled by imposing strict limits and killing / pausing workers that miss them.
> You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work.
Frankly, in most of your scenarios I feel the HTTP alternative is what poses the extra work. You're imposing extra steps to manage adding workers to the load balancers; you're adding complicated load monitoring instead of letting the workers self regulate which ones have capacity to process the messages just by virtue of which ones pull the next message first.
Brokers are not always the right choice. They're generally the wrong choice if you want synchronous, low latency end to end processing. But for many systems, different parts of the system may process different types of jobs at different rates, and with different scale out requirements. In those cases, introducing brokers means each component only has to worry about their specific task, as the overall data flow management gets separated out in a separate component monitoring the brokers and taking appropriate action (such as, for example, spinning up more VMs to run brokers at some point in the pipeline).