Re:I reckon your message broker might be a bad idea
by twic on Sunday March 30, 2014 @12:55PM
(#7496813)
> If you're pushing messages to a message broker, you can trivially regulate delivery on most of them by checking queue depth and throwing errors or reducing the delivery rate if they grow beyond a certain threshold.
I'm not sure what you mean. Are you talking about regulating messages going from clients to the broker? I was thinking about regulating messages from the broker to the workers. I haven't seen a broker which will let you regulate the flow on a per-consumer basis like that.
> I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.
It's one less hassle in the control plane, and one more in the worker. I would prefer to have this stuff in the control plane rather than the worker, since it's not something directly concerned with the work being done. Separation of concerns and all that.
> With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken
Yes. Exactly. I would like the system to do something sane in the face of broken workers.
> why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?
That's quite a bit more work than just taking it out of the pool. And killing the server isn't necessarily the right answer - if the instance is locking up because it's going a full garbage collection, for instance, the best thing is probably to let it finish and then put it back in the pool.
> What is the difference between monitoring for 500 errors and monitoring for errors from an MQ?
500 errors are thrown straight into the client's face. The client itself can log that something went wrong. The closest analogue with an MQ would be for the broker to log when a consumer rejects a message. Can you configure them to do that? I honestly don't know. You'll struggle to have the logs tell a story which makes as much sense, though, because the MQ doesn't have the context that exists in the sending app.
Don't get me wrong, i don't think message queues are worthless. There are problems for which they are indisputably the right solution. But i think people should be aware of the operational challenges in using them. I have seen a lot of naive enthusiasm for queues which doesn't seem to take this into account.
I'm not sure what you mean. Are you talking about regulating messages going from clients to the broker? I was thinking about regulating messages from the broker to the workers. I haven't seen a broker which will let you regulate the flow on a per-consumer basis like that.
> I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.
It's one less hassle in the control plane, and one more in the worker. I would prefer to have this stuff in the control plane rather than the worker, since it's not something directly concerned with the work being done. Separation of concerns and all that.
> With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken
Yes. Exactly. I would like the system to do something sane in the face of broken workers.
> why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?
That's quite a bit more work than just taking it out of the pool. And killing the server isn't necessarily the right answer - if the instance is locking up because it's going a full garbage collection, for instance, the best thing is probably to let it finish and then put it back in the pool.
> What is the difference between monitoring for 500 errors and monitoring for errors from an MQ?
500 errors are thrown straight into the client's face. The client itself can log that something went wrong. The closest analogue with an MQ would be for the broker to log when a consumer rejects a message. Can you configure them to do that? I honestly don't know. You'll struggle to have the logs tell a story which makes as much sense, though, because the MQ doesn't have the context that exists in the sending app.
Don't get me wrong, i don't think message queues are worthless. There are problems for which they are indisputably the right solution. But i think people should be aware of the operational challenges in using them. I have seen a lot of naive enthusiasm for queues which doesn't seem to take this into account.