Re:I reckon your message broker might be a bad idea
by vidarh on Sunday March 30, 2014 @07:23AM
(#7495857)
Message brokers can be very simple. My first production Ruby app was a message broker that took all of about 700 lines of code. An indeed I wrote it because I'd rather deal with something I knew intimately than deal with one of the over-engineered brokers.
But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it.
Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)
But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it.
Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)