> it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology

This is not because you do not understand how message broker works that they are not simple. Message brokers implement "reliable, well-understood" protocols, and you do not need to have engineers who "specialize in understanding message broker technology". This is downright FUD. The little complexity of those protocols is here for a reason, and, seriously, we're not talking Enterprise Java complexity.

It's just like saying that you could use bash scripts to automate your infrastructure unless you have Google scale, because Puppet, Chef, or Ansible are too complex and over-engineered.

Writing a database-backed RESTful service for a message queue is not without reminding me of those who reimplement RDBMS over MongoDB because RDBMSes are "over engineered".

If the AMQP protocol is too complex for you, there are other, simple protocols that message brokers implement.

Message brokers can be very simple. My first production Ruby app was a message broker that took all of about 700 lines of code. An indeed I wrote it because I'd rather deal with something I knew intimately than deal with one of the over-engineered brokers.

But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it.

Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)

They're just another one of the numerous examples of enterprise software out there. Overgeneralised and overabstracted, promising everything and probably capable of it, but not doing any one thing particularly well while consuming inordinate amounts of both machine and human resources... and they somewhat mirror the bureaucracy of the companies that use them.

Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.

I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems.

Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.

Do you find that the message broker makes that implementation easier/better than just using a database table to store the incoming messages until they can be processed?

I've worked on enterprise products, off-the-shelf and in-house, which use message brokers, and every time I could not fathom how the extra complexity was justified. If the messages were just queued in a simple database table instead, the implementations would have been much easier to work with: easier setup and admin (it's just part of the existing database), easier to check and monitor (just select counts from the table), easier to inspect the pending messages and fix problematic messages. In all cases I get the feeling the architects were just looking for an excuse to use a new technology.

I can only see high-load scenarios as a possible justification, where the volume of incoming message data is physically too much for the RDMS.

A database table is just a storage medium. The programming model is what is important. Some tasks don't fit the imperative model very well, some tasks have transactions open for days, some tasks need correlation across hundreds of different integrations, some tasks need to happen in guaranteed windows after initiated, some tasks need to be parallelised across lots of systems. These are a few cases I can think of.

From a current position perspective, we use MQs for quoting. Someone raises a quote, this is parallelised across 20 providers all with different integration methods, transformed back to a common object model, inserted as a single transaction into a database table, handed over to another system as an integration task which calculates risk, posted back from that to our system, a risk model is applied, the data is updated, the user is notified. This is all plugged into a 2 million line java and c# jumble that evolved over 15 years from a C++/COM nightmare.

We rewrote it with full test coverage in a month. We can add a new provider in an hour.

Imperative/table integration. No thanks!

People who distrust this methodology need to grab a copy of Enterprise Integration Patterns. There's a huge amount of wisdom in that book that puts Joe blogg's average dynamic language task queue to shame.

Without wishing to use the phrase, but enterprise software is about a hell of a lot of stuff people don't understand. This is probably 1% of it.

Many message brokers do use or support using a database table to store the messages.

What they tend to add is ease of scalability and failover, and higher level functionality such as some of what the author of the article complained about, like automatically marking the message as available again if the worker that was processed it did not give any liveness sign within X seconds to deal with hung workers etc.