So here's an alternative view of how he could've achieved what he wanted:

- Use a broker that supports a timeout on the messages. Many do. E.g. beanstalkd lets you reserve a message for a specified time period, otherwise it goes back on the queue automatically. You can let the server knows if you need longer to finish. But a locked process will not hold on to a job.

- Use a timeout on the crawlers. A hard timeout like alarm() that causes the process to die if it fails. Run them under a process monitor that'll respawn them.

- Have the crawler or a separate system monitor queue depth, and slow down / alert/ throw errors if the queue goes above a certain depth.

All of this is trivial.

If introducing a broker takes the whole app with it if its fails, then you're doing it wrong: Unless you need in-order guaranteed delivery of messages, making a message queue resilient to failures is trivial: Just add more of them, and let the client talk to all of them. If you need in-order delivery, then a active-passive setup is easy enough too. Put the queue in a synchronously replicated database if you can't afford to lose any of them. So his argument about failures, is an argument against letting "inexperienced programmers" set up systems with single points of failure, not against message brokers.

The irony of this is that his solution is that it's great when the process generating the messages to queue can be run on a single, reliable system, by someone who will notice when that system fails and recover from it. But you've now stuffed your crawler full of state, and created a central point of failure. Just what he complained about.

The broker solution is great when you want scale - you can multiply the number of brokers. You can multiply the number of machines the workers run on. You can multiply the number of machines processing the crawled documents. It is also great when you want resilience for the same reason: By decoupling, it makes it easier.

If you don't need that scale, then go for the simpler solution. But if you do need that scale, then his solution isn't a solution at all.

If you introduce a broker, and your queues start growing out of control. Guess what? You have a scaling problem you would likely have without the broker too. Only in a decoupled system you have a much easier way out: Add more machines/VMs with workers; add more brokers.

Don't feel you have a way of controlling it? This is what your monitoring and orchestration systems should be doing. Queues over a certain threshold? Trigger an alert, and optionally spin up an exra VM.