> We’d ignored and subsequently embraced three good design principles for reliable systems: Fail fast, Process Supervision, and the End-to-End principle.

Can't you have a message broker AND embrace those design principles... without pushing responsibility onto your crawler? What happened to the Single Responsibility Principle?