SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by btilly on Wednesday April 12, 2023 @12:05AM (#35534881)

If they knew that some jobs would take hours and some jobs would take seconds, why would you not immediately spin up four queues. Two for the short jobs (one acting as a DLQ), and two for the long jobs (again, one acting as a DLQ). Your DLQ queues have a low TTL, and on expiration those messages get placed back onto the tail of the original queues.

Here is why I would not recommend that.

Do that and you have to rewrite your system around predictions how long each job will take, deal with 4 sources of failure, and have more complicated code. All this to maintain the complication of an ultimately unneeded queueing system. You call it an "easy fanout architecture for simplicity." I call it, "an entirely unnecessary complication that they have no business having to deal with."

If they get to a volume where they should go back to the drawing board, then they can worry about it then. This would be a good problem to have. But there is no need to worry about it now.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by pushedx_ on Wednesday April 12, 2023 @01:25AM (#35535357)

Agreed. I previously worked for a well known service provider that relied on a single SQL database for queueing hundreds of jobs per second across all regions. The system was well understood by everyone at the company and the company had a surprisingly high market cap. You can get very far with the simple thing.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by noisy_boy on Wednesday April 12, 2023 @10:13AM (#35540005)

With the additional advantage that you can run adhoc queries against the SQL database which isn't an option with RMQ.

[ Parent | on HN ]