SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by seunosewa on Tuesday April 11, 2023 @02:22PM (#35529093)

Not exactly. For performance (I guess) each worker fetches an extra job while it's working on the current job. If the current job happens to be very long, then the extra job it fetched will be stuck waiting for a long time.

Your multiple queue solution might work but it is most efficient to have just one queue with a pool of workers where where each worker doesn't pop a job unless it's ready to process it immediately. In my experience, this is the optimal solution.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by mannyv on Tuesday April 11, 2023 @02:27PM (#35529170)

It's actually a misconfiguration (see the comment below with the documentation).

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by yarg on Tuesday April 11, 2023 @08:36PM (#35533636)

Better to post a link - the order of comments changes with votes and time.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by quintes on Tuesday April 11, 2023 @03:41PM (#35530397)

I used rabbit many many years ago but agree, scale consumers and only pop when ready to actually process

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by whakim on Tuesday April 11, 2023 @09:14PM (#35533853)

I think this is definitely optimal if your jobs take a long time (and if you add in worker autoscaling of some type, you can deal with quick jobs reasonably well too). But if you have a very large number of quick jobs, setting a high prefetch limit can increase throughput by a tremendous amount. It's in this context that multiple queues makes a lot of sense; you can optimize some workers to read from one queue and others to read from another.

[ Parent | on HN ]