SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by fake-name on Wednesday April 12, 2023 @03:11AM (#35536020)

I do something like this in some projects of mine.

RabbitMQ is completely happy to do this. On the other hand, literally every python client library would fall over and/or wedge forever periodically. Celery in particular was a flaming pile of garbage (at least circa 2018).

I wound up vendoring the best of the bunch (amqpstorm, fyi), and adding a bunch of additional logic to handle all the various corner cases. That, plus a bunch of careful thought about how the queues are structured, and I've been pushing millions of messages a day without issue.

I wasn't able to get a lot of my changes upstream due to the extent I had to alter the assumptions the library made about control flow.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by OOPMan on Wednesday April 12, 2023 @10:09AM (#35539927)

Don't worry, Celery has always been and will always be a flaming pile of garbage

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by xmcqdpt2 on Thursday April 13, 2023 @06:45AM (#35553502)

For what it's worth, I found rq to be much nicer than celery but only used it in a relatively constrained fashion.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by justinclift on Wednesday April 26, 2023 @10:58PM (#35722800)

rq doesn't interoperate with other languages though, so if you use it then your workers can only be written in Python. :(

[ Parent | on HN ]