SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by djm_ on Tuesday April 11, 2023 @03:54PM (#35530587)

This is so important if you want to avoid incredibly gnarly race conditions. In particular for us: jobs being run even before the transaction has been fully committed to the database.

We utilise a decorator for our job addition to external queues, such that the function that does the addition gets attached to Django's "on transaction commit" signal and thus don't actually get run until the outer database transaction for that request has been committed.

[ Parent | on HN ]

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by Mavvie on Tuesday April 11, 2023 @09:14PM (#35533855)

That is the simple, but unreliable way to fix the issue. If your python process crashes or is killed between sending the commit and enqueueing the job, the job will never be enqueued.

A possible solution to this is to use a "transactional outbox" pattern, but that has many of the same drawbacks of using postgres as a queue.

[ Parent | on HN ]