Re:Turning PostgreSQL into a queue serving 10k jobs per seco...
by bloody-crow on Thursday November 14, 2019 @01:31PM
(#21537958)
TIL. This is very useful. I believe 9.4 was the latest version around the time the article was published.
I'm glad the state of things has improved, since I really love database-backed job queues and all transactional guarantees it gives you.
In particular, Que’s new design locks jobs in a single connection per worker process, not a connection per-job. Job assignments are handled in an in-memory/unlogged table. Jobs are also automatically assigned to a free worker upon being enqueued via LISTEN/NOTIFY and an unlogged table of available workers. And jobs are locked in batches by each worker process, not one-at-a-time. So polling becomes much less frequent and is far more efficient when it happens.
This does not mean that it’s suddenly ok for jobs to take out transactions which run a long time (that is usually a bad idea in a production database) but it does substantially minimize the scenarios where these problems might occur.