News for hackers. Stuff that matters.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by bgentry on Thursday November 14, 2019 @12:36PM (#21537414)
While this is a great article and was completely accurate as of the time it was written, I believe it predates SKIP LOCKED, as well as some of the more recent design changes in Que which minimize the likielihood of this being an issue: https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock...

In particular, Que’s new design locks jobs in a single connection per worker process, not a connection per-job. Job assignments are handled in an in-memory/unlogged table. Jobs are also automatically assigned to a free worker upon being enqueued via LISTEN/NOTIFY and an unlogged table of available workers. And jobs are locked in batches by each worker process, not one-at-a-time. So polling becomes much less frequent and is far more efficient when it happens.

This does not mean that it’s suddenly ok for jobs to take out transactions which run a long time (that is usually a bad idea in a production database) but it does substantially minimize the scenarios where these problems might occur.

Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by bloody-crow on Thursday November 14, 2019 @01:31PM (#21537958)
TIL. This is very useful. I believe 9.4 was the latest version around the time the article was published.

I'm glad the state of things has improved, since I really love database-backed job queues and all transactional guarantees it gives you.