News for hackers. Stuff that matters.
Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by dymk on Tuesday April 11, 2023 @01:24PM (#35528230)
My hobby project does ~1.5M jobs per day enqueued into Postgres, no sweat. I use https://github.com/bensheldon/good_job which uses PG's LISTEN/NOTIFY to lower worker poll latency.
Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by avinassh on Tuesday April 11, 2023 @02:30PM (#35529224)
> which uses PG's LISTEN/NOTIFY to lower worker poll latency

Can you elaborate on how do you do this?

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by sideofbacon on Tuesday April 11, 2023 @02:57PM (#35529706)
I'm the GoodJob author. Here's the class that is responsible for implementing Postgres's LISTEN/NOTIFY functionality in GoodJob:

https://github.com/bensheldon/good_job/blob/10e9d9b714a668dc...

That's heavily inspired by Rail's Action Cable (websockets) Adapter for Postgres, which is a bit simpler and easier to understand:

https://github.com/rails/rails/blob/be287ac0d5000e667510faba...

Briefly, it spins up a background thread with a dedicated database connection and makes a blocking Postgres LISTEN query until results are returned, and then it forwards the result to other subscribing objects.

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by comboy on Tuesday April 11, 2023 @04:55PM (#35531416)
excited claps
Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by fweimer on Tuesday April 11, 2023 @04:18PM (#35530940)
These are somewhat obscure PostgreSQL SQL commands:

https://www.postgresql.org/docs/15/sql-listen.html

https://www.postgresql.org/docs/15/sql-notify.html

I think they have been around for ages, but handling the LISTEN responses may need special client library support.

Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a... by Mavvie on Tuesday April 11, 2023 @06:18PM (#35532391)
I can't speak for how they do it, but when your worker polls the table and finds no rows, you will sleep. While sleeping, you can also LISTEN on a channel (and if you get a message, you abort your sleep).

Then, whenever you write a new job to the queue, you also do a NOTIFY on the same channel.

This lets you keep latency low while still polling relatively infrequently.

NOTIFY is actually transactional which makes this approach even better (the LISTENer won't be notified until the NOTIFY transaction commits)