News for hackers. Stuff that matters.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by airocker on Sunday August 18, 2024 @07:36PM (#41286459)
We use listen notify extensively and it is great. The things it lacks most for us is guaranteed single recipient. All subscribers get all notifications which leads to problems in determining who should act on the message n our case.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by m11a on Monday August 19, 2024 @07:02PM (#41295297)
Could use the notify to awake, and then the worker needs to lock the job row? Whichever worker gets the lock, acts on the message
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by airocker on Monday August 19, 2024 @08:18PM (#41295637)
Thats exactly what we do but taking a lock takes 1 RTT to the database which means about 100ms. it limits the number of events receivers can handle. IF you have too many events, receivers will be just trying to take a lock most of the time.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by jeeybee on Tuesday August 20, 2024 @02:11AM (#41297259)
Of my head, you could attach a uuid or sequence number to the emitted event. Then based on the uuid or sequence you can let one or the other event consumer pick?

Ex. you have two consumers, if the sequence number is odd, A picks it, if its even B picks.

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by collingreen on Tuesday August 20, 2024 @10:24AM (#41300356)
Without more logic than this does this mean any workers that are busy or down mean you lose jobs?
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by airocker on Tuesday August 20, 2024 @10:52AM (#41300615)
We use a garbage collector to error restart if a job is not served within a specified amount of time.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by airocker on Tuesday August 20, 2024 @10:51AM (#41300611)
Great observation, I wrote a little wrongly. What we want ideally is guaranteed delivery to one random free worker. Uuid strategy is better than locking but this could mean that if one worker gets a longer job, all the others on this worker are delayed even if another worker is free.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by dopp0 on Tuesday August 20, 2024 @03:14PM (#41303023)
I think this concept you mentioned would be called sharding? It's kinda required for apache Kafka and others.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by LunaSea on Tuesday August 20, 2024 @04:56AM (#41298176)
Wouldn't it be possible for you to lose jobs if no subscriber is listening?
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by airocker on Tuesday August 20, 2024 @10:53AM (#41300627)
We use a garbage collector to error restart if a job is not served within a specified amount of time.