News for hackers. Stuff that matters.
Re:System design hack: Postgres is a great pub/sub and job s... by MuffinFlavored on Friday November 08, 2019 @02:40PM (#21485918)
What if your listeners crashed/were down at the time of the `PUB` message? Does this mean the message falls into oblivion (since it will never receive a reply/ACK/get worked on)?
Re:System design hack: Postgres is a great pub/sub and job s... by privateSFacct on Friday November 08, 2019 @07:05PM (#21488358)
Is the activity wrapped in a transaction?

If so if worker crashes you get a connection timeout on postgresql and it rolls back the transaction and releases row locks - then next select SKIP picks it up.

If you want fast response with safety listen and on restart do a poll always to pick up anything that's ready that you might have missed a pub for. If you want lots of coverage do a poll every 10s - postgresql can handle it, you still get immediate response by listening to notify.

If you need fancier approaches you can do a last_updated time on the job and or watch waiting on pg_stat_activity or similar etc - but all totally overkill I think - the simple approach get's pretty far (not an expert though at all).

Re:System design hack: Postgres is a great pub/sub and job s... by colinchartier on Friday November 08, 2019 @02:42PM (#21485936)
The pattern I follow is that workers immediately poll once when they start - this means that when your workers restart after crashing, they'll pick up any missing jobs.
Re:System design hack: Postgres is a great pub/sub and job s... by pm90 on Friday November 08, 2019 @02:58PM (#21486098)
I'm not seeing how this recovers from the failure scenario where a worker grabbed an event and then immediately crashed.
Re:System design hack: Postgres is a great pub/sub and job s... by Rapzid on Friday November 08, 2019 @03:07PM (#21486175)
The comment you are replying to wasn't addressing that scenario.

Typically you would implement visibility timeouts and other such stuff. Depending on the use case you could make specific optimizations or keep it generic and have SQS like semantics or something.

Re:System design hack: Postgres is a great pub/sub and job s... by disiplus on Friday November 08, 2019 @03:29PM (#21486379)
at this point why not use something like rabbitmq and not reinvent it in postgres.
Re:System design hack: Postgres is a great pub/sub and job s... by stickfigure on Friday November 08, 2019 @03:47PM (#21486542)
Because it's trivially easy. The dequeue operation updates the last_lease time; the query makes sure you don't dequeue something that was recently leased. It's far, far easier than setting up a separate job queue, especially if you are already using an RDBMS.
Re:System design hack: Postgres is a great pub/sub and job s... by rapind on Friday November 08, 2019 @03:46PM (#21486533)
Because then you're running and maintaining another service. Might make sense, but depends.
Re:System design hack: Postgres is a great pub/sub and job s... by tjungblut on Friday November 08, 2019 @03:07PM (#21486170)
You need some timeout and/or leasing mechanism to make it work in this scenario.