News for hackers. Stuff that matters.
Re:System design hack: Postgres is a great pub/sub and job s... by ubu7737 on Friday November 08, 2019 @02:00PM (#21485513)
Used the "FOR UPDATE SKIP LOCKED LIMIT 1" trick to implement a job server in PG a few years ago for the first time.

It's a great solution.

Re:System design hack: Postgres is a great pub/sub and job s... by MuffinFlavored on Friday November 08, 2019 @02:35PM (#21485872)
Can you show what your schema for your job table looked like?
Re:System design hack: Postgres is a great pub/sub and job s... by evv on Friday November 08, 2019 @02:46PM (#21485981)
Re:System design hack: Postgres is a great pub/sub and job s... by alberth on Friday November 08, 2019 @03:46PM (#21486537)
Why the “LIMIT 1”?
Re:System design hack: Postgres is a great pub/sub and job s... by ubu7737 on Friday November 08, 2019 @06:48PM (#21488198)
It makes the job-fetching more atomic, as the article points out properly. If you take >1, you have to sort the situation where 1 or more jobs fail, which adds an unreasonable amount of complexity.

On my first go I was taking batches, on the reasoning that it optimized for query performance. It was a complete disaster and I quickly realized that I was engaging in premature optimization.

Batching is out, stream-processing is in. If you design your job table correctly, PG will perform very well as an advanced stream processor.

Re:System design hack: Postgres is a great pub/sub and job s... by aidos on Friday November 08, 2019 @03:49PM (#21486570)
Just grabbing the next available task to work on.
Re:System design hack: Postgres is a great pub/sub and job s... by philips on Friday November 08, 2019 @03:06PM (#21486160)
It is really good! I started a Go implementation for a project I am working on https://github.com/philips/pg-go-queue
Re:System design hack: Postgres is a great pub/sub and job s... by davidbanham on Friday November 08, 2019 @03:53PM (#21486613)
I should really get around to writing some docs for this, but you may find this useful:

https://github.com/davidbanham/kewpie_go

I’ve been using it for years across multiple products.

I also built this for a client lately. It lets them interop their Haskell services with kewpie a bit easier without needing to write a native wrapper.

https://github.com/paidright/kewpie-http

Re:System design hack: Postgres is a great pub/sub and job s... by codesnik on Friday November 08, 2019 @07:44PM (#21488598)
me too. I was surprised it wasn't that common at the time.
Re:System design hack: Postgres is a great pub/sub and job s... by ubu7737 on Saturday November 09, 2019 @03:12AM (#21490317)
Right, I was working mostly off one very well-written article on the topic, it was an easy search because there is really only one correct way to implement atomic job-handling with PG.