News for hackers. Stuff that matters.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by cklee on Sunday August 18, 2024 @04:26PM (#41285205)
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban

Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while maintaining compatibility.

Alternatively, we could consider developing a core job queue library in Rust, with language-specific bindings. This would provide a robust, cross-language solution while leveraging the performance and safety benefits of Rust.

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by rileymichael on Sunday August 18, 2024 @05:51PM (#41285821)
If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by tmountain on Monday August 19, 2024 @06:15AM (#41289632)
Also, pgmq can run as a TLE (trusted language extension), so you can install it into cloud hosted Postgres solutions like Supabase. We're using pgmq, and it's solid so far.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by vantiro on Monday August 19, 2024 @09:20AM (#41290819)
What I like about https://github.com/tembo-io/pgmq is that it can be used with any programming language and does not require any background workers or running a binary in addition to postgres.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @04:35PM (#41285276)
I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq

It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by GordonS on Sunday August 18, 2024 @05:12PM (#41285552)
Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @09:31PM (#41287030)
Not today. It's a work in progress! There are several iterations that I'm working on:

1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication

However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently.

Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by justinclift on Sunday August 18, 2024 @09:08PM (#41286936)
Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @09:27PM (#41287015)
It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull.

The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by agrothberg on Sunday August 18, 2024 @10:40PM (#41287325)
SQS does offer long polling, which looks closer to "push" semantics.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @11:18PM (#41287503)
Fair enough. I do implement long polling!
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by earthnail on Sunday August 18, 2024 @05:31PM (#41285671)
This would be so immensely useful. I’d estimate that there are so many cases where the producer is Node or Rails and the consumer is Python.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by mind-blight on Sunday August 18, 2024 @08:27PM (#41286715)
This is the exact use case I'm running into right now. I've been looking at BullMQ some it has good typescript support, and is working towards a 1.0 for python. But, I have tried it out in a production stack yet
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by jmvoodoo on Sunday August 18, 2024 @09:37PM (#41287067)
We have been using bullmq in production for just over a year. It is a piece of technology that our team doesn't have to think about, which is pretty much all I could ask for.

We did end up adding some additional generics which allows us to get strong typing between producers and consumers. That I think has been a key piece of making it easy to use and avoiding dumb mistakes.

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by bgentry on Monday August 19, 2024 @01:46AM (#41288140)
River ( https://riverqueue.com ) is a Postgres background job engine written in Go, which also has insert only clients in other languages. Currently we have these for Ruby and Python:

https://github.com/riverqueue/riverqueue-ruby

https://github.com/riverqueue/riverqueue-python

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by stephenr on Sunday August 18, 2024 @05:38PM (#41285727)
Qless "solves" this problem (in redis) by having all core logic written as lua and executed in redis.

You could take a similar approach for pg: define a series of procedures that provide all the required functionality, and then language bindings are all just thin wrappers (to handle language native stuff) around calls to execute a given procedure with the correct arguments.

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by AlphaSite on Monday August 19, 2024 @12:31AM (#41287831)
River is my go to in Golang, it’s really handy to have transactional queuing with a nice little ui.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by kraihx on Monday August 19, 2024 @06:00AM (#41289539)
For JavaScript and Perl there is already Minion, which relies on listen/notify + FOR UPDATE SKIP LOCKED.

https://github.com/mojolicious/minion.js

https://github.com/mojolicious/minion

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by ukd1 on Monday August 19, 2024 @11:10AM (#41291711)
A common schema is one nice thing, but imho the win of these db backed queues is being able to do things, including enqueue background jobs in a single transaction. e.g. create user, enqueue welcome email - both get done, or not - with redid-based, this is ... not usually a thing; if you fail to do one, it's left half done, leading to more code etc

p.s. I maintain a ruby equivalent called QueueClassic