News for hackers. Stuff that matters.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @04:35PM (#41285276)
I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq

It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by GordonS on Sunday August 18, 2024 @05:12PM (#41285552)
Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @09:31PM (#41287030)
Not today. It's a work in progress! There are several iterations that I'm working on:

1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication

However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently.

Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by justinclift on Sunday August 18, 2024 @09:08PM (#41286936)
Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @09:27PM (#41287015)
It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull.

The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.

Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by agrothberg on Sunday August 18, 2024 @10:40PM (#41287325)
SQS does offer long polling, which looks closer to "push" semantics.
Re:Show HN: Transform Your PostgreSQL into a Powerful Job Queue by memset on Sunday August 18, 2024 @11:18PM (#41287503)
Fair enough. I do implement long polling!