News for hackers. Stuff that matters.
Re:Devious SQL: Message Queuing Using Native PostgreSQL by bfelbo on Saturday January 29, 2022 @04:21PM (#30130608)
We’ve implemented two types of queues using Postgres tables in production and it’s been working nicely. Happy to answer questions about our usage if it’s useful :)
Re:Devious SQL: Message Queuing Using Native PostgreSQL by kumarvvr on Saturday January 29, 2022 @08:59PM (#30133040)
How scalable do you think your system is? Can it handle 10k req/sec?

My usage is much lower than that but I want to target that number.

Re:Devious SQL: Message Queuing Using Native PostgreSQL by bfelbo on Sunday January 30, 2022 @04:25PM (#30141279)
We're currently keeping it as simple as possible. We just use a table and this query that runs every second: "DELETE FROM job_queue RETURNING type, info;"

Works great and reliably so far. I think our current volume is on the order of 100-1000 req/sec. We'll likely switch to graphile-worker when we need more performance, but we're all about avoiding premature optimization. That library has been benchmarked to handle 10k req/sec: https://github.com/graphile/worker#performance

Re:Devious SQL: Message Queuing Using Native PostgreSQL by pritambarhate on Monday January 31, 2022 @07:55AM (#30147387)
PgSQL table based job queues should be able to handle that kind of load, but to support it you will need very high IOPS based disk. If you have NVME SSD paired with decent CPU, it should be able to manage this.

This is mainly because PgSQL based queue is durable by default and based on how you implement your inserts, every new message will need a disk based commit.

Re:Devious SQL: Message Queuing Using Native PostgreSQL by kumarvvr on Saturday January 29, 2022 @09:48PM (#30133373)
Would also love to hear or read about your implementation, the general architecture and failsafe constructs. Thank you.