Devious SQL: Message Queuing Using Native PostgreSQL

Re:Devious SQL: Message Queuing Using Native PostgreSQL by kumarvvr on Saturday January 29, 2022 @08:59PM (#30133040)

How scalable do you think your system is? Can it handle 10k req/sec?

My usage is much lower than that but I want to target that number.

[ Parent | on HN ]

Re:Devious SQL: Message Queuing Using Native PostgreSQL by bfelbo on Sunday January 30, 2022 @04:25PM (#30141279)

We're currently keeping it as simple as possible. We just use a table and this query that runs every second: "DELETE FROM job_queue RETURNING type, info;"

Works great and reliably so far. I think our current volume is on the order of 100-1000 req/sec. We'll likely switch to graphile-worker when we need more performance, but we're all about avoiding premature optimization. That library has been benchmarked to handle 10k req/sec: https://github.com/graphile/worker#performance

[ Parent | on HN ]

Re:Devious SQL: Message Queuing Using Native PostgreSQL by pritambarhate on Monday January 31, 2022 @07:55AM (#30147387)

PgSQL table based job queues should be able to handle that kind of load, but to support it you will need very high IOPS based disk. If you have NVME SSD paired with decent CPU, it should be able to manage this.

This is mainly because PgSQL based queue is durable by default and based on how you implement your inserts, every new message will need a disk based commit.

[ Parent | on HN ]