News for hackers. Stuff that matters.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by bufferoverflow on Thursday November 14, 2019 @01:32PM (#21537966)
Now compare to OpenAMQ, ZeroMQ, RabbitMQ, NSQ, Kafka.

I have seen benchmarks reaching millions of messages per second.

Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by madhadron on Thursday November 14, 2019 @01:56PM (#21538217)
The article is discussing job queues, not messaging. The MQ systems plus Kafka that you are referring to are message transport systems.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by bufferoverflow on Thursday November 14, 2019 @02:13PM (#21538374)
What's the actual difference between a job queue and a message queue filled with job IDs?
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by mlyle on Thursday November 14, 2019 @02:22PM (#21538459)
In a decent job queue you know when jobs get completed, whether the worker died, and whether they're otherwise stuck.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by icheishvili on Thursday November 14, 2019 @02:56PM (#21538774)
Message queues achieving the rate you said are typically not durable.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by wjd2030 on Thursday November 14, 2019 @07:18PM (#21541067)
The difference is nothing. A message can be a serialized object representing a job. Or the message can be a jobId that points to a record in a db.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by ubu7737 on Friday November 15, 2019 @03:19PM (#21548340)
When I used PG for my job queue, I was already using Kafka instead, to handle immediate job entries. The reason I needed PG was for jobs scheduled at a future time.

I had no difficulty with long-running jobs because servicing jobs out of PG was simply a matter of pushing them onto the Kafka queue for immediate uptake there.

Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by wjd2030 on Thursday November 14, 2019 @07:15PM (#21541044)
Exactly. It is so easy to achieve using a dedicated queuing system. You could just as easily achieve much higher throughput by passing a jobId as a message to Rabbit and have the worker pop the required data out of the db. Thats assuming you dont want to just pass in a serialized object. All this talk of it not being a durable system is just wrong. Rabbit has strong durability guarantees across a cluster with queue mirroring, confirmations and acknowledgements.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by ukd1 on Friday November 15, 2019 @02:05AM (#21542892)
Rabbit and most others do; but only once it reaches the queue itself. The problem that using a fb to queue stuff gets around is guaranteeing it earlier; your changes to database and the queued messages are either all committed, or not - a single transaction. This is hard to impossible with queues outside a dB. However obviously it’s a trade off; it’s slower, it’s more work and connections to your dB. But, you don’t have to code around messages either never sent it from a later rolled back transaction.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco... by baq on Thursday November 14, 2019 @04:54PM (#21539892)
i don't need millions messages per second for my job queues, i'm not google.