Re:Turning PostgreSQL into a queue serving 10k jobs per seco...
by bufferoverflow on Thursday November 14, 2019 @02:13PM
(#21538374)
What's the actual difference between a job queue and a message queue filled with job IDs?
[ Parent
| on HN ]
Re:Turning PostgreSQL into a queue serving 10k jobs per seco...
by mlyle on Thursday November 14, 2019 @02:22PM
(#21538459)
In a decent job queue you know when jobs get completed, whether the worker died, and whether they're otherwise stuck.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco...
by icheishvili on Thursday November 14, 2019 @02:56PM
(#21538774)
Message queues achieving the rate you said are typically not durable.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco...
by wjd2030 on Thursday November 14, 2019 @07:18PM
(#21541067)
The difference is nothing. A message can be a serialized object representing a job. Or the message can be a jobId that points to a record in a db.
Re:Turning PostgreSQL into a queue serving 10k jobs per seco...
by ubu7737 on Friday November 15, 2019 @03:19PM
(#21548340)
When I used PG for my job queue, I was already using Kafka instead, to handle immediate job entries. The reason I needed PG was for jobs scheduled at a future time.
I had no difficulty with long-running jobs because servicing jobs out of PG was simply a matter of pushing them onto the Kafka queue for immediate uptake there.