Postgres Job Queues and Failure by MVCC

Re:Postgres Job Queues and Failure by MVCC by apinstein on Wednesday May 20, 2015 @12:52PM (#9577577)

We are using Postgres as our queue backing store. I tried switching to Sidekiq but ran into issues (read here https://github.com/mperham/sidekiq/pull/624). Fortunately our job throughput is small enough to not hit any scaling issues with Postgres, so I stuck with that because of my confidence and experience w/Postgres over the years. The issues I ran into on Sidekiq just made me skeptical of their architecture/code maturity, though that was several years ago and it may be much improved by now.

We use JQJobs (which we authored) to manage queueing and it's architected such that it could be ported to Redis or some other better backing store, or potentially even to QC/Que, which I wasn't aware of until your article (so thanks for that!).

[ Parent | on HN ]

Re:Postgres Job Queues and Failure by MVCC by brandur on Wednesday May 20, 2015 @01:12PM (#9577697)

Ah, nice, thank-you!

> Fortunately our job throughput is small enough to not hit any scaling issues with Postgres, so I stuck with that because of my confidence and experience w/Postgres over the years.

I think we're in a pretty similar situation. For what it's worth, I think that a queue in PG can scale up about as well as Postgres can as long as you keep an eye on the whole system (watch out for long-lived transaction and the like).

[ Parent | on HN ]