Re:Celery – Best Practices
by denibertovic on Wednesday June 18, 2014 @09:38AM
(#7909971)
It is true that Postgres supports Pub/Sub but unfortunately the Celery broker driver does not take advantage of this. It would be great if we could get support for it. Nevertheless, just because it has pub/sub doesn't mean it's a full AMQP implementation. Also, there's the fact that most amqp solutions are in memory, wheres a database is on disk... also has it's costs.
Re:Celery – Best Practices
by denibertovic on Wednesday June 18, 2014 @09:40AM
(#7909983)
Anyone that's interested in Postgres's pub/sub might find this useful: https://denibertovic.com/talks/real-time-notifications/#/
Just slides though. Haven't gotten around to writing a post about it yet.
Yep, in Ruby there is a background processing gem built around this: https://github.com/ryandotsmith/queue_classic
Unfortunately if you're using JRuby you can't benefit from this, as the Postgres JDBC driver does polling.
but each worker will occupy a connection/session/process which are heavyweight ?
Re:Celery – Best Practices
by denibertovic on Thursday June 19, 2014 @06:34PM
(#7918647)
It is true that each worker will use up a connection. Not sure how heavy weight it is, depends on your setup and use-case I guess.
With the psycopg2 module, you can use this mechanism together with select(), so your worker thread(s) don't have to poll at all. They even have an example in the documentation.
http://www.postgresql.org/docs/9.3/interactive/sql-notify.ht...
http://initd.org/psycopg/docs/advanced.html#async-notify