News for hackers. Stuff that matters.
Re:Celery – Best Practices by keosak on Wednesday June 18, 2014 @09:31AM (#7909930)
Points 1 and 2 are only valid because the Celery database backend implementation uses generic SQLAlchemy. Chances are, if you are using a relational database, it's PostgreSQL. And it does have an asynchronous notification system (LISTEN, NOTIFY), and this system allows you to specify which channel to listen/notify on.

With the psycopg2 module, you can use this mechanism together with select(), so your worker thread(s) don't have to poll at all. They even have an example in the documentation.

http://www.postgresql.org/docs/9.3/interactive/sql-notify.ht...

http://initd.org/psycopg/docs/advanced.html#async-notify

Re:Celery – Best Practices by denibertovic on Wednesday June 18, 2014 @09:38AM (#7909971)
It is true that Postgres supports Pub/Sub but unfortunately the Celery broker driver does not take advantage of this. It would be great if we could get support for it. Nevertheless, just because it has pub/sub doesn't mean it's a full AMQP implementation. Also, there's the fact that most amqp solutions are in memory, wheres a database is on disk... also has it's costs.
Re:Celery – Best Practices by denibertovic on Wednesday June 18, 2014 @09:40AM (#7909983)
Anyone that's interested in Postgres's pub/sub might find this useful: https://denibertovic.com/talks/real-time-notifications/#/

Just slides though. Haven't gotten around to writing a post about it yet.

Re:Celery – Best Practices by waffle_ss on Wednesday June 18, 2014 @10:27AM (#7910262)
Yep, in Ruby there is a background processing gem built around this: https://github.com/ryandotsmith/queue_classic

Unfortunately if you're using JRuby you can't benefit from this, as the Postgres JDBC driver does polling.

Re:Celery – Best Practices by ddorian43 on Thursday June 19, 2014 @07:25AM (#7915559)
but each worker will occupy a connection/session/process which are heavyweight ?
Re:Celery – Best Practices by denibertovic on Thursday June 19, 2014 @06:34PM (#7918647)
It is true that each worker will use up a connection. Not sure how heavy weight it is, depends on your setup and use-case I guess.