News for hackers. Stuff that matters.
Re:Celery 3.0 has been released by Herbert2 on Saturday July 07, 2012 @02:38PM (#4212349)
Just wanted to say thanks to Ask for Celery. Absolutely fantastic for distributed systems.
Re:Celery 3.0 has been released by rwhitman on Saturday July 07, 2012 @01:56PM (#4212248)
I like the Autechre album based version naming convention
Re:Celery 3.0 has been released by bryanh on Saturday July 07, 2012 @05:12PM (#4212742)
Celery is one of those rare libraries where improvements to itself are improvements to my sanity and well being.

That said, I did run into an issue with periodic tasks (our bread and butter at Zapier). Detailed here: https://github.com/celery/celery/issues/844

Thanks a bunch Ask.

Re:Celery 3.0 has been released by simonpantzare on Sunday July 08, 2012 @07:41AM (#4214368)
I find this[1] example in Getting Started/Next Steps to be backwards:

    # incomplete partial:  add(?, 2)
    >>> s2 = add.s(2)
    # resolves the partial: add(8, 2)
    >>> res = s2.delay(8)
    >>> res.get()
    10
Shouldn't it behave like functools.partial[2]?

    >>> import functools
    >>> abc = lambda a, b, c: (a, b, c)
    >>> bc = functools.partial(abc, "a")
    >>> bc("b", "c")
    ('a', 'b', 'c')
1. http://docs.celeryproject.org/en/latest/getting-started/next...

2. http://docs.python.org/library/functools.html#functools.part...

Re:Celery 3.0 has been released by asksol on Sunday July 08, 2012 @08:46AM (#4214447)
Very good question!

The reason is that I found that it better matches what you use them for in Celery, since they are used e.g. to forward results from the previous tasks to a callback. Often you have a signatures like: def blur_image(image, amount=1), where the partial is used like render_image.s(file) | blur_image.s()

functools.partial is usually used the other way; it's most common to create functools.partials where the first argument is satisfied.

Re:Celery 3.0 has been released by JimmyL on Saturday July 07, 2012 @11:41PM (#4213523)
Celery has it's quirks now and then (for reasons I can't nail down or reliably reproduce, enabling CELERYD_FORCE_EXECV breaks a lot of things), but I consider Ask and the rest of the #celery gang to be models for how to have a welcoming community around an open-source project.

Looking forwards to experimenting with the non-multiprocessing worker - something in my current setup regularly leaks memory, and for the above-mentioned reason I can't automatically recycle worker processes to clean it up.

Re:Celery 3.0 has been released by hymloth on Saturday July 07, 2012 @01:38PM (#4212210)
It seems that the Eventlet pool does not work with the celery umbrella command. With celeryd it works ok. Is that true or do I have misconfigured something?
Re:Celery 3.0 has been released by asksol on Saturday July 07, 2012 @01:42PM (#4212217)
could be a bug, could you please open an issue at http://github.com/celery/celery/issues?
Re:Celery 3.0 has been released by reinhardt on Saturday July 07, 2012 @09:23PM (#4213244)
> Over 600 commits, 30k additions/36k deletions.

The large number of deletions is a pretty good sign of the project's health. Congrats!

Re:Celery 3.0 has been released by level09 on Saturday July 07, 2012 @04:59PM (#4212710)
Brilliant, I have started to use celery a few months ago, now we a use it to back our video transcoding and automation platform, our search engine, and our mobile app backend.

Excellent job celery team !

Re:Celery 3.0 has been released by rjurney on Saturday July 07, 2012 @07:18PM (#4213017)
Does Celery still create a new queue for every worker on rabbitmq? This bizarre behavior makes it hard to use Celery with amqp :(
Re:Celery 3.0 has been released by asksol on Saturday July 07, 2012 @07:39PM (#4213053)
one queue for every worker instance, yes! this is for broadcasting remote control commands, but I don't see how this poses a problem?

If you mean the one queue for every task behavior of the amqp result backend then that is also a yes. Usually with replies in amqp you create one queue for every client, but Celery is often used in a web context where the process that initiated the task may not be the same process that collects the reply, so this is why it uses one queue for every task (it's also documented).

There's an experimental result backend for RPC-style replies, that uses transient messages and one queue per client: http://github.com/celery/celery/tree/kombuRPC

Re:Celery 3.0 has been released by rjurney on Saturday July 07, 2012 @08:24PM (#4213134)
That is a problem for me, because I want all events of one type running through one AMQP queue to monitor throughput, etc.
Re:Celery 3.0 has been released by asksol on Sunday July 08, 2012 @08:13AM (#4214414)
Several ways to accomplish that, but I would recommend using kombu in combination with celery to have the task manually send messages.

You could set the exchange type of the results exchange to be topic too, that way you would both have result queues and you could additionally bind a queue to the results exchange to get a copy of all the messages sent there. But if you don't need to listen for individual results then I'd rather just send messages manually. You have both connection and producer pools in Celery, so it's rather convenient to combine kombu with celery.

Re:Celery 3.0 has been released by rox on Saturday July 07, 2012 @12:10PM (#4211972)
Finally I could use officially supported 'Canvas' to design workflows, before that I had to use the celery-tasktree package.
Re:Celery 3.0 has been released by gonvaled on Sunday July 08, 2012 @04:03PM (#4215505)
Can somebody compare celery to beanstalk?
Re:Celery 3.0 has been released by asksol on Monday July 09, 2012 @01:22AM (#4216816)
Celery supports beanstalk among many other backends (RabbitMQ, Redis, MongoDB, ..., etc)

So you can use Celery as a driver for beanstalk in Python.

Re:Celery 3.0 has been released by gonvaled on Monday July 09, 2012 @02:48AM (#4216991)
But it seems to me that Beanstalk and Celery accomplish the same thing. Am I mistaken?
Re:Celery 3.0 has been released by misiti3780 on Saturday July 07, 2012 @01:14PM (#4212138)
im using redis instead of rabbitmq - have had zero problems so far. great work, this project is awesome
Re:Celery 3.0 has been released by conesus on Saturday July 07, 2012 @01:14PM (#4212140)
Are you using redis with slaves? When I do that, celery throws errors when trying to add tasks to my redis queue.
Re:Celery 3.0 has been released by asksol on Saturday July 07, 2012 @01:29PM (#4212182)
what's the error and traceback? (paste at pastie.org or similar)
Re:Celery 3.0 has been released by conesus on Saturday July 07, 2012 @01:35PM (#4212201)
Preface this by saying that I've been successfully and happily using celery since 0.8, which was over 2 years ago. Since 2.0, it's been rock solid. But then I switched from RabbitMQ to redis and when I added redis slaves, it broke.

Here goes: http://pastie.org/4216467

It's probably due to redis' INFO command giving different values. Now would be a great time to get it sorted out, too.

Re:Celery 3.0 has been released by asksol on Saturday July 07, 2012 @01:39PM (#4212211)
About your error, seems strange that a simple info command gives that error, maybe you're running an outdated version of redis-py/redis-server?
Re:Celery 3.0 has been released by conesus on Saturday July 07, 2012 @01:47PM (#4212226)
Latest redis (2.4.15) and using redis-py 2.4.10. Just checked the CHANGELOG for redis-py, which is now at 2.4.13, and voila:

    * 2.4.11
        * Made the INFO command more tolerant of Redis changes formatting. Fix for #217.
Doh! Off by a single patch release. Thanks for the help.

See, folks? Celery is amazing in multiple ways.

Re:Celery 3.0 has been released by misiti3780 on Saturday July 07, 2012 @01:35PM (#4212199)
no i am not, sorry
Re:Celery 3.0 has been released by simonpantzare on Sunday July 08, 2012 @07:27AM (#4214349)
Out of curiosity, why do you use Redis instead of RabbitMQ?
Re:Celery 3.0 has been released by misiti3780 on Sunday July 08, 2012 @12:40PM (#4214948)
because its much easier to configure in my opinion, and i dont really care if messages get lost + i am using it in other parts of my stack so it i decided that instead of bringing in another piece of technology, i would utilize what i already had and keep things less complex
Re:Celery 3.0 has been released by srj55 on Saturday July 07, 2012 @01:01PM (#4212106)
looks like redis is getting improved support in this release. Anyone using redis instead of rabbitmq?
Re:Celery 3.0 has been released by ergo14 on Saturday July 07, 2012 @01:57PM (#4212253)
I do for Errormator task queue, since 2.5, no issues at all.
Re:Celery 3.0 has been released by topbanana on Saturday July 07, 2012 @06:06PM (#4212878)
Is it compatible with Hummus 2.0?
Re:Celery 3.0 has been released by stevvooe on Sunday July 08, 2012 @11:30AM (#4214789)
Only peanut butter 1.0, and there may be legacy support for ranch3, but you'll have to check the docs.
Re:Celery 3.0 has been released by creamyhorror on Saturday July 07, 2012 @11:58PM (#4213566)
You say that now, but tomorrow Hummus.py 0.5 will be announced.
Re:Celery 3.0 has been released by fatiherikli on Saturday July 07, 2012 @12:08PM (#4211964)
chain is amazing.