We've had a lot of luck just using gunicorn with a post_response hook for doing some extra work without making the client wait on it before getting a response. It avoids adding another dependency and is pretty straightforward. It makes sense for us since we don't have HUGE tasks that need to be queued, but some things that can take a maybe 0.5 to 3 seconds and where the HTTP response can be sent back before doing this work. I should open source this at some point, it's a small amount of code but pretty handy.

Note that periodic execution, one of the more useful aspects of celery, is not supported on all backends. If you're using rabbitMQ, you're golden. If you're using most or all other transports (including SQS), you're likely SOL and will have to resort to either bare cron jobs or something like django-cronograph that accomplishes the same thing.

celery is definitely nice, but it does have its limitations (like everything else).

I'm really interested in peoples opinion on Celery vs Others when using Django.

I'm about to hit a point in my pet project where I need something but I havn't chosen my approach yet. I've ehard Pyres is better than Celery.

Once you wrap your head around one (embarrassingly, it took me a while to really understand what celery was doing -- I had an easier time getting comfortable with the multiprocessing library), it shouldn't be difficult to pick up another and compare them locally.

Celery does what it's supposed to; that's about all I can say for it. I'd suggest writing a few simple jobs and testing them in similar environments under as much load as you can. Look for things like consistency of logging and running tasks exactly once across multiple consumers despite the semantics of the queue you pick.

This is good to know. I've just started using it with redis.

I never really saw the point of that part of celery anyway. Cron's perfectly adequate on its own (and installed just about everywhere).

The nice thing about it from our point of view is the ability to keep everything in source control. Granted, you can keep your cron jobs as scripts, but celery+rabbitMQ allows you to keep it all in Python without the extra deployment step. It's not a huge deal, but it does help a bit with maintainability and getting new devs up to speed.

We keep all the deployment stuff in source control anyway. For me, it would just mean shifting it from being declared by salt in the "infrastructure" folder to being written in python in the django folder.

I agree that you should keep the code itself in python, but that can be handled just by making it a management command that gets called by cron.

Sysadmins understand and love cron, too. I think celery reinventing the wheel might make some of them a bit mad.

I'm using cron instead of celery daemon for those tasks, but here are two advantages:

- being able to modify the task frequency from the admin

- cron needs special care with setting the environment variables like PATH

Yea - that's going to be one of the packages I mention in the follow up. I ended up using RabbitMQ as the backend for it which wasn't that easy to setup so I am curious to see what others are using.

What about http://pythonhosted.org/APScheduler/ ? Celery is for more massive, high capacity solutions.

Using celery for the first time, coming from PHP-land, blew my mind.

Python is now my language of choice for "getting shit done".

Returning to PHP land afterwards feels like you're taping your code together with duct tape and bits of string.

At the risk of this becoming a "bash PHP" thread, that's exactly how I feel. Unless I'm doing something very simple, it's always easier, cleaner, and simpler to do it in Python. Even building a basic CRUD app is night and day between the two languages; Django is beautiful and intuitive, while the PHP frameworks come close but have a few snags that make working with them frustrating.

I write this not to meaninglessly bash PHP but to hopefully convince even a single developer passing by to try out Python. I thought the same thing as you, passer-by dev, once: "Why do I need Python? It's stupid. It uses whitespace weirdly. I can do whatever I need in PHP." But then you realize you could be doing everything you're doing in PHP... better, smarter, and faster.

I'm getting the same feeling now that I have to work in python/django after having used scala. So it's always worth trying new languages, even when you've found one that seems great at the time.

The fact is PHP has excellent frameworks. I tried Django and Flask , and love them. But for now , i'm sticking to PHP( to much legacy code to port ). For new projects ? sure. The only thing which is a bit more complicated with serverside python is deployment on client servers, the rest ( development , testing ), is easier than PHP. I only wished Python had type hinting for functions and methods like PHP does though. Aside from these issues , Python is perfect,more consice and have awesome stuffs like decorators. And a whole lot of awesome libraries.