Re:Celery, non-blocking code and quest against coroutines
by brianwawok on Thursday June 09, 2016 @09:34AM
(#11869243)
The biggest overhead is really setting up RabbitMQ right? Because Django and Celery play nice, you can just deploy one codebase and run 1 Django process and 1 Celery + Celery Beat process pretty easy...
Bummer you lose so many Celery features when you use Amazon or Google Cloud Queues instead of RabbitMQ....
Re:Celery, non-blocking code and quest against coroutines
by MollyR on Thursday June 09, 2016 @10:35AM
(#11869616)
Not just setting up, but celery has a learning curve which may or may not be worth your time.
Also you have an additional points of complexity (rabbitmq and celery workers) that also have an additional operations maintenance burden.
For a small app that doesn't have to scale, I'd avoid it.
But when building a business focused web app, I use it a lot, especially for large file parsing.
Re:Celery, non-blocking code and quest against coroutines
by mdomans on Thursday June 09, 2016 @09:43AM
(#11869301)
AFAIK you can have RabitMQ SaaS these days.
And yes, RabbitMQ is very good.
Re:Celery, non-blocking code and quest against coroutines
by rhodin on Thursday June 09, 2016 @11:07AM
(#11869859)
Yes CloudAMQP is such a service. It includes a free plan. If you are going to use Celery and CloudAMQP, I'd advice using the recommended settings[1], otherwise Celery will produce a large number of connections + messages.
(Disclaimer: I work with CloudAMQP)
Re:Celery, non-blocking code and quest against coroutines
by brianwawok on Thursday June 09, 2016 @10:28AM
(#11869576)
cloudamqp? I guess, not terribly cheap though.
Re:Celery, non-blocking code and quest against coroutines
by mdomans on Thursday June 09, 2016 @10:36AM
(#11869623)
I think there are a few other options. But deploying RabbitMQ isn't that hard. I can actually write something about that.
Is Linode cheap enough for you?
Re:Celery, non-blocking code and quest against coroutines
by brianwawok on Thursday June 09, 2016 @12:59PM
(#11870647)
I mean last time I needed RabbitMQ I just used
https://github.com/Mayeu/ansible-playbook-rabbitmq
and it took 3 minutes. But I hear a lot of people saying not to use Celery because RabbitMQ is hard.. maybe people just have different standards for hard?
I've used celery with a few django and flask projects. It's nice, but it added a lot of overhead for me. So I've used http://flower.readthedocs.io/en/latest/ to help handle the monitoring.
For some really small sites and very specific use cases, I actually wrote some small custom helper scripts to just have cron jobs.
If you are using django, I found this library helpful. https://bitbucket.org/wnielson/django-chronograph/