Re:Celery, non-blocking code and quest against coroutines
by wacowz on Thursday June 09, 2016 @11:32AM
(#11870055)
Every time I read an article like this, all I can think is, this is what Elixir and Erlang are built for. I love Python and use it on a daily basis, but developing concurrent systems in Python is beginning to look like a questionable choice, while there exists tools better suited for this.
Celery works great, and the integration with django is superb. However, it's consistently been a memory hog for us. Hell, we had to write a shell script to regularly check for memory use by celery and take necessary precautionary steps.
[ Parent
| on HN ]
Re:Celery, non-blocking code and quest against coroutines
by brianwawok on Thursday June 09, 2016 @01:06PM
(#11870691)
What level of concurrency is required though? 99.9% of web request only needs concurrency at the user level, which standard python + fork will do just nicely.
Making a realtime system to crunch some massive numbers in 50 micros? Then you might want some C or Java concurrency. But much of the world is not coded at that level.
Re:Celery, non-blocking code and quest against coroutines
by wacowz on Thursday June 09, 2016 @10:35PM
(#11873915)
It's often the case that some work needs to be done outside of the request/response lifecycle, eg external api calls, data aggregation, sending notifications etc. Most of it is essentially IO bound. We happen to have lots of such tasks and therefore use celery, but there's only so much celery can do, given it's concurrency level is bound by the number of cores available since it uses multiprocessing. An actor based approach or sth like Go's goroutines would be a better fit.