Re:Celery, non-blocking code and quest against coroutines
by wacowz on Thursday June 09, 2016 @10:35PM
(#11873915)
It's often the case that some work needs to be done outside of the request/response lifecycle, eg external api calls, data aggregation, sending notifications etc. Most of it is essentially IO bound. We happen to have lots of such tasks and therefore use celery, but there's only so much celery can do, given it's concurrency level is bound by the number of cores available since it uses multiprocessing. An actor based approach or sth like Go's goroutines would be a better fit.
Making a realtime system to crunch some massive numbers in 50 micros? Then you might want some C or Java concurrency. But much of the world is not coded at that level.