Re:Celery, non-blocking code and quest against coroutines
by asksol on Thursday June 09, 2016 @12:29PM
(#11870438)
Celery can use async I/O too! Using the gevent pool you can have e.g. 4 celery worker instances with 1000 threads each.
Re:Celery, non-blocking code and quest against coroutines
by sciurus on Thursday June 09, 2016 @01:55PM
(#11871062)
Right, but as you mention elsewhere in the thread there are caveats, like discussed at
http://docs.celeryproject.org/en/latest/userguide/concurrenc... . At this point the blog post author's contention that celery is an better alternative to doing async i/o breaks down.
Take the author's example of scraping 1000 webpages. Let's say their computer running celery has enough memory to run 50 celery processes. This means they can request 50 web pages concurrently. Most of the time those celery processes are going to be sitting idle, waiting on the remote web host to respond. This is terribly inefficient compared to coroutines with asyncio/aiohttp, or even using a thread pool since urllib will release the GIL and let another thread run when it's blocked on network i/o. You could have a single process performing the work just as fast as 50 celery processes.