Re:Developing a computational pipeline using the asyncio mod...
by hyperion2010 on Tuesday December 01, 2015 @05:51PM
(#10659515)
Using a threadpool with asyncio doesn't buy you any parallelism, you have to use a processpool, which is fine for a data processing pipeline. Asyncio does not let you bypass the gil but it does vastly reduce the cost of spinning up absurd amounts of sockets and the like.
[ Parent
| on HN ]
Re:Developing a computational pipeline using the asyncio mod...
by superbatfish on Tuesday December 01, 2015 @08:52PM
(#10660356)
I saw this article, enjoyed it, and then said to myself "Now let me go read the inevitable well-meaning but misguided comments about the GIL." And sure enough, the very top comment is a well-meaning but misguided comment about the GIL.
The author of this article is talking about CPU-intensive tasks that involve C-extensions. Such extensions typically release the GIL during their execution, which means that you can effectively parallelize such workloads in a ThreadPool. (He explicitly mentions this about Numpy/Scipy, but most other CPU-intensive extension libraries behave the same way.) Heavy users of Numpy use Thread-level parallelism very frequently.
Re:Developing a computational pipeline using the asyncio mod...
by hyperion2010 on Tuesday December 01, 2015 @10:21PM
(#10660697)
Yep. Thought I was missing something since he was taking about cpu bound tasks and threads, if you are using c extensions then it is wasteful to spin up additional interpreters and you can also get fun bugs like numpy reusing the same random seed in child processes!