Re:Developing a computational pipeline using the asyncio mod...
by hyperion2010 on Tuesday December 01, 2015 @10:21PM
(#10660697)
Yep. Thought I was missing something since he was taking about cpu bound tasks and threads, if you are using c extensions then it is wasteful to spin up additional interpreters and you can also get fun bugs like numpy reusing the same random seed in child processes!
The author of this article is talking about CPU-intensive tasks that involve C-extensions. Such extensions typically release the GIL during their execution, which means that you can effectively parallelize such workloads in a ThreadPool. (He explicitly mentions this about Numpy/Scipy, but most other CPU-intensive extension libraries behave the same way.) Heavy users of Numpy use Thread-level parallelism very frequently.