Re:Two Performance Walls Approaching (2009)
by chubot on Friday September 18, 2015 @11:08AM
(#10239607)
One way to ameliorate this is to write multi-threaded Python extensions -- i.e. with the threads on the C side, not the Python side.
You can use one core for the Python interpreter, and then 63 or 127 cores for the C/C++ code. Python is the control plane, and C is the data plane. This is a really useful pattern, but there is not much built-in support for it.
It would be nice if there were Go-like channels between Python and C (i.e. to communicate between threads, instead of locking and unlocking the GIL and mutating shared data structures). And this means that you need to write/rewrite your own extensions too. You need to factor your program differently, but I find this can make it more modular.
You can use one core for the Python interpreter, and then 63 or 127 cores for the C/C++ code. Python is the control plane, and C is the data plane. This is a really useful pattern, but there is not much built-in support for it.
It would be nice if there were Go-like channels between Python and C (i.e. to communicate between threads, instead of locking and unlocking the GIL and mutating shared data structures). And this means that you need to write/rewrite your own extensions too. You need to factor your program differently, but I find this can make it more modular.