Re:Python’s Multiprocessing Performance Problem
by bee_rider on Wednesday March 01, 2023 @03:48PM
(#34988116)
And if you are using for example Numpy, you aren’t using Python for anything performance sensitive of course, because Numpy is almost certainly calling the system’s tuned BLAS implementation. Which should handle the parallelism I guess. If anything I’d expect parallel Python calls to Numpy to result in oversubscription…
[ Parent
| on HN ]
Re:Python’s Multiprocessing Performance Problem
by fest on Wednesday March 01, 2023 @04:13PM
(#34988433)
I don't think numpy functions are neccessarily multithreaded and probably many are inherently sequential by their nature, so there are definitely case where multiprocessing can speed up the overall program.
Re:Python’s Multiprocessing Performance Problem
by bb88 on Wednesday March 01, 2023 @09:31PM
(#34991179)
Someone once said that python + numpy is probably going to be faster than writing it using basic C++, since numpy is using highly tuned libraries underneath.
I don't know for certain this is the case, but I'd like to see some benchmarks about it.
Re:Python’s Multiprocessing Performance Problem
by kelipso on Thursday March 02, 2023 @10:09AM
(#34996570)
You would almost never use raw C++ when working with linear algebra stuff. You use a library like Eigen that interfaces with BLAS, LAPACK, etc., so you definitely get all the advantages of those highly tuned libraries, plus the speed of C++ and potential flexibility of not having to make multiple array copies and so on.
Re:Python’s Multiprocessing Performance Problem
by cozzyd on Thursday March 02, 2023 @12:24PM
(#34998733)
Right, but you can use those highly tuned libraries yourself.
I do kind of wish numpy had a stable C API that didn't require a Python interpreter though.
Re:Python’s Multiprocessing Performance Problem
by bee_rider on Wednesday March 01, 2023 @10:23PM
(#34991545)
They aren’t necessarily threaded, but if you care about Numpy performance on an Intel chip at least you are already using MKL for Numpy’s BLAS, and MKL’s gemm is threaded.
Re:Python’s Multiprocessing Performance Problem
by anonkogudhyfhhf on Wednesday March 01, 2023 @05:33PM
(#34989275)
Parallelism using CPU instructions and vectorization most likely. Threading would still improve performance
Re:Python’s Multiprocessing Performance Problem
by bee_rider on Wednesday March 01, 2023 @10:21PM
(#34991529)
Multiplying a large enough matrix in Python using MKL for Numpy, I can watch the cpu usage go to 400% in top. You may need to run it in a loop or make the matrices quite large, a surprisingly large amount of computation has to happen before it’ll show up in top.