Re:Multiprocessing in Python with Fortran and OpenMP
by sevensor on Monday November 13, 2017 @02:47PM
(#15689574)
OpenMP is great and all, but you're going to have an awfully hard time beating numpy on either speed or correctness, since it's using LAPACK under the hood already.
[ Parent
| on HN ]
Re:Multiprocessing in Python with Fortran and OpenMP
by username223 on Monday November 13, 2017 @07:50PM
(#15691668)
They're very different tools. Your system's LAPACK is probably crazily optimized (e.g. ATLAS or MKL) [1], so if you can reduce your problem to LAPACK subroutines, that's probably the way to go. If you have lots of cores and your problem doesn't neatly reduce to LAPACK, OpenMP will probably help you. In either case, you're well-served spending some quality time with a profiler, and possibly fighting your compiler's optimizer.
[1] In case you're not familiar with it, ATLAS runs a bunch of small programs to figure out what approach is fastest on your specific machine, trying different instructions and memory access patterns for different array/matrix sizes. Cool stuff.
Re:Multiprocessing in Python with Fortran and OpenMP
by milancurcic on Monday November 13, 2017 @04:22PM
(#15690375)
False. OpenMP does shared-memory parallelism and has nothing to do with LAPACK. numpy is completely serial.
Re:Multiprocessing in Python with Fortran and OpenMP
by Derbasti on Tuesday November 14, 2017 @01:59AM
(#15693258)
Numpy (or rather, BLAS) is running many threads in parallel. The amount of concurrency is actually configurable, and is something well worth optimizing for multiprocessing workloads.
Re:Multiprocessing in Python with Fortran and OpenMP
by auxym on Monday November 13, 2017 @04:38PM
(#15690500)
Numpy can be compiled against a multi-threaded BLAS, for example MKL.
Re:Multiprocessing in Python with Fortran and OpenMP
by santaclaus on Monday November 13, 2017 @07:33PM
(#15691583)
I've seen OpenMP-based, non-BLAS implementations beat out MKL, on Intel hardware nonetheless. We emailed the MKL folks and they just shrugged. :) Don't presume that MKL is the lower limit on performance.
Re:Multiprocessing in Python with Fortran and OpenMP
by pletnes on Monday November 13, 2017 @04:57PM
(#15690644)
Anaconda python is a great option for exactly that. MKL gives great multicore performance. Many packages, e.g scikit-learn, dask and others, lean on numpy for performance, and are thus much faster.
Re:Multiprocessing in Python with Fortran and OpenMP
by milancurcic on Monday November 13, 2017 @05:00PM
(#15690669)
Yes it can, in which case you embed OpenMP parallelism in low-level library calls, and is another way to accomplish what the article suggests.
But it's still a fine-grained parallelism so numpy can match plain Fortran performance in special cases (and not the other way around).