Most high performance software on super computers uses the Intel C and Fortran compilers, and much engineering and scientific software on workstations uses the Intel Maths Kernel Library (MKL) for high performance linear algebra.

Now that AMD EPYC processors are powering a lot of next generation super-computer clusters, we're going to have to figure out some workarounds!

I just compiled tensorflow on amd epyc and had no idea https://github.com/oneapi-src/oneDNN was actually MKL... now Im wondering if I even am getting all that power

The actual cpuid checking code is drilled from here: https://github.com/oneapi-src/oneDNN/blob/master/src/cpu/x64...

to here: https://github.com/oneapi-src/oneDNN/blob/master/src/cpu/x64...

It's using feature-flag checks, not family checks, so you shouldn't be affected if you're using oneDNN.

thank you!

I took a reversing course some years ago and during the first part we learned how to identify the compiler using common patterns. Long story short, the Intel compiler did a phenomenally amazing job optimizing. This was 10 years ago so things may be different now.

10 years ago LLVM was a baby and GCC was still on version 4. Intel probably have an advantage in areas where people pay them for it but GCC and LLVM are excellent compilers today.

Anecdotally, (ignoring that I'm still not sure whether to trust it or not) Intel stopped developing IACA and suggested (but not recommended) LLVM's MCA - which does suggest a changing of the guard in some way.

I think this: https://developer.amd.com/amd-aocl/amd-math-library-libm/ is supposed to be the alternative to MKL for those applications.

No need to develop and alternative when you can trick MKL into not crippling AMD: https://www.pugetsystems.com/labs/hpc/How-To-Use-MKL-with-AM...

Edit: the link I posted follows Agner's advice from the bottom of OPs link. However I think the extra information that it adds is that Zen2 Threadrippers outpaced then-current Intel's top contender. Once Zen3 and Intel's 11th Gen become available, repeating this benchmarks would be very valuable.

Thank you! I wasn't aware of this. But this is only a replacement for libm (i.e. basic trig, exp function), not the matrix-orientated BLAS, LAPACK and SCALAPACK routines that scientific codes spend >90% of their time.

I'm not personally familiar with those, but seems like BLAS, SCALAPACK, & others are also available:

https://developer.amd.com/amd-aocl/

I think you meant Intel Compiler? Yes. Intel Compiler consistently produces the highest performing binaries on Intel processors, often by a big margin. Intel MKL used to be the highest performing math library, and may still be so. As a result, most performance critical software, such as scientific applications, are compiled using ICC.

This is an overstatement. ICC consistently compiles the slowest and produces the largest binaries. It also defaults to something close to -ffast-math, which may or may not be appropriate. If your app benefits from aggressive inlining and vectorization at the expense of potentially huge increases in code size, ICC is likely to do well for you. However, I've seen lots of cases where well-vectorized code is faster with GCC or Clang, including some very important cases using Intel intrinsics. (Several such cases reported to/acknowledged by Intel; some have been fixed over the years, but these observations are not uncommon.)

BLIS is used by AMD and is a good open alternative to MKL (for BLAS) across many platforms. https://github.com/flame/blis/blob/master/docs/Performance.m...

I have been hearing about the superiority of Intel's compiler for a couple of decades now. Back when GCC was a tiny baby compared to what it is now, and when Clang/LLVM didn't even exist.

I wonder if this Intel compiler 'superiority' is still the case today, or if this is just a meme at this point.

For matrix manipulation based Fortran scientific codes, ifort/MKL can give +30% compared to gfortran. It's difficult to disentangle where the speedup comes from, but certainly as jedbrown aludes to, the Intel compilers seem to make a better go of poorly optimised / badly written code.

For C based software, its a much closer run thing, and often sticking with GCC avoids weird segfaults when mixing Intel and GCC-compiled Linux libraries.

> This is an overstatement.

To be generous...

Where do you typically see lack of inlining and vectorization with GCC? I'm curious because most times people have said GCC wouldn't vectorize code that I've been able to try, it would, at least if allowed -ffast-math a la Intel (as in BLIS now).

Can you explain "BLIS is used by AMD"? In what way do they use it?

It's their official BLAS [1] since 2015 when they moved away from their proprietary ACML implementation [2].

[1]https://developer.amd.com/amd-aocl/blas-library/

[2] https://developer.amd.com/open-source-strikes-again-accelera...

Amusingly, OpenBLAS significantly beat the bought-in ACML, on DGEMM, over the six(?) generations of Opteron I had available. AMD learnt.

The fact that MKL is the highest performing library has nothing to do with the quality of icc's output.

It is a myth that icc produces faster binaries that may have been true 25 years ago.

So what are they compiled with for non Intel processors?

The HPC codes I worked on we would compile with gcc, clang, icc and whatever vendor compiler was installed (Cray, PGI, something even worse). Then we'd benchmark the resulting binaries and make a recommendation based on speed. (Assuming the compiled binaries gave the correct results, which would sometimes fail and trigger further debugging to find out if we had undefined behaviors (or implementation defined) or had managed to find a compiler bug. For codes that are memory-bandwidth dominated the results are pretty much a toss-up. For compute bound codes intel would often win.

You can do the same when your machine has non-intel CPUs that are supported by a lot of compilers. If you are on power9 or arm the compiler list gets shorter. And a lot of supercomputers start to contain accelerators (often, but not always Nvidia GPUs) in which case there is often only one supported compiler and you have to rewrite code until that compiler is happy and produces fast code.

It's a depressingly common choice for educational installations. Folks are trained to use ICC instead of GCC and then they keep using ICC when they leave school.

Unsure, but I think that's the implication of this bit in the OP:

"The same effect was documented with some of the most popular mathematical software packages, including Mathematica, Mathcad, and Matlab."

A lot of scientific/HPC code running on supercomputers is compiled with icc.