> It's different if you're compiling everything yourself, then those benchmarks might be relevant.

And even if you do it's irrelevant. Most common large/important frameworks won't compile with proprietary compilers.

Doing Bench's with ICC, XLC or other is hypocritical and often does not reflect anything useful.

Only the HPC world can afford to recompile everything with proprietary compilers and justify the man power to do so. And even so, they already have passed most compute intensive kernels on GPGPU with cuda a long time ago.

No, the correct thing to do is to publish multiple columns showing performance under the different compilers. Large co do use the specific compiler that will give them better performance. I know many big software compiled with ICC. Then, there is the lack of tlak about MSVC. That's one standard compiler used extensively.

In benchmarking you have two choices: publish the real numbers or not. Which option you choose marks you as honest or not.

You can argue about why the numbers for your product are lower in the discussion section of your report. Not in an asterisk.

IBM, Intel and PGI reign on HPC.

"Really Intel and AMD are being dishonest by publishing benchmarks that don't match reality"

People can discard with the various SPEC* benchmarks as unrealistic -- they generally are -- but this is a bridge way too far.

The whole point of the SPEC* benchmarks is that you bring out the best of the best and do everything you can to achieve the pinnacle of performance.

And those SPEC* benchmarks are most interesting, and most applicable, to HPC users, who actually do use hardware-specific compilers. Of course another person mentioned that "everything is on GPUs now" (it isn't): If it's on a GPU, then you don't need a 60 core CPU. If it's on a GPU, then SPEC* benchmarks aren't relevant.