AVX code is known to make the CPU run way hotter than usual. Perhaps that caused throttling that made general code running at the same time, or within a short span thereafter, perform worse?

That is one theory I had but i'm not sure how to determine if CPU is throttling (on Ubuntu Linux.)

You can either use lscpu, which is less accurate, or the best way is to check:

cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq

where instead of cpu0 you can write any core number, and it will give you the current frequency of that core in KHz.

I'll try that out. Thanks.

Note that with pstate /proc/cpuinfo is not reflective (not suggested here but, in the past, the MHz used to change to reflect the scaling speed). You could also look at 'powertop'

One way I've found to isolate this is to turn off turbo boost and underclock the cpu, while making sure the avx offset is set to 100%. While you would never want to run a production system like this, it does help to eliminate any issues with cpu throttling. If the avx-512 version of the program still runs slower then something else is interfering besides cpu throttling.

Did you remember to put a vzeroupper instruction at the end of your AVX functions? If not then you pay a performance penalty when transitioning from AVX to SSE code, and the rest of your program might be inadvertently touching SSE code hidden inside the Go runtime.

Yes I already do that.

Running AVX-512 on all cores will lower all core operating frequency, in order to maintain the TDP of the processor.

When you run it in isolation, there's more headroom since only one core can run at the higher freq. and use AVX512 registers.

Also the power license ensures the cores running AVX512 code runs at a lower frequency.

Make sure to guard your AVX512 code block with VZEROUPPER when exiting it

cache invalidation may be a reason