We are very much entering the age where mechanical sympathy is waning
I used to do assembly and count cycles, but now I wouldn't dare; it's hardcore compiler and library makers stuff. It's like "don't do your own crypto (optimization)".
Everyone knows why it is so, though - we cannot solve the problem by throwing more Gigahertz at it.
Writing your own crypto is very different; the stakes are higher if you get it wrong.
I guess if you run it often enough it could still give useful numbers, but I understand the author is picking the best run.
Also I don't see to be any attempt at avoiding compiler optimisations.
Benchmarking is hard
Is there at least a checklist? It should ideally be automated. IIRC there's some library, maybe BLAS, that at compile time computes some machine-specific constants and also refuses to do it if a few things are not right.
Rant continued. I would prefer if the performance sections of papers were restricted to single paragraphs and that artifacts were required. When I'm reading papers I'm more interested in their ideas than in the third decimal place of the result on that given day on some machine I don't have. The Unix Timesharing System has no performance results. The Case For A Reduced Instruction Set Computer has no performance results. The notion that not only having results is somehow important but that providing the testing methodology and hardware configuration is also necessary is nonsense. It just lards up papers and a couple of years later the cited configuration is irrelevant.
The CRAY-1 Computer System has no performance results but its section on vector processing still rings true today. It's what the RISC-V vector extension uses.
There are a ton of tradeoffs that have to be made in the microarchitecture and in general the focus is real code and normal expected sizes rather than benchmarks. Big page multiples are common and then variable sized calls that can be totally unaligned and random sizes.
A proper study of this would test a pile of different mixes of sizes and alignments and test on a bunch of different processor generations.
And then you get results where a hand coded memcpy loop is faster in benchmarks, but in practice in a big program you might find that 'rep movs' is better because of the smaller code footprint. When you only optimize the few cases that matter.
The general rule is that the compiler and system libraries do a really good job optimizing for processors that shipped 10 years ago. So architects get to make decisions about how to help these operations. You can add a nifty feature that makes it faster (like CLZERO) but while it is good for benchmarks it will takes years after you ship before normal programs benefit. You might optimized 'rep movs' but then you find that since it was slow in the past most real system librarys don't call it.