For chips like these, they are complex enough that even benchmarking them to figure out which one is faster is a chore.

It depends on how the memory channels work, what kind of optimizations you use in your benchmarking tools, how power delivery is handled on the motherboard, and that's before you consider the way the CCDs or cross-core communication happens (which is unique on each of the major architectures (Xeon/EPYC/Ampere)).

Anandtech had a great article comparing previous-generation server chips and found that memory access, L2/L3 access, and other cross-core communication varied dramatically on different chips, and even when BIOS was configured certain ways!

To that end, I wonder if there are benchmarks or standard methods that also measure “deep” utilisation — not just keeping cores “busy” but actually saturating their processing capacity.

> For chips like these, they are complex enough that even benchmarking them to figure out which one is faster is a chore.

If two chips are hard to tell apart in terms of performance, that just means that they are practically equivalent. This means other factors come in play, such as ISA and cost.

They are not hard to tell apart, AFAICT; they are hard to reduce to a single number for easy comparison.

Your point stands though.

There are a pile of BIOS settings which are BIOS settings precisely because there isn't a right answer for every workload. DIMM channel ganging, DIMM bank interleave, socket interleave, NUMA chiplet vs socket descriptions, NUMA weights, and on an on frequently have 5%-10% or more of an impact on a given workload. Change the workload and those numbers all change. Its like playing with compiler flags, this or that flag raises or lowers the perf of this or that part of a given benchmark. If there was a concrete answer, they wouldn't be flags.

In a lot of cases when machines are within 20% of each other, its quite possible the real difference between the two results is how many hours a perf engineer has spent tuning the workload.

In a way this is why the simple microbenchmarks are as important as the system level ones. The simpler ones are easier to tune and give a speed of light for a given operation.