Re:Wild claims about K performance
by still_grokking on Wednesday September 01, 2021 @08:20PM
(#28387224)
Exactly! And even those very wrong results give quite interesting insides.
Such stuff is worthless as benchmarks, sure. But all the other things you can learn form such stuff is sometimes even quite fascinating.
So discussing completely wrong benchmarks has imho often some value.
It's entirely possible that the net utility of such benchmarks being published is negative; enough people come away with the initial wrong conclusion without critically investigating it.
It's almost certainly true that the net value to the company whose product looks bad in the benchmarks is negative, which is why these clauses exist.
I've seen so many benchmarks where the timing or loop overhead dominated to the point that they weren't measuring anything but the cost of taking timing values
I've seen benchmarks where they run a single time without warming up the disk cache, so whichever benchmark they happen to run first shows as 2 orders of magnitude slower.
I've seen benchmarks where the standard deviation was much larger than the difference between the times reported, but they didn't bother to check for this.
I've seen benchmarks where they don't even use the same algorithm, and the difference in the algorithm chosen dominates any other concerns.