News for hackers. Stuff that matters.
Re:Wild claims about K performance by aidenn0 on Wednesday September 01, 2021 @12:35AM (#28376482)
There's flawed benchmarks and there are completely wrong benchmarks.

I've seen so many benchmarks where the timing or loop overhead dominated to the point that they weren't measuring anything but the cost of taking timing values

I've seen benchmarks where they run a single time without warming up the disk cache, so whichever benchmark they happen to run first shows as 2 orders of magnitude slower.

I've seen benchmarks where the standard deviation was much larger than the difference between the times reported, but they didn't bother to check for this.

I've seen benchmarks where they don't even use the same algorithm, and the difference in the algorithm chosen dominates any other concerns.

Re:Wild claims about K performance by still_grokking on Wednesday September 01, 2021 @08:20PM (#28387224)
Exactly! And even those very wrong results give quite interesting insides.

Such stuff is worthless as benchmarks, sure. But all the other things you can learn form such stuff is sometimes even quite fascinating.

So discussing completely wrong benchmarks has imho often some value.

Re:Wild claims about K performance by aidenn0 on Wednesday September 01, 2021 @08:39PM (#28387360)
It's entirely possible that the net utility of such benchmarks being published is negative; enough people come away with the initial wrong conclusion without critically investigating it.

It's almost certainly true that the net value to the company whose product looks bad in the benchmarks is negative, which is why these clauses exist.

Re:Wild claims about K performance by kaba0 on Friday September 03, 2021 @08:14AM (#28403138)
Of course you can learn from these. But do they describe anything related to the product at hand?