News for hackers. Stuff that matters.
Re:Estimating the chances of something that hasn’t happened yet by taneq on Thursday November 15, 2018 @10:11PM (#18466035)
> If the all three measurements of X are smaller than all three measurements of Y, you have X < Y with 95% confidence.

On a multitasking system, you should use the fastest benchmark run (assuming you're running the same code on the same data in all cases, and if you're not then you're not really benchmarking). This will be the one that is least influenced by any other processing going on.

Re:Estimating the chances of something that hasn’t happened yet by gugagore on Friday November 16, 2018 @10:00AM (#18469159)
That's the case when you think the measurement noise is only positive. But are there kinds of noise that reduce your benchmark time?

On one hand, I believe a piece of code has a true benchmark time, but we only measure noisy versions that are skewed positive due to multitasking noise. On the other hand, it seems important to make benchmark measurements in the context of the whole system. If A beats B in the best case (unloaded system), but doesn't win in the average case (taking the context into account), then that's important information.

Re:Estimating the chances of something that hasn’t happened yet by skybrian on Friday November 16, 2018 @02:23PM (#18471260)
The way I think about it is that benchmarks include deterministic operations (that happen every time) and nondeterministic operations (that don't). And all of these numbers are positive, since there are few things in the world that take negative time.

By taking the minimum, we focus attention on the deterministic operations. This isn't likely to be realistic for production, but production won't look much like your benchmark hardware anyway.

When making code changes, deterministic operations are the part you have most influence over and have the most impact. Removing an unnecessary, deterministic operation will speed up every run, not just some of them.