News for hackers. Stuff that matters.
Re:Web Framework Benchmarks Round 4 by tlarkworthy on Thursday May 02, 2013 @01:12PM (#5645835)
"average and std dev are only revelent if the distribution is Gaussian in distrition"

technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models.

Low variance is good, regardless of underlying distribution.

Re:Web Framework Benchmarks Round 4 by mynegation on Thursday May 02, 2013 @04:14PM (#5646924)
Probably the statement should be that comparing mean and variance are only relevant if both metrics follow the same distribution. In the absence of distribution information (and it is usually absent in empirical tests like that) quantiles would help to do a better job at comparing performance.
Re:Web Framework Benchmarks Round 4 by diroussel on Friday May 03, 2013 @03:42AM (#5649267)
Indeed. Often latency measurements are clumpy.

You might have one clump of fast responses when no GC occurs, another when some GC occurs, and a smaller clump where a stop-the-world full GC has occurred.

In such a case average is not meaningful.