Re:Web Framework Benchmarks Round 4
by mynegation on Thursday May 02, 2013 @04:14PM
(#5646924)
Probably the statement should be that comparing mean and variance are only relevant if both metrics follow the same distribution. In the absence of distribution information (and it is usually absent in empirical tests like that) quantiles would help to do a better job at comparing performance.
[ Parent
| on HN ]
Indeed. Often latency measurements are clumpy.
You might have one clump of fast responses when no GC occurs, another when some GC occurs, and a smaller clump where a stop-the-world full GC has occurred.
In such a case average is not meaningful.