Re:Intel Fortran now 20% faster than C, fastest language on ...
by xixi77 on Thursday January 12, 2012 @01:26PM
(#3457395)
Mean would be a better choice if a typical real-world application were split approximately equally between each of these tasks. Which is of course patently untrue, particularly so because writing number-crunching code in Fortran does not prevent you from writing your hash-table code in something else (note how the best C binary-trees code is actually calling an external library).
Comparison of medians is nice because it shows that for approximately half of fairly diverse tasks one language is faster than another, which imo is a much more interesting statistic.
Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C).
What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be).
If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.