Re:How Firefox Got Fast Again
by Manishearth on Monday November 13, 2017 @02:58PM
(#15689658)
Whilst benchmarking stylo for example we noticed there were perceptible wins on interaction time on real world sites; but speedometer didn't budge as much because it tends towards small DOM trees which stylo can't parallelize much.
In the case of Speedometer, it's measuring one specific type of webapp interaction in a single tab; it's not capturing things like perceived speed of opening or closing the browser, switching tabs, loading pages, etc. It doesn't test how the browser performs when many JS-heavy tabs are loaded in the background. It also has lots of corner cases: installing a common password manager halves the score, and a common adblocker nearly does the same again, yet neither show that kind of noticeable degradation in real-world browsing.
Benchmarks are useful for saying "we've improved our browser this much in this area compared to where we were last year," but they're much less useful for making broad generalizations across heterogeneous browsers; there are too many other variables at play.