Re:The Most Misleading Measure of Response Time: Average
by amikula on Wednesday December 11, 2013 @03:27PM
(#6890426)
I think in Optimizely's case, the most important factor is making sure that there's no statistically significant correlation between higher response times and A/B testing. In other words, if the higher response times result an imbalanced impact on the test, the test is invalid.
[ Parent
| on HN ]
Re:The Most Misleading Measure of Response Time: Average
by dsiroker on Wednesday December 11, 2013 @03:35PM
(#6890492)
You are right that if one variation takes longer to load and is noticeable it is likely to cause a lower conversion rate. However with Optimizely the response time (on average and 99th percentile) is the same between control and test variations since our implementation is a single snippet of code regardless of what bucket you are put into.
What we are optimizing here for is end-user experience and minimizing the chance they have a higher-than-tolerable response time regardless of the variation they see.