News for hackers. Stuff that matters.
Re:The Most Misleading Measure of Response Time: Average by amikula on Wednesday December 11, 2013 @03:27PM (#6890426)
I think in Optimizely's case, the most important factor is making sure that there's no statistically significant correlation between higher response times and A/B testing. In other words, if the higher response times result an imbalanced impact on the test, the test is invalid.
Re:The Most Misleading Measure of Response Time: Average by dsiroker on Wednesday December 11, 2013 @03:35PM (#6890492)
You are right that if one variation takes longer to load and is noticeable it is likely to cause a lower conversion rate. However with Optimizely the response time (on average and 99th percentile) is the same between control and test variations since our implementation is a single snippet of code regardless of what bucket you are put into.

What we are optimizing here for is end-user experience and minimizing the chance they have a higher-than-tolerable response time regardless of the variation they see.