leo_pekelis2

While our solution is different than the current industry standard in A/B testing platforms, all the techniques we are using have been around in the statistics literature for decades, and are tried and true. The particular sequential test of power one that we use has been around since the 1970s and goes back to the time of Herbert Robbins. And FDR control has been well documented in the past 25 years, most notably by Yoav Benjamini, and Yosef Hochberg. We really are standing on the shoulders of giants.

I think our biggest contribution is presenting a principled, powerful mathematical solution in a way that is accessible to practitioners without a formal statistical background. Even if you do have this knowledge, it’s a chance to use these methods without having to reinvent the wheel every time.

There are various methods which could have been used as solutions, and we looked at many different ones to determine a fit to the user model and experience Optimizely is presenting. We are currently doing an AMA on our community portal and I would be happy to discuss potential solutions or any other comments with you there, https://community.optimizely.com/t5/Product-What-s-New/Ask-m...

Glad you like that we’re changing things up!

You do bring up a good point. Even though a sequential test is able to be called much earlier than a fixed horizon test (note this only happens when the effect size is large enough to still guarantee Type I error control), it does not change the fact that estimates of the effect size are more variable when there are fewer visitors. The way we are addressing this is to make confidence intervals more prevalent in our platform. The width of confidence intervals represents our uncertainty in the magnitude of the true effect size with the information currently available. They correctly get more narrow as the experiment goes on as there is increased information from more visitors.