Re:Optimizing sample sizes in A/B testing
by paraschopra on Friday February 07, 2020 @06:31AM
(#22265012)
Founder of vwo.com here.
This happens because no two businesses have exactly the same end goals and therefore there is no "one-size-fits-all" testing algorithm.
Some of our clients want to prioritize absolute certainty because they're testing something as fundamental as product positioning, while others want to test different colors and want a result better than control (irrespective of how much better it is).
Ultimately, you can think most A/B testing algorithms on a continuum of certainty and speed. You can't have both.
This happens because no two businesses have exactly the same end goals and therefore there is no "one-size-fits-all" testing algorithm.
Some of our clients want to prioritize absolute certainty because they're testing something as fundamental as product positioning, while others want to test different colors and want a result better than control (irrespective of how much better it is).
Ultimately, you can think most A/B testing algorithms on a continuum of certainty and speed. You can't have both.
More here: https://vwo.com/blog/multi-armed-bandit-algorithm/