Re:Determining A/B test sample size
by bryanh on Tuesday September 20, 2011 @04:01PM
(#3019319)
I rarely see many people take into account the opportunity cost of letting a really close A/B test reach 99.99% confidence when the benefit is by definition very marginal (that's why its taking so long, right?). I mean, is it really that bad to go on "close enough" results and move on to bigger and better tests?
[ Parent
| on HN ]
In academia, we used p values of .95 or greater. I was taught that in business, though, the rule of thumb for making decisions is more like .8, and that's typically the standard I use as well.