Re:Simple Sequential A/B Testing
by tristanz on Wednesday October 14, 2015 @02:54AM
(#10385333)
Why not just estimate p(A - B | observed data) and be done with it?
[ Parent
| on HN ]
Re:Simple Sequential A/B Testing
by yummyfajitas on Wednesday October 14, 2015 @09:35AM
(#10386493)
Because that's a lot harder. At VWO we do exactly this in our new SmartStats tool - we believe Bayesian stats are far more intuitive to the end user. But we've got a whole compute cluster running these calculations 24/7. For comparison, our old frequentist calculations were done in PHP at page render time.
A formula that can be easily plugged into Excel does have value over a formula that requires someone to write a python script and do a monte carlo simulation.
Evan's method is also nice in that it doesn't require a lot in terms of knowing your (possibly very small) conversion rate ahead of time.