News for hackers. Stuff that matters.
Re:Simple Sequential A/B Testing by tristanz on Wednesday October 14, 2015 @02:54AM (#10385333)
Why not just estimate p(A - B | observed data) and be done with it?
Re:Simple Sequential A/B Testing by yummyfajitas on Wednesday October 14, 2015 @09:35AM (#10386493)
Because that's a lot harder. At VWO we do exactly this in our new SmartStats tool - we believe Bayesian stats are far more intuitive to the end user. But we've got a whole compute cluster running these calculations 24/7. For comparison, our old frequentist calculations were done in PHP at page render time.

A formula that can be easily plugged into Excel does have value over a formula that requires someone to write a python script and do a monte carlo simulation.

Evan's method is also nice in that it doesn't require a lot in terms of knowing your (possibly very small) conversion rate ahead of time.