Re:Bayesian A/B Testing
by punee on Thursday May 29, 2014 @10:32AM
(#7816375)
Assuming I'm a marketer who's using A/B testing tools and who's only interested in getting "statistically significant" results, and assuming Bayesian methods provide some advantage in terms of regret minimization that translates to real dollars earned, I feel that I could totally outsource my understanding of the theoretical underpinnings that get me that result. After all, what percentage of Optimizely or VWO users perfectly understands the statistical framework the tools are based on? So I'm not really convinced by that line of reasoning.
[ Parent
| on HN ]
Bayesian methods don't perform so much better in the situation you just named. They're also more expensive.
The really shine in that they provide a really uniform vocabulary of producing new, more sophisticated models while "frequentist" methods usually rely on ingenuity to get to better models. So if you're actively exploring a model space and want to attach a bunch of assumptions and degrees-of-freedom to correspond with a theory you're testing... then Bayesianism is the way to go.
Theoretically, marketers are doing that exact process. In practice, they don't see that is a statistical process, though. Maybe someday a tool will bridge that gap successfully... but again, you're unlikely to get a huge advantage with Bayesian methods without that increased work investment.