You could use identical priors for each variant, with the mean equal to the empirical mean of your existing system, and the variance controlling how plausible big improvements are.
In this case your prior mean could be something as simple as an average conversion rate for all your past articles.
Remember that by using frequentist statistics your are implicitly assuming a uniform prior---I think it's quite easy to find weakly informative priors which are more plausible than saying that any conversion rate is equally likely.
Re:Bayesian A/B Testing
by yummyfajitas on Thursday May 29, 2014 @11:42AM
(#7816788)
Or even better, you can look at the conversion rates of past articles and fit a beta distribution to them.
You use an non-informative prior, like a Jeffreys prior.
Unfortunately, the requirement of priors (seemingly) precludes Bayesian stats from many A/B testing applications. For example, our site launches every article with 3 or 4 headline variations. A priori, we have no idea which headline will be most successful or even what the expected conversion rate is for that article.
Is there any way to use Bayesian methods in such a scenario?