Interesting that they chose this approach. At Myna we normally squish data through the logistic function to rescale (if necessary) and then use a beta model. The beta model expects binary observations but you either interpret fractional observations as the probability of a weighted coin flip, or just do fractional updates to the beta. Both worked about the same in my tests. (This difference is mainly due to our different approaches A/B testing. Both are fairly textbook.)

The more interesting question, to me, is how you get credible intervals. I'm assuming you just MCMC 'em up, but it would be nice if there was a more computationally efficient trick.

We don't actually use Gaussian models for any of our Bayesian tests =). For continuous data, we use log-normal models, and the next blog post will be about how to make all that work.

And for intervals: yup, we just use sampling. But it's super fast because all you need are independent samples. For more details, see this first post:

http://engineering.richrelevance.com/bayesian-ab-tests/