ballison

I think using Bayesian methods is a good thing. The reservation I have about this post is that it's motivated as A/B testing, which in classical frequentist testing usually equates to a null hypothesis that some parameter is equal in two populations (in this example, conversion rate under method A is equal to conversion rate under method B). The rest of the post then describes a test of a single population against a known value (is the conversion rate under method B = 5%).

These are not the same problems at all, and it's not clear how the authors propose to extend the tests they're informally describing to the test of equality of a parameter in two populations. It is possible in a Bayesian framework, but it's not this simple.