I generally agree about the pros you list of Bayesian, but I think in companies that are doing a lot of experiments and are not just optimizing a checkout page, they don't hold as well. For example, you often are testing out features that really are different from any feature before, and so a prior is harder to get alignment on. There are also frequentist methods like CUPED to mitigate the effect of confounding variables, and then usually you're analyzing the results across user segments so that you can try to look for heterogeneous treatment effects. (FWIW, having analyzed a lot of experiments, I've been surprised at how often there is NOT a heterogeneous treatment effect across user segments; the baselines are often very different, but the direction and general magnitude of the lifts are typically similar. And, the confounder that is most often relevant is simply some broad "engagement" dimension: users who use the product a lot might behave differently from users who do not.)

In my experience, the biggest blocker to using Bayesian approaches has been about data and computation requirements; there are some closed-form solutions (e.g. https://www.evanmiller.org/bayesian-ab-testing.html , also https://www.chrisstucchio.com/blog/2014/bayesian_asymptotics... ) but even those are computationally difficult at scale even if you don't have to do a lot of MCMC sampling.

But what is really interesting to me about your comment is:

> A/B Tests are not really controlled experiments, at least not in the same way clinical trials are. User behavior is always observational even if you have a control group

Could you expand on that? Isn't clinical data also observational: you observe the patient after some time, and see what their symptoms are or what various endpoint measurements are?

> Could you expand on that? Isn't clinical data also observational

Generally in statistics there is a divide between controlled experiments and observational studies. The former is how most medical trials are run and the latter is how most anthropologists work.

In medical trials you can make sure all the demographics, lifestyle differences etc are controlled for before you even start the trial. In the more extreme case of experiments in the physical sciences you can often tightly control for everything involved where the only difference between test and control is precisely the variable of interest.

In anthropology you can't go back in time and say "what if this society had a higher ratio of male/female and didn't go to war?", you can only model what happened and attempt to bake some causal assumptions into your model to see what happened. This is why detailed regression analysis is very important in these fields.

Having a test and control group in your A/B test is not really enough to establish the equivalent of a controlled study of rats in a laboratory, or bacteria in petri dishes. In my experience it really helps to include causal models regarding what you know of your A/B testing population to ensure that what you're observing is really the case. I've had concert examples where it looks like one variant is winning and then checking for the causal assumptions find that we were accidentally measuring something else.

In practice you can reframe all of this into being some sort of ANOVA test and fit it just fine into a classical framework, but I find starting with a Bayesian methodology make the process easier to reason about.

Many A/B tests as RCTs do suffer from the fact that they're randomizations of convenience sample of users coming to a web site, but it's always seemed to me that randomization addresses most confounders due to population heterogeneity. Is there some specific confounding variables you worry about that are not addressed by randomization in this way?

One challenge for me is how to explain to marketers what the "Probability A is better than B" means from a Bayesian perspective. How do you convey to them that such a probability is not a frequency of an event? I haven't met anyone who is not a statistician that won't default to "you know, how often it happens" when pressed to define probability. My attempts to engage marketers on this fall like water off a duck's back.

In general I prefer NHST methods because I often require reasonably tight error control. Do you do any kind of calibration to ensure error rate control when doing your procedures? Or are your clients comfortable with less tight inferential error controls?

Even though you can analyze a test Bayesianly, you still need to know about power (or some Bayesian alternative to power) to plan a sample size.

This comment reminded me of yet another reason to prefer the Bayesian set up for A/B tests: you work with the data you have not some bizarre idea of "sample size".

Especially with proper priors (which again, everyone running an A/B test should have at this point) you don't have to "plan a sample size". You work with the data you have, and assess the risks in the decision give what you know. There is likewise no risk of "early stopping".

If you're doing Bayesian analysis and only have say 100 visitors, and are using priors, the only situation in which you'll have a strong posterior is when the winning variant is notably superior to the current one. All of the ad hoc rules Frequentists put in place are not necessary when you do proper analysis of your posterior probability that A > B.

Again we see where Bayesian vs Frequentist doesn't really matter for a clinical trials but does for running an A/B test. In a clinical trial you are going to have to plan how many people are involved anyway, so doing a test power calculation makes sense since you'll need some number and that's a good way to get one.

For running an A/B test the bigger problem is usually time not numbers (an exception being email campaigns in which both tend to matter). In the Bayesian setup you if your boss says "I need an answer tomorrow morning" you can take the data you've got, show the probabilities of improvement as well as the distribution of the risks if you're wrong. This allows you to make riskier moves when it makes sense, and be more conservative when it does not. This is something that the out-of-the-box NHST does not allow you.

I am also team bayes for all the reasons you stated, but do want to argue a couple counterpoints:

* While you don't have to have a fixed sample size up front, you can still "cheat" in a bayesian analysis if you peek constantly and end early on promising results that you want to win, and let them run longer otherwise. So you want to do something to account for this (put some structure in place, approach with skepticism, laugh and put on sunglasses, whatever).

* It's very often useful in practice to have some idea of what kind of answer you're going to see in how long for planning reasons -- for example, rather than your boss saying "I need an answer tomorrow" they say "I need an answer as quick as you can". Bayesian methods give you the flexibility to be risky when you need to and accurately count for uncertainty, but sometimes you still need to predict and strategize around ideas like "We'll be about this certain in 2 days, and about this certain in 1 week, and about this certain in 4 weeks and it seems like planning on next Tuesday is the right call"

I've found understanding these frequentist methods to help inform my guesstimates of how experiments will play out with regards to sample size and impact as well as honestly evaluate the trade-offs in evaluating the tests where I wasn't running it -- AB testing is really widespread so I feel like it's important to understand frequentist tests well even if you intend to never use them if you can help it.

Well, at least in my day job, a very common question is "how much more data do we need to collect?" or "how long will the experiment take?". A response of "The analysis will be Bayesian, so those questions don't apply" is not helpful. Planning matters!

Even a Bayesian with a proper prior can make good guesses about sample sizes, for example, by saying that they have a goal to reduce the length of the 95% credible interval for the most relevant parameter by 80%.