MLWiDA

I like to throw in the first chapter of "Statistical Inference for everyone" for a little bit more explanation for these rules (https://github.com/bblais/Statistical-Inference-for-Everyone).

It looks to me like an easier approach to the heavy weight "Probability Theory: The Logic of Science" by E.T. Jaynes

My two cents:

1. As pointed out correctly in the article and "Most A/B-Test Results are illusionary", using a non-sequential frequentist method in a sequential way leads to wrong conclusions. In this regard, the comparison is unfair. On the backside, the frequentists' sequential methods for more than one dimension seem to be rather complicated (for one dimension and binomial distribution, check out the Sequential Probability Ratio Test (SPRT)).

2. Finding a good prior is really really hard. Choosing a weak prior may lead to jumping to wrong conclusions. I found it useful to a) choose a uninformative prior b) do not perform a statistical analysis until 1-2 weeks of data are in (to be sure that all special-day-effects like weekends are caught) ... which could be seen as a prior with weight of the same interval ;) c) using a simple old-fashioned permutation test to control the alpha error and prevent jumping to conclusions

While this approach has turned out to be too conservative in the binomial case, it becomes extremely important when dealing with revenue. Single customers do turn the tide. Often these are clear outliers, but sometimes it is not that easy to say. But this is a topic for another day.