Re:Most Winning A/B Test Results are Illusory [pdf]
by tristanz on Sunday February 23, 2014 @05:56PM
(#7288069)
Putting aside bandits and all that, it seems like the first step should be to set up a hierarchical prior which performs shrinkage. Multiple comparisons and stopping issues are largely due to using frequentist tests rather than a simple probabilistic model and inference that conditions and the observed data.
Gelman et al, "Why we (usually) don't have to worry about multiple comparisons" http://arxiv.org/abs/0907.2478