p-Hacking and False Discovery in A/B Testing

Re:p-Hacking and False Discovery in A/B Testing by boron1006 on Sunday July 22, 2018 @05:07AM (#17585939)

Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake.

In a realistic scenario for us, bayesian and frequentist approaches will probably converge to a point that's close enough for a company that runs a website (i.e., we weren't killing anyone by leaving our experiment running). We also weren't getting massive fluctuations in effect size.

The cost of the Bayesian approach, in terms of learning a new system of statistics, programming everything up from scratch, and interpreting results, probably wouldn't be worth the efficiency gains. If we were creating an AB testing program, then I would probably do so.

[ Parent | on HN ]

Re:p-Hacking and False Discovery in A/B Testing by gbrown on Monday July 23, 2018 @08:43AM (#17592305)

Adaptive designs and early stopping rules are really good when:

1. Data collection is expensive (time or money)

2. Keeping with the status quo in the presence of new evidence is problematic (withholding a promising new drug)

3. Continuing with the experiment in the presence of new evidence is problematic (the new drug is hurting people)

Absent one of those features, it's probably not worth the added complexity.

[ Parent | on HN ]