News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results are Illusory [pdf] by pak on Sunday February 23, 2014 @06:44PM (#7288266)
This article's title echoes a paper which continues to influence the medical research and bioinformatics community, "Why Most Published Research Findings Are False" by JPA Ioannidis.

http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj...

While the OP's article targets some low-hanging fruit, like halting criteria, multiple hypotheses, etc. which should be familiar to anyone serious about bioinformatics and statistics, Ioannidis takes these things a little farther and comes up with a number of corollaries that apply equally well to A/B testing.

After all, the randomized controlled trials that the FDA uses to approve new drugs are essentially identical to what would be called an A/B test on Hacker News.

There are no comments on this comment yet.