News for hackers. Stuff that matters.

A comment on p-Hacking and False Discovery in A/B Testing. Parent comment

Re:p-Hacking and False Discovery in A/B Testing by aalleavitch on Sunday July 22, 2018 @04:12AM (#17585801)
http://www.evanmiller.org/how-not-to-run-an-ab-test.html
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tedsanders on Sunday July 22, 2018 @04:19AM (#17585817)
>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward.

Completely agree with Evan Miller here, thank you for sharing the link.

[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by rpier001 on Monday July 23, 2018 @05:11AM (#17591494)
Except that the quote reflects a common misunderstanding. The problem of optimal stopping is mostly a function of decision making over multiple looks. A Bayesian approach that makes decisions over multiple looks has similar issues. This can be mitigated by strong priors, but typically to an unknown degree and at some cost to 'power'. How/why does this claim arise? It is because a 'true' Bayesian approach makes no decisions/inferences - it just describes the current state of knowledge. If describing the distribution of the posterior, then 'stop at any time and be valid' else 'you got your Neyman-Pearson in my Bayesian analysis'.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by stochastic_monk on Sunday July 22, 2018 @04:59AM (#17585914)
Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tedsanders on Sunday July 22, 2018 @11:59AM (#17587341)
Even in web testing there isn't an equality between false positives and false negatives.

Suppose you're Facebook and you decide to test a new landing page on 1 million users. You roll out the test and notice that after 10,000 users, the new page is killing engagement. Whoops, turns out it had a bug and isn't even loading. Even though no medical patients are dying, this is still a very negative outcome for Facebook. Obviously they shouldn't test on 990,000 more users before fixing the bug, but that's what slavish adherence to pre-registered trial lengths would tell you to do, because it's 'cheating' to notice that there's a problem after the first 10,000 tests.

Some of you might say, sure, for extenuating circumstances like a bug you can break procedure. But in that case I think the logic slides down a slippery slope. What if in the example above instead of a bug we just had a feature performing terribly? In either case, the right move is to end the test early, since you don't need all the data points to measure a strong signal.

[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tomrod on Sunday July 22, 2018 @09:46AM (#17586706)
Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by stochastic_monk on Sunday July 22, 2018 @10:48AM (#17587018)
A lot of real world cases have imbalanced reward functions. Autonomous vehicles are another example. I don’t doubt regulation systems on spacecraft are similar.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tomrod on Tuesday July 24, 2018 @07:30AM (#17600014)
Great examples.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by webmaven on Sunday July 22, 2018 @12:28PM (#17587527)
Consider Facebook's news feed experiment on user's emotions:

https://www.nytimes.com/2014/06/30/technology/facebook-tinke...

[ Parent | on HN ]