News for hackers. Stuff that matters.

A comment on p-Hacking and False Discovery in A/B Testing. Parent comment

Re:p-Hacking and False Discovery in A/B Testing by stochastic_monk on Sunday July 22, 2018 @04:59AM (#17585914)
Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tedsanders on Sunday July 22, 2018 @11:59AM (#17587341)
Even in web testing there isn't an equality between false positives and false negatives.

Suppose you're Facebook and you decide to test a new landing page on 1 million users. You roll out the test and notice that after 10,000 users, the new page is killing engagement. Whoops, turns out it had a bug and isn't even loading. Even though no medical patients are dying, this is still a very negative outcome for Facebook. Obviously they shouldn't test on 990,000 more users before fixing the bug, but that's what slavish adherence to pre-registered trial lengths would tell you to do, because it's 'cheating' to notice that there's a problem after the first 10,000 tests.

Some of you might say, sure, for extenuating circumstances like a bug you can break procedure. But in that case I think the logic slides down a slippery slope. What if in the example above instead of a bug we just had a feature performing terribly? In either case, the right move is to end the test early, since you don't need all the data points to measure a strong signal.

[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tomrod on Sunday July 22, 2018 @09:46AM (#17586706)
Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by stochastic_monk on Sunday July 22, 2018 @10:48AM (#17587018)
A lot of real world cases have imbalanced reward functions. Autonomous vehicles are another example. I don’t doubt regulation systems on spacecraft are similar.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by tomrod on Tuesday July 24, 2018 @07:30AM (#17600014)
Great examples.
[ Parent | on HN ]
Re:p-Hacking and False Discovery in A/B Testing by webmaven on Sunday July 22, 2018 @12:28PM (#17587527)
Consider Facebook's news feed experiment on user's emotions:

https://www.nytimes.com/2014/06/30/technology/facebook-tinke...

[ Parent | on HN ]