Re:p-Hacking and False Discovery in A/B Testing
by delecti on Sunday July 22, 2018 @12:10AM
(#17585233)
In statistics, a p-value [1] is a the odds of getting a result by chance if there's not actually an effect from the thing you're testing. A common threshold to use is .05, which means that you'd expect the given result less than 5% of the time by chance. Typically this indicates a relationship, but if you simultaneously test enough things that aren't causally linked you'd expect some to have p<0.05. A good example is this xkcd [2]. In the example, green jelly beans aren't actually related to acne, but just by chance the result seems to indicate they are.
[1] https://en.m.wikipedia.org/wiki/P-value [2] https://xkcd.com/882/