Re:Scientists make further step towards understanding dark e...
by jjoonathan on Monday April 26, 2021 @11:10AM
(#26943329)
You can trivially get to 11 sigma by making your null hypothesis sufficiently garbage.
That's called "p-hacking" and is unfortunately very common. It is encouraged whenever someone innocently reads meaning into a sigma value without evaluating whether or not the null hypothesis is any good.
If I saw someone going around saying "The sky is reddish green, 11 sigma result!" my first suspicion would not be that their numbers were incorrect but rather that their null hypothesis was something silly like "the sky is entirely the solid sRGB color #0000FF." Obviously this garbage null hypothesis can be rejected with enormous confidence, that's where the 11 sigma came from, but the confident rejection of a garbage null hypothesis should not be taken to support their proposed alternative theory unless you have reason to believe the null hypothesis was well modeled, either because you checked it or because someone else did. It all comes back to trust, in other words.
That's called "p-hacking" and is unfortunately very common. It is encouraged whenever someone innocently reads meaning into a sigma value without evaluating whether or not the null hypothesis is any good.
If I saw someone going around saying "The sky is reddish green, 11 sigma result!" my first suspicion would not be that their numbers were incorrect but rather that their null hypothesis was something silly like "the sky is entirely the solid sRGB color #0000FF." Obviously this garbage null hypothesis can be rejected with enormous confidence, that's where the 11 sigma came from, but the confident rejection of a garbage null hypothesis should not be taken to support their proposed alternative theory unless you have reason to believe the null hypothesis was well modeled, either because you checked it or because someone else did. It all comes back to trust, in other words.