News for hackers. Stuff that matters.

A comment on Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values. Parent comment

Re:Statisticians Find They Can Agree: It’s Time to Stop Misu... by eanzenberg on Monday March 07, 2016 @01:32PM (#11240555)
Multiple comparisons? 1 million independent tests? Hello?

>Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be.

>Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

^Means you should start a new trial with n > 20 given the same placebo/drug split.

[ Parent | on HN ]
Re:Statisticians Find They Can Agree: It’s Time to Stop Misu... by minimaxir on Monday March 07, 2016 @01:43PM (#11240628)
In the scientific world, it is economically infeasable (cost/time) for 1 million tests for a given hypothesis. Even n > 20 can be difficult for certain studies. Bootstrapping the results to simulate 1 million trials won't fix the aforementioned issue either.
[ Parent | on HN ]
Re:Statisticians Find They Can Agree: It’s Time to Stop Misu... by _snydly on Monday March 07, 2016 @02:44PM (#11241064)
> Even n > 20 can be difficult for certain studies.

For some expensive experiments even n=3 is difficult to reach. "Should we do another replicate and go from n=2 to n=3, or should we hire another post-doc?" is a relatively common (rhetorical) question.

[ Parent | on HN ]