Re:We Should Not Accept Scientific Results That Have Not Bee...
by return0 on Friday July 29, 2016 @09:03AM
(#12186523)
> but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out.
How certain are we that its 'the nature' of these fields rather than political choices that were solidified long ago? There could be much larger samples and six-sigma confidence in life sciences if the grants were allocated differently. I know this is happening, for example, in neuroscience where the (private) Allen Institute is creating significantly more rigorous (and useful) datasets compared to the bulk of studies, because they are funding their studies differently.
Unfortunately for medical drugs and psychology, researchers are mostly gathering data without an understanding of the underlying mechanisms. There are also virtually never have proposals which can be tested for compliance with reality in a quantifiable and isolated way as we can with physics, chemistry, or parts of biology.
So I feel the replication crisis is not a matter of various fields just not knowing how to do "good science", but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out.