Re:30k medical studies may be invalid due to contaminated ce...
by arkh on Thursday October 19, 2017 @03:21AM
(#15506056)
> we found a very interesting biological pattern in the data
Isn't it related to p-hacking? Get lot of data, search for a pattern then publish some theory the pattern would "prove". Instead of having a theory first, setting up experiments which can disprove it then analyzing the data from those experiments.
Re:30k medical studies may be invalid due to contaminated ce...
by biomcgary on Thursday October 19, 2017 @04:16PM
(#15511198)
The observed pattern exactly fit into a previously established hypothesis and was totally unexpected. I did not slice and dice the data with multiple statistical tests. The only way it could have been related to p-hacking is that I am familiar with a bunch of biological theories and could recognize data patterns that are compatible with any of them. So, maybe a Bonferroni-type correction of 30-70x (i.e., number of theories my internal neural net has been trained on)?
I think part of the decline in biology specifically may be the rapid growth of NIH funded research in the 90's due to the success of molecular biology and the necessity of many warm bodies at the bench, which was followed by the NIH budget "crunch" of the last decade. This combination dramatically increased the competitive nature of getting grants and high-impact papers. In this environment, those who are careful in their claims are selected out of the career pool (not all obviously, but enough to affect the proportion).
A couple of my experiences: I had one paper in a mid-range journal where we found a very interesting biological pattern in the data, which we discussed but ultimately decided to dismiss because there were confounding factors. A very well known lab took the exact same data, did a couple minor experiments to justify dismissing our concerns, and published in a top-tier journal.
In another case, I reviewed a paper from a VERY famous lab and their own metrics proved that 95% of the data was noise. They simply removed the offending metrics and published in a different, equally high-profile journal.