What does it say about X when you have 10 studies saying X is true with p < .05, and another 10 studies saying that X is false with p < .05? It seems like the closer you come to actually taking p values at face value given conflicting results, the amount of things that can be considered to be 'scientifically proven' rapidly approaches zero.
Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)
On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely different from whether it has ontological value).
P values are estimates of the probability that a given observation is likely to have come about by chance, rather than due to a hypothetical model. They have many problems and it is best to think about them with a solid understanding of probability.
If there was a case where you have as many conflicting results as you say, then it would be indicative of a very active scientific community and problem. People would be working hard to resolve the problem!
Ioannidis' article on false pos/neg and stat.power: http://www.plosmedicine.org/article/info:doi/10.1371/journal...
Bohannon's "Who's afraid of a little peer review": http://www.sciencemag.org/content/342/6154/60.full (fake submissions)
Also related I think is this: http://neurotheory.columbia.edu/~ken/cargo_cult.html ;-)
Kind of like the fear of becoming obsolete. But instead of testing and maybe even becoming better, they just forbid us to link out.
This is the internet, it shouldn't cost any more to let me see the fact checkers report, and it will save me a massive amount of time reproducing their work. Expandable margin notes on a per paragraph basis would be perfect.
Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to share in a Hacker News comment. That's because we see submissions here every day of preliminary studies that can be analyzed by Norvig's checklist on research issues to look for when reading about a scientific finding.
I discuss psychology research weekly with a group of psychologists who study human behavior genetics in a "journal club" (graduate seminar course). Those researchers have told me about other researchers who are trying to clean up the published literature in psychology, for example Jelte Wicherts, whose article "Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science"[2] in an open-access journal suggests general procedures to improve scientific publishing, for example by changing the incentive structure around reviewing papers submitted for publication. Another helpful researcher on statistical tests to verify results is Uri Simonsohn. The papers he and his colleagues produce[3] are thought-provoking, pointed, and sometimes laugh-out-loud funny.
[1] http://norvig.com/experiment-design.html
[2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020
http://www.frontiersin.org/Computational_Neuroscience/10.338...
[3] http://opim.wharton.upenn.edu/~uws/