News for hackers. Stuff that matters.
Re:To an alarming degree, science is not self-correcting by tokenadult on Thursday October 17, 2013 @03:44PM (#6567664)
This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications.

Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to share in a Hacker News comment. That's because we see submissions here every day of preliminary studies that can be analyzed by Norvig's checklist on research issues to look for when reading about a scientific finding.

I discuss psychology research weekly with a group of psychologists who study human behavior genetics in a "journal club" (graduate seminar course). Those researchers have told me about other researchers who are trying to clean up the published literature in psychology, for example Jelte Wicherts, whose article "Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science"[2] in an open-access journal suggests general procedures to improve scientific publishing, for example by changing the incentive structure around reviewing papers submitted for publication. Another helpful researcher on statistical tests to verify results is Uri Simonsohn. The papers he and his colleagues produce[3] are thought-provoking, pointed, and sometimes laugh-out-loud funny.

[1] http://norvig.com/experiment-design.html

[2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020

http://www.frontiersin.org/Computational_Neuroscience/10.338...

[3] http://opim.wharton.upenn.edu/~uws/

Re:To an alarming degree, science is not self-correcting by Alex3917 on Thursday October 17, 2013 @10:27PM (#6569448)
> Warning Sign I5: Taking p too Seriously

What does it say about X when you have 10 studies saying X is true with p < .05, and another 10 studies saying that X is false with p < .05? It seems like the closer you come to actually taking p values at face value given conflicting results, the amount of things that can be considered to be 'scientifically proven' rapidly approaches zero.

Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)

Re:To an alarming degree, science is not self-correcting by einhverfr on Friday October 18, 2013 @04:26AM (#6570326)
> If you read the modern scientific literature on research, the rabbit hole goes vastly deeper.

On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely different from whether it has ontological value).

[1] http://archive.org/details/PhysicsPhilosophy

Re:To an alarming degree, science is not self-correcting by skylan_q on Friday October 18, 2013 @09:01AM (#6571097)
Try telling that to economists who claim to discover macroeconomic theories through econometric models and call it "empirical"
Re:To an alarming degree, science is not self-correcting by dekhn on Friday October 18, 2013 @09:57AM (#6571371)
Nothing is ever "proven". We merely make statements that become increasingly challenging to show are likely to be false based on human judgement of experimental methods.

P values are estimates of the probability that a given observation is likely to have come about by chance, rather than due to a hypothetical model. They have many problems and it is best to think about them with a solid understanding of probability.

If there was a case where you have as many conflicting results as you say, then it would be indicative of a very active scientific community and problem. People would be working hard to resolve the problem!

Re:To an alarming degree, science is not self-correcting by shanusmagnus on Thursday October 17, 2013 @05:45PM (#6568346)
This beautiful (parent) comment, in contrast to the main article, actually contains links and references to the items it refers to. Is there some compelling reason I don't understand why _so_ many magazines / websites, even high profile reputable ones, cannot manage to link or cite?
Re:To an alarming degree, science is not self-correcting by unhammer on Friday October 18, 2013 @09:54AM (#6571347)
Science Exchange: https://www.scienceexchange.com/reproducibility

Ioannidis' article on false pos/neg and stat.power: http://www.plosmedicine.org/article/info:doi/10.1371/journal...

Bohannon's "Who's afraid of a little peer review": http://www.sciencemag.org/content/342/6154/60.full (fake submissions)

Also related I think is this: http://neurotheory.columbia.edu/~ken/cargo_cult.html ;-)

Re:To an alarming degree, science is not self-correcting by cousin_it on Thursday October 17, 2013 @08:00PM (#6568990)
I suspect the reason is that outbound links drive traffic away from ads and internal links. The more visitors you have, the more profit you lose by adding outbound links... It's sad and I wish there was some way to fix the incentives.
Re:To an alarming degree, science is not self-correcting by sdoering on Friday October 18, 2013 @12:22AM (#6569763)
That's exactly what people in charge fear in my industry (working in online publishing). The do fear, that the people find the linked ressources better, so that they may never even return.

Kind of like the fear of becoming obsolete. But instead of testing and maybe even becoming better, they just forbid us to link out.

Re:To an alarming degree, science is not self-correcting by kybernetikos on Friday October 18, 2013 @03:32AM (#6570228)
I find this particularly galling since many of these publications have fact checkers who have gone to the effort of identifying claims in the original article, tracking down the evidence to prove/disprove it and checking that the article doesn't materially misrepresent the evidence.

This is the internet, it shouldn't cost any more to let me see the fact checkers report, and it will save me a massive amount of time reproducing their work. Expandable margin notes on a per paragraph basis would be perfect.

Re:To an alarming degree, science is not self-correcting by foxlisk on Thursday October 17, 2013 @05:26PM (#6568241)
Norving missed an excellent chance to title this paper "The structure and interpretation of science experiments"
Re:To an alarming degree, science is not self-correcting by sillysaurus2 on Thursday October 17, 2013 @04:57PM (#6568069)
Norvig's essay is an absolute must-read. You may learn more counter-intuitive facts in the next hour than in the last week. For convenience to mobile users, here's the link again: http://norvig.com/experiment-design.html
Re:To an alarming degree, science is not self-correcting by unhammer on Friday October 18, 2013 @09:56AM (#6571361)
For people interested in reproducability in computational linguistics and NLP, I recommend http://lingpipe-blog.com/2013/09/25/fokkens-et-al-on-reprodu... and the article linked within. One of the authors of "Replication Failure" has another, slightly older but really awesome paper: http://www.d.umn.edu/~tpederse/Pubs/pedersen-last-word-2008....