News for hackers. Stuff that matters.
Re:Just 11% of 53 cancer research papers were reproducible by wheaties on Thursday May 09, 2013 @07:59AM (#5679896)
This is one of the reasons I went into math and software. In math there is only black and white, provably true and patently wrong. Still, part of the reason this is so is that to show any results everything must be laid on the table. CS and Math papers are therefore open by design and the world has benefited. I hope the rest of science will move in this direction.
Re:Just 11% of 53 cancer research papers were reproducible by btilly on Thursday May 09, 2013 @09:34AM (#5680332)
I'm guessing you have not read Proofs and Refutations by Imre Lakatos. It is worthwhile.

Also if you read on the history of mathematics, the various back and forth foundational issues raised by infinitesmals, Fourier series, proof by contradiction and the like are very interesting. Sure, it is possible for a modern mathematician to look at all that and say, They just didn't know how to do it, HERE you go. But over decades and centuries people were somehow successfully continuing to do math, even though they knew that something serious was wrong in their understanding.

Re:Just 11% of 53 cancer research papers were reproducible by scott_karana on Thursday May 09, 2013 @03:30PM (#5682453)
At least those historical mathematicians acknowledged their gaps of knowledge. Cancer research is a far cry different.
Re:Just 11% of 53 cancer research papers were reproducible by btilly on Thursday May 09, 2013 @03:44PM (#5682509)
You clearly are unfamiliar with the history.

If you can find a copy, George Berkeley's The Analyst was a cogent criticism of the foundations of calculus in his day. (He didn't have answers, but he identified real problems.) A variety of books were written by mathematicians in response. Uniformly these were much lower quality, and consisted of defenses of the foundations of mathematics, rather than acknowledgements that there were real issues. His criticisms were not taken seriously until that mathematical framework completely fell apart when Fourier series constructed "obviously impossible" things.

Today we are used to a very general notion of a "function". But historically a square wave simply wasn't a function. (Because if it was, how did you know how it interacted with infinitesmals?) And getting one out of the sum of a bunch of well behaved sins, with some simple integration, was a huge shock.

Re:Just 11% of 53 cancer research papers were reproducible by scott_karana on Friday May 10, 2013 @05:21PM (#5688753)
Yikes! Thanks for setting me right.

I should have said some historical mathematicians then.

Re:Just 11% of 53 cancer research papers were reproducible by jamessb on Thursday May 09, 2013 @08:20AM (#5679978)
There are 'proofs' that contain errors, just as there are science papers that reach incorrect conclusions.

Also, it's not necessarily easier to verify a proof than it is to repeat an experiment (or perform additional supporting experiments), especially for complex proofs which very few people know enough to completely understand (eg. of Fermat's last theorem, or the Poincare conjecture).

Re:Just 11% of 53 cancer research papers were reproducible by rajeevk on Thursday May 09, 2013 @08:04AM (#5679920)
CS has similar problem, specially in AI fields. Researches often claim the accuracy rate more than 95%, but in reality that is much lesser (in fact less than 10% in most of the claims).
Re:Just 11% of 53 cancer research papers were reproducible by 5kg on Thursday May 09, 2013 @08:20AM (#5679979)
Yes, that's why I no longer consider pursuing PhD in CS. It's a waste of life to reproduce non-reproducible papers that is full of buzzword.

Hopefully, more journals and confs will follow the Open Access and Reproducible Research model like IPOL[1].

> Each article contains a text describing an algorithm andsource code, with an online demonstration facility and an archive of online experiments. The text and source code are peer-reviewed and the demonstration is controlled.

[1]http://www.ipol.im/

Re:Just 11% of 53 cancer research papers were reproducible by gordaco on Thursday May 09, 2013 @08:20AM (#5679977)
Having worked in computer vision and robotics, I can attest to this.
Re:Just 11% of 53 cancer research papers were reproducible by jacobparker on Thursday May 09, 2013 @08:33AM (#5680026)
Yes. Of the fields I've implemented algorithms from papers in, computer vision seems to be particularly adverse to discussing the (often critical) downsides of their algorithms. Often its something like "Camera calibration isn't exactly perfect? Well this scene reconstruction technique simply wont work."
Re:Just 11% of 53 cancer research papers were reproducible by pfortuny on Thursday May 09, 2013 @08:23AM (#5679989)
Not to be too picky (am a mathematician too) but you ought to read about the history of the theory of "limit cycles" of planar vector fields and Dulac's "Theorem". Quite a lesson.

Edit: sorry, could not help writing this later (was on my iPhone before). The thing is that Dulac 'solved' an important part of one of Hilbert's problems. Was a 'proof' for about 60 years.

Re:Just 11% of 53 cancer research papers were reproducible by gavinh on Thursday May 09, 2013 @08:24AM (#5679994)
Pick a paper on NLP. Measures of document relevance and summary quality are not black and white, especially when you account for the ways the authors munged their test sets and scores.
Re:Just 11% of 53 cancer research papers were reproducible by throwaway1980 on Thursday May 09, 2013 @09:17AM (#5680255)
> In math there is only black and white, provably true and patently wrong.

You'd probably be interested in Godel's incompleteness theorem.

Also, exact reproduction of other people's experimental work is highly unusual in CS, despite the theoretical possibility. Usually the materials and methods sections of papers in hard sciences like physics, chemistry, and biology are much more detailed, to the point of being recipes.

Finally, CS is not really a science. It lies somewhere between math and engineering, which are also not sciences.

Re:Just 11% of 53 cancer research papers were reproducible by arrrg on Thursday May 09, 2013 @01:03PM (#5681552)
If that were an actual possibility in other fields everyone would be delighted. Sadly, biology (human minds and societies as well) are much too complicated for this to work.

This is an actually challenging problem. I don’t think running away from it and resorting to cheap tribalism is a great idea.