News for hackers. Stuff that matters.
Re:Something Doesn't Add Up: John Ioannidis and medical stat... by alanctgardner2 on Sunday December 30, 2012 @06:33PM (#4987554)
As a serious question, do papers typically get published if they simply disprove a promising idea?

As an example, my friend is doing honours research right now, and her supervisor is very emphatic that she should do an experiment that will be 'successful' so she can publish it. My question is, doesn't performing an 'unsuccessful' experiment progress the state of the art as much as proving something? If a reasonable researcher might have an idea, and I prove it won't work, why wouldn't that be shared within the community as a negative result?

Re:Something Doesn't Add Up: John Ioannidis and medical stat... by eric_bullington on Sunday December 30, 2012 @06:43PM (#4987583)
No, and this problem even has a name: publication bias. It's a known issue in most modern peer-reviewed research. There are now multiple efforts underway to counteract its effects, such as a US regulation called FDAAA 801 and its resulting data registry clinicaltrials.gov.
Re:Something Doesn't Add Up: John Ioannidis and medical stat... by carbocation on Monday December 31, 2012 @02:30AM (#4988704)
There is something of a sea change going on right now (perhaps it's still too early to use that term, but the tide is surely shifting).

Let's distinguish two types of "negative" studies: one is a study whose result simply failed to be positive. "We performed exome sequencing in a kindred but failed to identify a common mutation in all affected members of the family." There are so many possible error modes, including erroneous fundamental assumptions about where this family's defect lies, that this is possibly uninteresting to virtually anyone.

The second type is a study that was done in a way that was powered to detect a positive result, had it been there. "We performed a randomized controlled trial in 1,000 patients with hemochromatosis to determine whether iron chelation was non-inferior to bloodletting for the treatment of hemochromatosis. We found that iron chelation failed to achieve non-inferiority." In this example (which is made up), this is a negative result that teaches us something immediately, and which could change practice (or could tell everyone that practice is not about to change despite rumors of a potentially new therapy).

Along these lines, to prevent these valuable yet "negative" trials from going unpublished, high-end medical journals such as the New England Journal of Medicine are now requiring that all trials enter into a registry.

Re:Something Doesn't Add Up: John Ioannidis and medical stat... by elemeno on Sunday December 30, 2012 @06:39PM (#4987570)
Medical Journals tend not to be interested in publishing 'negative' result papers. I can see why to an extent - they're not of particular value or interest to the vast majority of the journal's readers. They are however exceptionally valuable to anyone else running trials or doing research into similar drugs in the future, so there should be, IMO, somewhere where those papers can be made publically accesible.
Re:Something Doesn't Add Up: John Ioannidis and medical stat... by archgoon on Sunday December 30, 2012 @07:57PM (#4987791)
"Journal of Plausible, but Ultimately Unworkable Hypotheses"

Problem is, you don't want a high publication count in such a hypothetical journal. That's the other side of publication bias.

Re:Something Doesn't Add Up: John Ioannidis and medical stat... by alanctgardner2 on Sunday December 30, 2012 @11:52PM (#4988428)
But we have computers! This would be final step of proposing a project; check that a negative, peer-reviewed version hasn't been published. Putting them in separate journals actually solves a lot of the complaints in this thread
Re:Something Doesn't Add Up: John Ioannidis and medical stat... by archgoon on Monday December 31, 2012 @12:59AM (#4988559)
Sorry for the confusion, that wasn't my point. The issue isn't that there would be a ton of research to wade through, the issue is that scientists whose research didn't pan now have a track record of failure. I can easily imagine a academic position hiring conversation going like "This guy has a higher failure to success ratio than average, and his successful projects aren't above average."

EDIT: But perhaps I'm being too cynical (must be the New Year). Such a journal has been constructed.

http://www.jnrbm.com/

With luck, it will be wildly (un)successful :).

Re:Something Doesn't Add Up: John Ioannidis and medical stat... by tedunangst on Monday December 31, 2012 @12:57AM (#4988554)
You as a researcher do not want your name to appear in such a journal. Especially not lots of times.
Re:Something Doesn't Add Up: John Ioannidis and medical stat... by munin on Sunday December 30, 2012 @06:44PM (#4987585)
yes. most likely, what the supervisor means is that the research should have some kind of story and conclusion. an 'unsuccessful' piece of research is one that does not say anything, i.e. "well we tried to replicate this other paper and you know what, we did" or "I had an idea and I thought really hard about it for about a year, but nothing really concrete ever presented itself in either theory or experimentation".

a lot of people are kind of grumpy about null hypotheses not being published, for example papers of the form "we tried X and weren't able to make it work, here is how we tried". that is an 'unsuccessful' result (maybe you just suck), but some people would like to be able to read through stories of failure to see if there is a way to be successful. the journal of the null hypothesis (http://www.jasnh.com/) is an effort to do something like this, but it's generally the punchline of jokes...