News for hackers. Stuff that matters.
Re:We're Incentivizing Bad Science by Vaslo on Tuesday October 29, 2019 @12:21PM (#21388906)
I agree with this article and have always thought about this but never voiced it.

I think a few things could improve the quality and discovery of published papers:

-After so many publications, could it be mandatory for random samples in that author’s publication be tested? I get that there is a limit on resources, but some advanced undergrads could do this with guidance.

-I would love to see some version of a journal of failures. That is, well intentioned research that had poor outcomes. It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. Unfortunately, there is no “Journal of Failed Chemistry.” Only research that ostensibly make a contribution with a clear outcome get published. So much time wasted experimenting where you could save another scientist time and encourage them down another path.

Re:We're Incentivizing Bad Science by djtango on Tuesday October 29, 2019 @12:34PM (#21389052)
>It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that.

This mirrors my experience where a lot of the post-docs carried war stories of things like lab/country-specific humidity playing a role in synthetic methods succeeding (or failing). There were a lot of dark arts/tricks of the trade that people carried around with them: stuff like going that extra mile to dry things of water super thoroughly (even if it was not mentioned in the paper we were referencing).

Re:We're Incentivizing Bad Science by ISL on Tuesday October 29, 2019 @05:21PM (#21392179)
Many "dark arts" aren't known as dark arts to the people who know them. They're just the things you always do, because if you don't do them, nothing works.

An approximate CS analog: Writing great commit messages, and using an SCM.

You don't have to do them, nobody writes it up because those who know regard it as trivial, but if you don't do them, almost nothing works. Everyone who knows what they're doing does it.

Re:We're Incentivizing Bad Science by djtango on Wednesday October 30, 2019 @02:01PM (#21400595)
>An approximate CS analog: Writing great commit messages, and using an SCM.

This is definitely not the analogue. The analogue is always doing a clean install of the OS before running your experiment. Or only ever using Arch version x.y.z for replicating lab 1 and maybe a.b.c for replicating lab 2.

It's knowing all the magic undocumented JVM flags ahead of running the application.

Some you know to use as part of war scars/best practices. Some are just pure inside information from working in that lab or having a personal relationship with people in that lab.

Re:We're Incentivizing Bad Science by Al-Khwarizmi on Tuesday October 29, 2019 @03:04PM (#21390853)
There are a bunch of journals like that. If you google "Journal of Negative Results", you will see several for different fields.

However, for some I know, it's really difficult to get in, because there are many more journals of positive than negative results, and there are probably many more negative results than positive... so your negative result had better be especially interesting.

Re:We're Incentivizing Bad Science by bumby on Tuesday October 29, 2019 @02:21PM (#21390419)
I recently noticed SURE journal, intent on publishing "unsurprising" economic articles.

I just wish there were more, or at least more journals rewarding sound methodology instead of unsound "results". Who knows how much effort is wasted reproducing sound, but unremarkable studies simply because they weren't published

Re:We're Incentivizing Bad Science by codegladiator on Tuesday October 29, 2019 @12:38PM (#21389109)
Partly the issue is because of the naming... Like "poor outcomes" and "failed"...

Obviously there would be no awards for the poor outcomes and failed results and we would be back to square one where no one is incentivized to look at them positively.

Re:We're Incentivizing Bad Science by tasogare on Tuesday October 29, 2019 @12:41PM (#21389140)
What are you waiting for creating one? Motivate a few scientists to form a credible review comity, fill one paper [1] and voila you have the journal you dreamt about on a recognized and perennial web platform.

This is of course some work, but if you really want it to exists it’s better not to wait for someone else to do it. Also allowing anonymous authors would help I think (to avoid the name being associated with failure).

[1] https://www.openedition.org/15974?file=1

Edit: openedition.org is for humanities, but there probably is alternative for other fields.

Re:We're Incentivizing Bad Science by MaxBarraclough on Sunday November 03, 2019 @04:13PM (#21436648)
> What are you waiting for creating one?

Creating a journal is a tremendous task, and only highly respected researchers have any chance.

> if you really want it to exists it’s better not to wait for someone else to do it

Again, most people simply cannot create a credible journal. Let's be realistic here.

Re:We're Incentivizing Bad Science by cepp on Tuesday October 29, 2019 @02:07PM (#21390252)
On failures: many years ago I stumbled across the idea of a `CV of Failures` where you list all of the accomplishments you missed/were rejected from, etc. I think it's a very humbling idea and can see some applicability here.

Example: https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Fai...

Re:We're Incentivizing Bad Science by glofish on Tuesday October 29, 2019 @01:28PM (#21389749)
This is a variant of "the carrot and stick" approach that happens to be very short on the carrot and be very large on stick.

Very unlikely to be effective.

Time and again people think that when they have a problem it must mean the punishment was not severe enough. It won't work.

The problem is that good science is not rewarded enough. Nobody rewards you for having published reliable stuff five years ago.

Re:We're Incentivizing Bad Science by weberc2 on Tuesday October 29, 2019 @02:08PM (#21390259)
Unless the GP edited, there's no suggestion of punishment in his comment.
Re:We're Incentivizing Bad Science by wolfram74 on Tuesday October 29, 2019 @12:47PM (#21389216)
maybe a more serious version of the journal of irreproducible results? https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Resu...