News for hackers. Stuff that matters.

A comment on Fitting to Noise or Nothing at All: Machine Learning in Markets. Parent comment

Re:Fitting to Noise or Nothing at All: Machine Learning in M... by shimon on Sunday August 06, 2017 @09:08PM (#14944637)
Sure. But for approximately the same level of effort, you could generate a new paper in the same template. It would likely be easier to get that "novel result" published than the refutation.
[ Parent | on HN ]
Re:Fitting to Noise or Nothing at All: Machine Learning in M... by beagle3 on Monday August 07, 2017 @12:27AM (#14945304)
And that's generally the truth in every scientific field.

Right now you have a "reproduction renaissance" in psychology and cancer, where you can get published, and get acknowledged, if you tear down a high level paper. But in general, unless you aim at refuting the highest level, it is hard to get published and advances your career nowhere, so people have no incentive to do it.

[ Parent | on HN ]
Re:Fitting to Noise or Nothing at All: Machine Learning in M... by nonbel on Monday August 07, 2017 @08:44AM (#14947074)
A non-replication does not "tear down" a paper... It is not any kind of "attack", your attitude is one of the sources of the problem.

Edit:

When I did medical research I wished people would try to replicate my findings so I could see how consistent the results were and how well my model fit future data... In fact it was really annoying that it was unlikely anyone would ever try to do a direct replication.

[ Parent | on HN ]
Re:Fitting to Noise or Nothing at All: Machine Learning in M... by beagle3 on Monday August 07, 2017 @05:27PM (#14951412)
My attitude? What attitude is that?

A single non-replication does not. Consistent failure to replicate, after communicating with the original author, does.

See e.g.

http://www.slate.com/articles/health_and_science/cover_story...

http://www.nature.com/nature/journal/v483/n7391/full/483531a...

Or just google "reproducibility crisis", the two most common subjects that come up are cancer research and psychology research.

[ Parent | on HN ]
Re:Fitting to Noise or Nothing at All: Machine Learning in M... by elviejo on Monday August 07, 2017 @11:24AM (#14948169)
That is the problem: Why wouldn't try to do a replication?

Because the incentives are to provide "new research", not to confirm or deny a known result.

It seem to me that most probably no-one has gotten tenure by reproducing, disproving results.

[ Parent | on HN ]