A Code Glitch May Have Caused Errors in More Than a Hundred Published Studies

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by Aperocky on Saturday October 12, 2019 @06:06PM (#21236701)

What’s worse, sometimes the bugs are deliberate. A slight tuning of a parameter that can be explained away but having a significant impact on the output data. This alone with statistical bullshiting ( there’s always a statistical manipulation that will create the desired result), renders 80% of all research practically worthless.

edit: code change -> parameter tuning.

[ Parent | on HN ]

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by bonoboTP on Saturday October 12, 2019 @06:14PM (#21236747)

Or in machine learning if that manipulation is not enough to get state of the art results, just don't cite anyone who is better! Now you're SOTA, congrats. Happens way more than people realize. Not as much in the top 2-3 conferences.

[ Parent | on HN ]

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by GordonS on Saturday October 12, 2019 @06:25PM (#21236803)

I can imagine this might happen in theory, but have you come across evidence of this in practice?

[ Parent | on HN ]

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by Aperocky on Saturday October 12, 2019 @07:19PM (#21237028)

Parameter tuning is prevasive, but I guess I went too far to suggest that it's a code change.

[ Parent | on HN ]

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by haihaibye on Sunday October 13, 2019 @01:46AM (#21238591)

Another common trick is to tune your program to a dataset then run all of the other programs with default parameters.

Bam, you've got a benchmark where you're fastest, just like everyone else.

[ Parent | on HN ]