Troubling Trends in Machine Learning Scholarship

Re:Troubling Trends in Machine Learning Scholarship by chriskanan on Wednesday July 11, 2018 @06:24AM (#17505772)

A major factor is that machine learning is so conference deadline driven. Does the main result look good enough? Submit it! Ablation studies are the last thing you do, and often you are pressed for time due to the deadline. Since so much work is done by PhD students, they are incentivized to get their work out as fast as possible because they need to graduate and papers in top-conferences play a huge factor in getting jobs in academia and industry.

I'm aware of multiple papers by top labs where their state-of-the-art results are really the result of some very minor change (e.g., some pre-processing technique, bigger images) that was swept under the rug in their paper. All the math and the complex model had minimal impact on the actual numeric result, with the actual reason being unpublishable in a top venue.

[ Parent | on HN ]

Re:Troubling Trends in Machine Learning Scholarship by at-fates-hands on Wednesday July 11, 2018 @09:47AM (#17506806)

Your entire first paragraph makes me cringe. Coming from an academic/research background, all of the things you mentioned goes against everything I was taught in terms of being rigorous with your research.

Is this more evidence we have really entered the "click baiting" era? Where the end result of the research is now secondary to getting published and into big conferences to get some notoriety?

[ Parent | on HN ]

Re:Troubling Trends in Machine Learning Scholarship by michaelt on Wednesday July 11, 2018 @12:02PM (#17507945)

  all of the things you mentioned goes against
  everything I was taught in terms of being
  rigorous with your research.

There are certain areas where "the system" asks people to do things right - but provides vast rewards for doing things wrong, if you can avoid detection.

If you're a pro cyclist who isn't doping, good for you! But there's no medal for finishing fourth; you'll be rewarded with medals and sponsorship cash for doping undetectably.

Academics whose rigorous papers aren't quite getting into top-tier journals are in a similar situation.

[ Parent | on HN ]

Re:Troubling Trends in Machine Learning Scholarship by 3pt14159 on Wednesday July 11, 2018 @11:03AM (#17507462)

I also struggle to understand why PhD led research in AI is considered so highly. Only very rarely does is hold a candle to team lead research in the private sector.

[ Parent | on HN ]