AAAS: Machine learning 'causing science crisis'

Re:AAAS: Machine learning 'causing science crisis' by Waterluvian on Sunday February 17, 2019 @07:35AM (#19184155)

We need to find resources to fund "Failures in Science" journals that exclusively seek to publish interesting research that went nowhere. Personally I'd find these far more interesting to study. "Here's some background. Here's a pretty logical, plausible hypothesis we came up with and how, here's our experiment, here's our results, here's our thoughts as to why we were wildly wrong."

[ Parent | on HN ]

Re:AAAS: Machine learning 'causing science crisis' by parmegv on Sunday February 17, 2019 @08:25AM (#19184327)

There is a new conference in cryptology that promotes exactly this: CFAIL.

From their 2019 website[0]: "Do you have insightful and exciting work sitting in a drawer somewhere because it never quite panned out? Are you willing to share your failed approaches so that others can learn from them without having to re-travel the same road? Are you tired of reading papers that pretend the incremental result they happened to achieve was well-motivated and was their goal all along?

We are! That’s why we are founding a new conference: a place for papers that describe instructive failures or not-yet-successes, as they may prefer to be called."

[0] https://www.cfail2019.com/

[ Parent | on HN ]

Re:AAAS: Machine learning 'causing science crisis' by hannob on Sunday February 17, 2019 @08:35AM (#19184367)

We have that. People don't use it.

PLOS e.g. specifically stated multiple times that they'll publish what meets their quality standards regardless of a positive or negative outcome.

The problem is: Even if you publish failed research it won't get cited as much. And people still use citation metrics to evaluate "quality" of science.

Just having journals that publish your "failed" research is a good start, but it's not changing the incentive structure.

[ Parent | on HN ]

Re:AAAS: Machine learning 'causing science crisis' by Waterluvian on Sunday February 17, 2019 @06:02PM (#19187455)

Agreed. This isn't a one-action problem to solve. But we need to lay the foundation for alternative incentives to be possible.

[ Parent | on HN ]

Re:AAAS: Machine learning 'causing science crisis' by thegabriele on Sunday February 17, 2019 @07:45AM (#19184192)

Instead of allows failures (which would be also good btw), i would increase the time at disposal needed to pubblish something worthwhile.

The greatest thinkers in history had no pressure outside their own urgence to solve/explore. Surely, not everyone is Galileo but today science need that kind of freedom.

[ Parent | on HN ]

Re:AAAS: Machine learning 'causing science crisis' by michaelmior on Sunday February 17, 2019 @08:50AM (#19184430)

The FADS workshop[0] at VLDB was a nice attempt at this although I think the work ended up focusing on known historical failures as opposed to failed work that was not otherwise published.

[0] https://fads.ws/

[ Parent | on HN ]

Re:AAAS: Machine learning 'causing science crisis' by trextrex on Sunday February 17, 2019 @11:50AM (#19185342)

The problem with a failed experiment, at least in machine learning, is that it's not always clear what caused the failure. Was it not enough training, was there some small "trick" the model could have used, were the hyper parameters off etc.

There are an infinite set of configurations that could fail, and it's not sufficiently useful to know they failed without understanding why they failed. And analysing failures in a useful way is an extremely difficult and fundamental problem. On the other hand, a successful experiment is an extremely rare event and hence interesting by itself.

[ Parent | on HN ]