Re:Deep learning for twelve hour precipitation forecasts
by zamfi on Sunday February 12, 2023 @04:09PM
(#34766993)
Oh, we're on the same page -- I don't think they're engaging in fraud currently, just overlooking the implications of their process, as you note.
But explicitly holding out a test set, throwing it away because your results on that set were no good, trying again, and then representing that held-out set as, indeed, held-out in later work -- as the parent suggested? I'm not sure what other word to use there.
Fraud is not needed. My intuition is that most people publishing in machine learning have simply not thought carefully enough about their experimental methods. They're not defrauding anyone, they're misunderstanding their own results.
And since the majority of published papers do the same, it's very difficult to convince anyone what the good practice is.
Good on you for doing it the right way.