News for hackers. Stuff that matters.
Re:Some thoughts on machine learning with small data by workingon on Friday June 03, 2022 @06:05AM (#31606365)
I would take the opposite approach here. Instead of overfitting, bake probability into your model. Whether using a Bayesian weights or using an ad-hoc version (I.e. dropout and batch normalization), you can do your predictions in ensemble and look at the deviation of predictions. the combination of this method and a small dataset usually leads to wider ranges of prediction, which can be interpreted as a model uncertainty. When using these techniques we have found smaller datasets lead to more uncertainty, but it often will bound the error. I think this is much more useful than assuming a small dataset means an overfit model will do good on future data.

There are no comments on this comment yet.