Some thoughts on machine learning with small data

Re:Some thoughts on machine learning with small data by nri on Friday June 03, 2022 @05:23AM (#31606040)

Interesting, that advice is exactly the opposite of the common wisdom. You mean overfitting as in aggressively maximizing your crossvalidation scores? How do you decide for which problems that is a good approach and for which a more conservative approach is better?

[ Parent | on HN ]

Re:Some thoughts on machine learning with small data by smartmic on Friday June 03, 2022 @05:46AM (#31606219)

Well, I mean not actively maximizing your CV score, just accepting normally insufficient CV scores.

For example, in a tree based regression with a very small number of training samples, the leaves will almost resemble some of the training samples. A good generalization might not be possible at all. But this is not too bad if you know that your prediction samples have high similarity with at least one of the training samples. In the end, this is an edge case for Machine Learning but goes more into the direction of Expert Systems.

[ Parent | on HN ]

Re:Some thoughts on machine learning with small data by isoprophlex on Friday June 03, 2022 @06:18AM (#31606442)

Is this a complicated way of saying "just use k-nearest neighbors"?

[ Parent | on HN ]

Re:Some thoughts on machine learning with small data by crabbygrabby on Friday June 03, 2022 @12:16PM (#31610368)

In a lot of cases knn doesn't model systems appropriately unless you understand the space very well. At least knn offers some sort of theoretical assumption of the bayes error rate though so it's a decent tool.

[ Parent | on HN ]

Re:Some thoughts on machine learning with small data by MrMan on Friday June 03, 2022 @06:26AM (#31606501)

I think its implying that ML often works because the manifold assumption is partially correct. so samples will lurk in the same neighborhood.

[ Parent | on HN ]