Re:An overview of the theory of overparameterized machine le...
by naomisperfume on Tuesday September 21, 2021 @01:07PM
(#28607163)
Do you have any good references of this phenomenon in hierarchical models?
Re:An overview of the theory of overparameterized machine le...
by bigfudge on Tuesday September 21, 2021 @06:48PM
(#28610987)
German and Hill has a brief intro and some references.
Re:An overview of the theory of overparameterized machine le...
by sjg007 on Tuesday September 21, 2021 @04:11PM
(#28609355)
This is likely part of the reason. The only problem is said models require a lot of data but Humans can learn from a very small number of examples.
Re:An overview of the theory of overparameterized machine le...
by sdenton4 on Tuesday September 21, 2021 @05:34PM
(#28610285)
Humans are continuously pretrained on a variety of tasks, though. Teaching a kid to say one word takes about a year...
Re:An overview of the theory of overparameterized machine le...
by gota on Tuesday September 21, 2021 @10:40PM
(#28612585)
And the very same system that is being trained to say a word is also being trained to recognize intent from intonation all the while. As soon as it can say it, the child will likely use that one word with different tones to mean different thing successfully.
We are insanely complex machines...
Re:An overview of the theory of overparameterized machine le...
by oleg_myrk on Tuesday September 21, 2021 @05:19PM
(#28610105)
Any chance you could share a link to a relevant paper?
Re:An overview of the theory of overparameterized machine le...
by talolard on Tuesday September 21, 2021 @01:48PM
(#28607618)
I don’t know if it’s correct , but I often think of a classification model as learning the parameters of a dirchlet distribution with the final softmax layer being a sample from it
A similarly surprising result from an adjacent community, Bayesian Statistics, is that in the case of hierarchical models, increasing your number of parameters can paradoxically reduce overfitting.
The scale of parameters in Bayesian model's is no where near that of these deep neural nets, but nonetheless this is a similarly shocking result since typically adding parameters is penalized when model building.
It's a bit more explainable in Bayesian stats since what you're using some parameters for is limiting the impact of more granular parameters (i.e. you're learning a prior probability distribution for the other parameters, which prevents extreme overfitting in cases with less information).
I wouldn't be too surprised if eventually we realized there was a similar cause preventing overfitting is overparameterized ml models.