Re:Too much efficiency makes everything worse
by Filligree on Friday November 11, 2022 @07:56AM
(#33560240)
Can’t provide a reference, but I can confirm that this is common knowledge. It’s why e.g. GPT-3 outperforms GPT-2.
Though as stable diffusion shows, network architecture still matters a lot!
Note that the article points out you’ll get more overfitting as your number or parameters approaches that of the training set, which is what I suspect you’ve seen. The trend does reverse later on, but only once the parameter count is orders of magnitude beyond that point, and I don’t know if that ever happens outside of ML. It’s a lot of parameters.
Re:Too much efficiency makes everything worse
by marcosdumay on Friday November 11, 2022 @09:51AM
(#33561488)
It it very counterintuitive. It is also a very common observation that has taken everybody by surprise for almost 2 decades by now. At the beginning, people were very resistant to the idea, even when every experiment confirmed it.
The catch is that you need a huge amount of data to train those.
It also seems to have limits. There has been a few well documented cases where our current huge and very well trained kind of networks got errors there were lower than the rate of mislabeling of the data.
Do you have some references for this claim? For me, it seems counterintuitive.