News for hackers. Stuff that matters.
Re:Double Descent in Human Learning by clbrmbr on Wednesday April 26, 2023 @08:26AM (#35712588)
In the polynomial fit case: it’s pretty clear to me that if the number of parameters is higher than the degrees of freedom in the data, then the model can simply memorize the input.

That doesn’t explain why it generalizes well, though.

Re:Double Descent in Human Learning by occamrazor on Wednesday April 26, 2023 @01:08PM (#35716941)
It”s only mentioned at the end. If you use some regularization, you have many ways to fit the data (almost) perfectly, and you choose the one with the smallest coefficients, which is often one that generalizes well. And even without explicit regularization, some training methods implicitly apply sone form of regularization, when the number of parameters exceeds the number of data points.