News for hackers. Stuff that matters.
Re:Understanding deep learning requires rethinking generaliz... by benlivengood on Thursday March 04, 2021 @02:54PM (#26347315)
This is only tangentially my field, so pure speculation.

I suppose it's possible that generalized minima are numerically more common than overfitted minima in an over-parameterized model, so probabilistically SGD will find a more general minima than not, regardless of regularization.

Re:Understanding deep learning requires rethinking generaliz... by hervature on Thursday March 04, 2021 @03:39PM (#26347946)
I think the general consensus (from my interactions) is that a local minima requires the gradient to vanish. When you have many dimensions, it's unlikely that they are all 0. Coupled with modern optimization methods (primarily momentum), this encourages the result to be in a shallow valley as opposed to a spiky minima. The leap of faith is equating shallow=general and spiky=overfitted.