Re:Why deep learning works even though it shouldn’t
by machinelearning on Wednesday October 21, 2020 @06:12PM
(#24852807)
I think you misunderstood the point of deep double descent. The x-axis is not number of training epochs, it is model capacity.
I think you'd be interested in https://arxiv.org/abs/1611.03530. It discusses how SGD is an implicit regularizer. We also actually want high variance weights for symmetry breaking.
Re:Why deep learning works even though it shouldn’t
by owenshen24 on Wednesday October 21, 2020 @02:11AM
(#24845186)
To add some more context, here's a rather readable summary:
https://www.greaterwrong.com/posts/FRv7ryoqtvSuqBxuT/underst...
Overall it looks like gradient descent is a strong regularizer -- we know it tends to prefer small and low-variance weights, for example. So part of deep generalization has to do with how SGD is able to pick "good" features early, and then optimization pushes the unimportant weights to zero later (hence lottery tickets).
[1] https://openai.com/blog/deep-double-descent/ and other papers. [2] https://arxiv.org/abs/1711.08856 and others.