Re:Why deep learning works even though it shouldn’t
by machinelearning on Wednesday October 21, 2020 @06:12PM
(#24852807)
I think you misunderstood the point of deep double descent. The x-axis is not number of training epochs, it is model capacity.
I think you'd be interested in https://arxiv.org/abs/1611.03530. It discusses how SGD is an implicit regularizer. We also actually want high variance weights for symmetry breaking.
I think you'd be interested in https://arxiv.org/abs/1611.03530. It discusses how SGD is an implicit regularizer. We also actually want high variance weights for symmetry breaking.