Re:Double Descent in Human Learning
by esafak on Wednesday April 26, 2023 @01:56AM
(#35710032)
Gradient descent finds the minimum L2 norm solution of the least squares? So adding a L2 regularization term should do nothing if the objective is convex, right? Is this common knowledge? I must have missed the memo, or be getting rusty.
[ Parent
| on HN ]
The formal claim is in Appendix A here: https://arxiv.org/pdf/2303.14151.pdf
It finds the minimum with regards to the training data. The double descent phenomenon is about error on unseen data.
There’s definitely more to the picture than that - it’s really about how models in the interpolation region behave (see https://mlu-explain.github.io/double-descent/ or https://mlu-explain.github.io/double-descent2/)
Which assumption of the bias-variance decomposition is flawed? I assume the variance term must not be increasing as supposed, because overparameterization enables sparsity?
It depends on the inputs. E.g., if two elements of the input are highly correlated, the network can assign an arbitrarily high positive weight to one element and then correct this by assigning an arbitrarily high negative weight to the other, with the total contribution of the two elements being reasonable. However, if these input elements are not as tightly correlated in the test data, the results may be bad. L2 regularisation ensures that this does not happen. (In practice, the effect will be mitigated by the activation function, but weights do blow up in training, leading to "saturated" models, which exhibit weird behaviours.)
The objective is usually not convex in deep learning.