Here is an interesting paper that pinpoints the influence of regularization on learning with Neural networks. From the paper:
> Our central finding can be summarized as:
Deep neural networks easily fit random labels.
and later:
> While simple to state, this observation has profound implications from a statistical learning perspective:
> 1. The effective capacity of neural networks is large enough for a brute-force memorization of the entire data set.
> 2. Even optimization on random labels remains easy. In fact, training time increases only by a small constant factor compared with training on the true labels.
> 3. Randomizing labels is solely a data transformation, leaving all other properties of the learning problem unchanged.
I subscribe to Schmidhuber's notion that algorithms that rely upon randomness to select parameters have a very illusory "simplicity" that relies upon externalizing the complexity of the PRNG. http://people.idsia.ch/~juergen/newai/
Here is an interesting paper that pinpoints the influence of regularization on learning with Neural networks. From the paper:
> Our central finding can be summarized as:
Deep neural networks easily fit random labels.
and later:
> While simple to state, this observation has profound implications from a statistical learning perspective:
> 1. The effective capacity of neural networks is large enough for a brute-force memorization of the entire data set.
> 2. Even optimization on random labels remains easy. In fact, training time increases only by a small constant factor compared with training on the true labels.
> 3. Randomizing labels is solely a data transformation, leaving all other properties of the learning problem unchanged.
I subscribe to Schmidhuber's notion that algorithms that rely upon randomness to select parameters have a very illusory "simplicity" that relies upon externalizing the complexity of the PRNG. http://people.idsia.ch/~juergen/newai/