Re:Machine Learning: Curse of Dimensionality
by Fede_V on Wednesday August 05, 2015 @04:37PM
(#10012664)
Local minima are only nearly as good as global minima in very very deep networks.
[ Parent
| on HN ]
Re:Machine Learning: Curse of Dimensionality
by deepnet on Wednesday August 05, 2015 @04:56PM
(#10012809)
And yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present.
(Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4
On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that :
" All minima are global minima, and are linked to each other in a continuous manifold. "
Which is pretty exciting because the received wisdom was local-minima are a really big problem.