News for hackers. Stuff that matters.
Re:Machine Learning: Curse of Dimensionality by Fede_V on Wednesday August 05, 2015 @04:37PM (#10012664)
Local minima are only nearly as good as global minima in very very deep networks.
Re:Machine Learning: Curse of Dimensionality by deepnet on Wednesday August 05, 2015 @04:56PM (#10012809)
And yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present.

(Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4

On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that :

" All minima are global minima, and are linked to each other in a continuous manifold. "

Which is pretty exciting because the received wisdom was local-minima are a really big problem.