News for hackers. Stuff that matters.
Re:Machine Learning: Curse of Dimensionality by deepnet on Wednesday August 05, 2015 @02:01PM (#10011587)
As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima.

In David Silver's RLDM 2015 talk @20 mins in

http://videolectures.net/rldm2015_silver_reinforcement_learn...

he recounts this as his experience.

the paper The Loss Surfaces of Multilayer Networks goes in depth http://arxiv.org/abs/1412.0233

[edit] and the paper Qualitative Characterisation of Neural Net Optimisation http://arxiv.org/abs/1412.6544v4

Re:Machine Learning: Curse of Dimensionality by Fede_V on Wednesday August 05, 2015 @04:37PM (#10012664)
Local minima are only nearly as good as global minima in very very deep networks.
Re:Machine Learning: Curse of Dimensionality by deepnet on Wednesday August 05, 2015 @04:56PM (#10012809)
And yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present.

(Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4

On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that :

" All minima are global minima, and are linked to each other in a continuous manifold. "

Which is pretty exciting because the received wisdom was local-minima are a really big problem.