Previously, neural networks were trained by taking single steps down the direction of sharpest gradient for the network. However, in deep networks with lots of layers, the backprop algorithm (which I assume you already know about) got stuck in local minima.

Deep learning got started when Hinton observed that a certain way of training restricted Boltzmann machines wouldn't get stuck as easily, and hence by pretraining the network as if it were an RBM and then switching to backprop, it wouldn't get stuck in a local minimum as early.

As I understand it, nowadays the best method looks something like a generalization of Newton's Method, wherein the direction you move takes into account the second differential and not just the first differential or direction of sharpest descent. You move furthest in the directions that curve the least, and move the least in the directions that are most sharply curved. It turns out that this (plus some other tricks) make it way easier to follow continuous gradients in big weird parameter spaces, so now it's possible to train deep nets, which are a kind of continuous gradient in a big weird parameter space.

Tl/dr: People figured out how to move better through the parameter space of neural networks, by taking into account the second differential plus some other tricks. So now we can train deeper nets.

While there are newer algorithms (almost all first order btw), one of the most commonly used for deep convnets, sgd with momentum, is from the 80s. It really is mostly about computing power - one gtx 1080 has more tflops than world's fastest supercomputer till 2001 [0]. The actual speed difference is probably an order of magnitude larger due to an absence of communication overhead and latency inherent in sharing work across thousands of separate cpus and lots of slow ram. That would make one gtx 1080 equivalent - for neural net training purposes - to a supercomputer from 2004.

[0] https://en.wikipedia.org/wiki/History_of_supercomputing#Hist...

More specifically techniques such as Momentumn https://www.willamette.edu/~gorr/classes/cs449/momrate.html (great article visualizing the problem space discussed above) and thigns like RMSPROP http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slid... are used.

The rectified linear unit also converges about 75% faster than the classic Sigmoid or Tanh. the ReLu is F(x) = max(0, x) so the gradient propogated is 1 rather than a more "curvy" value.

Thank you! That makes a lot of sense.

Thanks, this explanation made a lot of sense to me.