As to the GP: Geoff Hinton (probably the most well-known neural networks researcher) said in his Coursera course that neural networks thrive at problems with a lot of structure that could be encoded, while simpler models like SVMs or Gaussian processes might be better for problems without as much deep structure to discover.
Also, a lot of the current research with neural networks involves using neural networks to learn better representations of data. These cleaner representations of data (which can be thought about as a sort of semantic PCA) often make classification far easier, which explains the great results. Learning representations also makes transfer learning (transferring knowledge from one domain to another) much easier/more possible.
Recent papers show that NN are coming back, but I think most of the speech recognition out there is still Hidden Markov Models and most of the image recognition is definitely based on tailored detectors & descriptors. This is especially true when you do these things on the users' mobile devices.
Definitely image recognition: http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf
Speech recognition: http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf
Natural language processing: http://www.socher.org/index.php/DeepLearningTutorial/DeepLea..., http://aclweb.org/anthology/N/N13/N13-1090.pdf
If you're into kaggle competitions: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it...
I don't think there are going to be any further major advances in eg SVMs or random forests (famous last words maybe...) Neural nets, on the other hand, are just scratching the surface of what's possible. So right now they are state of the art in some historically very difficult areas. But these are early days still.