The scientific revolution in computer vision right now is due to deep convnets, trained in a supervised way using backprop and SGD. All of the systems we're talking about that have started blowing away records are members of this family, and were trained this way. If Alex Krizhevsky had not entered ImageNet 2012, we would not be having this conversation (in part because I probably would never have gotten curious enough about it to leave my home territories of systems and programming languages). Second order methods, unsupervised pre-training, RBMs, graphical models, etc. etc. etc. were exciting for those inside the field, definitely provided encouragement to those optimistic about deep models, and still might prove important, but they have had little impact and visibility to skeptical people outside, in the way that entering a computer vision competition and murdering all the computer vision systems did.
The SGD we use today actually has some strong ties back to that second order optimization work - see some papers by Ilya Sutskever relating a special form of momentum back to second order methods like HF. http://www.cs.utoronto.ca/~ilya/pubs/2013/1051_2.pdf His dissertation covers this at some length as well.
Using Adagrad, Adadelta, etc. isn't really SGD as it was back in 2012, and this years entrant "GoogLeNet" basically halved the error again using these and other tricks (we think) - which is even more impressive considering 11% - 6.7% is a HUGE increase in difficulty, just my 2 cents.
However, there is a good reason the colloquial name for these things is "Alexnets"... that work was truly incredible and has not stopped behind the doors of Google I don't think.
Single layers can indeed be made to have convex error surfaces fairly easily. One can do so by matching the error/loss function with the squashing/link function. What some old NN folks got wrong was mixing up square loss with logistic function, that is an unhealthy mix. Now if one were to use KL divergence instead of square loss then one would indeed have a convex loss function. In fact this would be nothing but logistic regression. One can however push this idea further, with any choice of a monotonic squashing function one can derive a 'matching' loss that would give you a convex loss. Classical statisticians know this and call it with a different name: canonical generalized linear models. I am not from that tribe, mine is more ML we may perhaps call it minimizing Bregman loss.
Just so that its clear I am talking about single layer networks not single hidden layer networks, there are plenty of cases were the former is useful.
> There have been no magical optimization breakthroughs
It is arguable whether Hessian free methods, contrastive divergence or auto-encoder based training methods qualify as 'breakthroughs' but they have definitely equipped invigorated researchers in this broad area with their capabilities.