Neural networks are "universal" function approximators in the sense that given sufficient number of hidden layer units and training data, they can approximate any Borel measurable function. This makes them theoretically powerful. (There are issues with this "universalness" claim though when doing cognitive modelling. See "Cognition and the computational power of connectionist networks", Hadley at http://www.informaworld.com/smpp/content~db=all?content=10.1...)

In practice, the best handwritten digits classifier is still neural networks. Lecun keeps scores at http://yann.lecun.com/exdb/mnist/.

Are there challenges with training neural networks? Certainly. But for many applications, there are good reasons to use a neural network. For instance, inference in a neural network is fast, and the learnt model is small in size (in feedforward networks, just a few large matrices).

SVMs on the other hand are "easier" to train, but care has to be taken to use the "right" kernel, and the learnt model in practice has to, in many cases, carry around with it up to half or more of the training data set (ie, the support vectors), making the model bigger in size, and inference slower (although there is work in making a sparser set of support vectors. See "An Effective Method of Pruning Support Vector Machine Classifiers", Liang, at http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5342...).

I know neural network research is kinda old school these days, has been overhyped in the past, and SVM research is kinda "in" nowadays, but that's no reason to think SVM is "much, much better" than ANN.