I also agree but while neural networks are a sort of black box they are also entirely mathematical. Also some of the most impressive current research is in the area of deep learners, an example of which is a type of neural network: RBMs and Deep Belief Nets. In particular Deep Belief Nets have the added advantaged of being able to display their interal abstractions and state and so are not so boxed up.

I will concede that, but they're a very roundabout way to building a classifier. An SVM, for example, is really much simpler and works much, much better than a neural network. I'm not familiar with RBMs and DBNs, so I can't comment on that, sadly...

SVMs may make a lot of intuitive sense due to the intuitive nature of the max-margin method of building a classifier, but that doesn't make them "much, much better" than a neural network.

Neural networks are "universal" function approximators in the sense that given sufficient number of hidden layer units and training data, they can approximate any Borel measurable function. This makes them theoretically powerful. (There are issues with this "universalness" claim though when doing cognitive modelling. See "Cognition and the computational power of connectionist networks", Hadley at http://www.informaworld.com/smpp/content~db=all?content=10.1...)

In practice, the best handwritten digits classifier is still neural networks. Lecun keeps scores at http://yann.lecun.com/exdb/mnist/.

Are there challenges with training neural networks? Certainly. But for many applications, there are good reasons to use a neural network. For instance, inference in a neural network is fast, and the learnt model is small in size (in feedforward networks, just a few large matrices).

SVMs on the other hand are "easier" to train, but care has to be taken to use the "right" kernel, and the learnt model in practice has to, in many cases, carry around with it up to half or more of the training data set (ie, the support vectors), making the model bigger in size, and inference slower (although there is work in making a sparser set of support vectors. See "An Effective Method of Pruning Support Vector Machine Classifiers", Liang, at http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5342...).

I know neural network research is kinda old school these days, has been overhyped in the past, and SVM research is kinda "in" nowadays, but that's no reason to think SVM is "much, much better" than ANN.

Hmm, I didn't know that about handwritten digits, thanks. I agree that you have to carry around the SVMs, but I've gotten better results with an SVM. That said, it might have just been a suitable problem.

I would remove one maybe both muches. I agree that in general SVMs are better but they are also every bit as black box. Deep belief networks though, are on another level. literally, svms are depth 2, dbn's are depth unbounded, i think.

http://www.youtube.com/watch?v=VdIURAu1-aU

That was just an example of my original point, though. These sorts of things perform better than "natural" processes, even though they aren't as easy to understand.

On very large real world data sets my experience has been the opposite.

Typical academic hubris.

Neural networks aren't toys. I've seen billion dollar systems based on them. And I believe an RBM was one of the strongest components of the winning Netflix algorithm.

The ML academics are biased against neural nets because they're difficult to analyze mathematically making it hard to fill blackboards with impressive looking equations.