One limitation of using GPUs is the fact that they have comparatively little on-board memory (max 6GB for now) and shuffling data in/out of it is expensive (because of communication costs).
Now there's also the idea that you can maybe obtain something interesting from a learning perspective when training on 16k cores. Because you have to essentially train a bunch of different models, that have different parameters and see potentially different subsets of your training set, ultimately you perform some sort of model averaging (since training is doing it for you). In essence, by doing distributed gradient descent you are potentially training a clever ensemble of networks, which arguably could give better generalization.
Can you have an ensemble of GPUs? Now that's an interesting question too :)
The ImageNet winning model is a essentially glorified ConvNet :)
AI research died off in the 1960s and 1970s because the results didn't even get close to matching the expectations. Machine learning's current commercial success ensures that won't happen in the foreseeable future. That said, it's worth tempering our enthusiasm. To put it perspective, deep networks are extremely far from recreating general, human-level intelligence. There's a long way to go, and I'm not convinced the path we're on is the 'correct' one to reaching the kind of general AI people see in the movies.
We have the learning part down (somewhat), but the intelligence part not so much.
Before Deep Learning, Neural Networks required a lot of hand tuning. Without that kind of tuning, Neural Networks with more than 1 or 2 hidden layers tended to perform very badly, because they would get stuck in plateaus or poor local optima.
In contrast, (one facet of) Deep Learning focuses on using unsupervised learning to better initialize NNs, which in turn allows for much higher levels of accuracy. In essence, Deep Learning helps pick the NN parameters automatically. So it's actually decreasing the amount of hand tuning needed.
This is in contrast to hand engineering features for your particular domain which, unsurprisingly, doesn't generalise well.
How far this could go is hard to say right now but things like this [2] (10 million images, 1 week training, 16,000 cores) are bringing enormous processing power to bear for the first time on these kind of problems.
Is anyone able to comment on what moving to custom hardware will mean for performance on these kind of tasks?
[1] http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf
[2] http://static.googleusercontent.com/external_content/untrust...