* AI researchers do things that get good results faster out of Nvidia cards
* Nvidia makes their cards faster at the things AI researchers are doing
It's getting good results. We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum.
At least until there's some other insight along the way.
Also, I'd like to point out that we've seen some big breakthroughs in the past 10 years. But for the past 10 years, the whole field of deep learning has been looked at with skepticism and has been very niche. Over the past couple of years, a lot of money and resources have been put in place to pursue this area of research. More money doesn't necessarily mean more results, but there are many many more people working on these problems than ever before. A lot of them are legitimately brilliant researchers in the prime of their careers. I think there's still more to see.
I am concerned about an Nvidia monopoly around deep learning hardware. They give away tons of free cards to deep learning research groups, but at some point they'll want people to start buying. I assume they expect that will be the enterprise set, but if suddenly they manage to move all their capable deep learning cards to Teslas only (which have a huge markup), it will put the hobbyist deep learning developer at a big disadvantage. The only check and balance on that is the fact that Nvidia makes cards for gamers too, and gaming card competition is still somewhat robust, so any technology that gives enterprise a big boost will probably make its way to their flagship gaming cards quickly. Nvidia's only real competition, AMD, is so far behind that they might as well not be in the business. As someone who usually roots for the underdog, it pains me to see AMD fumble so badly in this whole area.
Quantum computing could be a new hill, but I think that's a ways out and I don't know enough about the topic to speak with any real confidence.
1. GPU as a service: While all major providers (AWS, Azure) offer Teslas on their servers, there is no AMD on the cloud (that I know of)
2. Key libraries: Nvidia comes with matrix libraries and cuDNN out of the box, while for AMD, there are only open source offerings that are a bit difficult to manage.
But, If you write your own software or rely on open source, AMD is quite performant and affordable. The problem is that it is really obscure. So, yes, they are really bad at marketing and if you are looking at them as a user instead of as a developer, they are invisible.
AMD hardware is more or less the same as Nvidia; Nvidia doesn't really have any special sauces or patents that make their cards better than AMD. And you're right: in a lot of benchmarks, AMD is either faster or (at least) better-bang-for-buck than Nvidia.
Yet AMD doesn't have community support in software or cloud adoption or key libraries. The deep learning community has gone for what's easiest to work with and most readily available. Even though you might conceivably get some performance gain from AMD, it's outweighed by the amount of software that's been written in CUDA already, and how quickly that allows you to move.
Nvidia has taken an early lead and jumped on it, while AMD has only fallen further behind. I think one company just fundamentally understands the trend more than the other. But, for what is essentially hardware that answers the question, "how fast can you do matrix multiplication?", library support becomes a key differentiator. And that's where AMD is behind.
Their management has been fast asleep for at least 2 years
On the other hand, finding and installing those libraries is nothing compared to actually developing GPU computing software, so, as I said, if you want to program GPUs, even such scattered state of AMD platform is not that worse than Nvidia. Because, in Nvidia, you install CUDA and you get everything set up. And then - what do you do? You still have to learn a not-so-easy black art of optimizing kernels for the actual hardware.
For NVidia I have seen some faster kernels than the ones supplied by NVidia - https://github.com/NervanaSystems/neon - though CuDNN introduced Winograd kernels too in their last update
And in the case of the dollars spent on R&D, they seem to be very happy doing stuff in the car industry, and long may that continue—good luck to them. We're spending our dollars the areas we're focused on[1]
To be clear, stuff in the car industry is "just" stuff like self driving cars[2][3].
[1] http://arstechnica.co.uk/gadgets/2016/04/amd-focusing-on-vr-...
[2] https://blogs.nvidia.com/blog/2016/05/06/self-driving-cars-3...
Yes, however, this process has the nice side effect of spurring a huge interest in fundamental AI research, which will lead to alternative algorithms that ultimately perform better.
None of these developments really explain why AI has become so popular though. Neural nets don't actually perform very well for most problems (image and speech problems being the major exceptions) compared to tree-based methods. I think it has more to do with the development of easy-to-use machine learning libraries as well as distributed processing libraries and hardware.
Also, containerization.
They're essentially an informed prior on image and sound problems, which vastly reduces the size of the parameter space and thus makes both vanishing gradient and overfitting less of a problem.
The factor is more like 5-10x for truly optimized CPU versus GPU in fp32 (e.g., nnpack on CPU versus cuDNN).
At the roofline of performance, the difference in both mem b/w and arithmetic throughput between CPU and GPU is only 5-10x (for fp32, Pascal fp16 is a different story of course), and proper implementations on the CPU will get you there.
Source?
Is there any reading material you can recommend on the recent history of ML/AI for someone like me, with a dilettante's understanding of these problems and an interest in optimization?
(Disclaimer: I don't know that much about ML/Deep Learning, the above was recommended by deeplearning.net and is what I'm slogging through ATM. I do have some background in "classical" statistics & optimization theory, and the above article is not that math-heavy)
1) Use convolutional layers, ReLUs, and a few other tricks to ameliorate the vanishing gradient problem,
2) Perform continuous, high-dimensional stochastic gradient descent on graphics cards, and
3) Apply these things via stochastic grad-student descent to sufficiently massive datasets that even the most brute-force models and training methods (backpropagation of errors on a loss functional) can actually work.
In the meantime, the hardware for doing it has become commoditized and the software has consolidated and become standardized. So now it's A Thing in industry.
There are lots of "smarter" algorithms that almost definitely come closer to human cognition, for instance probabilistic program induction. But those aren't fast, and don't always neatly separate training from prediction: you're just not gonna be able to train those models ahead-of-time on a 10,000 image corpus inside a single week with today's hardware and software.
We need to find ways to make machine learning fast even when it's not just a bunch of matrix multiplies. Otherwise, every time we make our algorithms more interesting, we cripple ourselves computationally.