The first transformer models were developed at Google. NVIDIA were the card du jour for accelerating it in the years since and have contributed research too, but your statement goes way too far
Deep learning was only possible because you could do it on NVidia cards with Cuda without having to use the big machines in the basement.
Trying to convince anyone that neural networks could be useful in 2009 was impossible - I got a grant declined and my PhD supervisor told me to drop the useless tech and focus on something better like support vector machines.
The difference is AMD killed theirs to use OpenCL and NVIDIA kept CUDA around as well.
I tried using AMD Stream and it lacked documentation, debugging information and most of the tools needed to get anything done without a large team of experts. NVidia by comparison could - and was - used by single grad students on their franken-stations which we build out of gaming GPUs.
The less we talk about the disaster that the move to opencl was the better.
That said ROCm is quite a recent thing borne out of acknowledging that OpenCL 2 was a disaster.
OpenCL 1 had a reasonable shot and was gaining adoption but 2 scuppered it.
I did write quite a bit of OpenCL prior to that on Intel/AMd/NVIDIA, both for training and for general rendering though, and did some work with Stream before then.
Cuda by comparison JustWorks^tm.
1 was definitely a lot easier to work with than 2. CUDA is easier than both but I don’t think I hit anything I could do in CUDA that I couldn’t do in OpenCL, though CUDA of course had a larger ecosystem of existing libraries.
Thanks for the trip down memory lane.
The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later.
I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.
NVIDIA absolutely contributed to the foundation but they are not the foundation alone.
Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.
To say it wouldn’t have been possible on AMD is ludicrous and there is a pattern to your comments where you dismiss any other companies efforts or capabilities, but are quite happy to lay all the laurels on NVIDIA.
The reality is that multiple companies and individuals got us to where we are, and multiple products could have done the same. That's not to take away from NVIDIA's success, it's well earned, but if you took them out of the equation, there's nothing that would have prevented the tech existing.
> It was just what happened to be available and most straightforward at the time
AMD made better hardware for a while and people wanted OpenCL to succeed. The reason why nvidia became dominant was because their competitors simply weren’t good enough for general purpose parallel compute.
Would AI still have happened without CUDA? Almost certainly. However nvidia still had a massive role in shaping what it looks like today.
It’s one of the projects I think the Khronos group mishandled the most unfortunately.
If I were to categorize the successful ones:
glTF, KTX, SPIR-V, OpenGL and its variants, WebGL
People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success.
OpenCL was and is a failure of grand magnitude. As was colada.
I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space.
Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.