News for hackers. Stuff that matters.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by belval on Friday February 23, 2024 @09:18AM (#39480846)
> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast.

I am a big fan of Jim Keller, but this is semantics that argues CUDA the language is not a moat, when most people refer to CUDA the libraries.

CUDA has first party support in most of the libraries in use today, if you were to stray from that happy path you must have deep pocket to work through the hurdles of working with something like XLA.

There's a reason why Microsoft, Google, Amazon and Meta are still buying Nvidia accelerators even when they have their own in-house accelerators.

Re:Jim Keller criticizes Nvidia's CUDA, x86 by westhanover on Friday February 23, 2024 @09:43AM (#39481124)
Well he is obviously just talking his book. He is selling a Nvidia competitor. If CUDA isn’t beautiful and the future deserves to be beautiful, why don’t you make it and show us, Jim? All we are getting is more hot air about CUDA. That is all AMD and Intel have as well, bold statements about the future and no code.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by neural_thing on Friday February 23, 2024 @10:08AM (#39481432)
He isn't though. He hasn't been at AMD or Intel for quite some time.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by westhanover on Friday February 23, 2024 @10:25AM (#39481688)
He sells AI chips that don’t run CUDA.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by fisf on Friday February 23, 2024 @11:14AM (#39482464)
Jim is at tenstorrent now.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by HarHarVeryFunny on Friday February 23, 2024 @10:22AM (#39481654)
CUDA/etc is certainly a pain.

A good number of years ago I wrote my own Torch-like C++ NN framework using CUDA (cuDNN, cuBLAS) for the GPU, and one of the annoyances is that cuDNN has incomplete coverage of even the basic operators needed. Add/Sub/Min/Max/Sqrt/Negate are all provided as part of cuDNN, but if you want other common NN building blocks like Div/Exp/Log/Pow/Inv/InvSqrt then you have to write them yourself in CUDA and either forgo cuDNNs tensor-descriptor layout flexibility or re-implement that yourself too.

Of course frameworks like PyTorch support all the operators you'd expect, since they've written their own kernels in CUDA where the functionality is missing from cuDNN.

It's not clear exactly what Keller is referring to there. When people say CUDA they might be referring to the entire ecosystem (nvcc CUDA C/C++ compiler, CUDA API's, higher level cuBLAS, cuDNN, etc), or maybe just the base compiler (which lets you write your own kernels) and API for allocating memory, queueing kernels, etc.

The cuDNN kernels (convolution, etc) are highly optimized, as is cuBLAS (e.g. matmul), and I doubt anyone is going to do better writing these themselves. Does Keller consider using cuDNN as "writing CUDA" ?

As far as I'm aware the higher level, performant, CUDA libraries, as well as specialized components like TensorRT are written in CUDA, although that could mean a combination of C/C++ & PTX pseudo-assembler (ptxas is really a compiler, not an assembler). The alternative would be they they were written in hand optimized SASS assembler which afaik is only available outside of NVDIA via the Open Source CuAssembler.

I believe Mojo support for NVIDA is based on PTX. I'm not sure if that would be really be considered as "CUDA" or not if they are not using nvcc at all.

Most people, outside of framework vendors, would have no reason to use CUDA anyway, since it's just too low level. The only sane use case would be where writing a custom kernel in (e.g.) PyTorch or Mojo doesn't get the performance you want and you write than one kernel in CUDA. The hope would be that the Mojo compiler is good enough that this would not be necessary.

Re:Jim Keller criticizes Nvidia's CUDA, x86 by dist-epoch on Friday February 23, 2024 @09:32AM (#39480997)
The impossibility of buying NVIDIA GPUs in quantity today is a giant incentive to port the most important libraries to AMD/....
Re:Jim Keller criticizes Nvidia's CUDA, x86 by cjbgkagh on Friday February 23, 2024 @11:58AM (#39483092)
I think you’re assuming these big companies are serious about AI itself and not just joining in on a trend. They would much rather spend huge amounts of money on chips than hire a competent researcher who might tell them that they didn’t really need to waste so much money.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by photonbeam on Friday February 23, 2024 @11:03AM (#39482282)
Im surprised we dont hear more about this, it seems like a huge motivator
Re:Jim Keller criticizes Nvidia's CUDA, x86 by hackerlight on Friday February 23, 2024 @03:31PM (#39485728)
Andrew Ng says ROCm is a lot better today than early 2023
Re:Jim Keller criticizes Nvidia's CUDA, x86 by pjmlp on Friday February 23, 2024 @04:06PM (#39486091)
The computing shops on my city shopping malls have enough laptops with NVIDIA GPUs, while they aren't at the same level as desktop or server GPUs, there are available and usable for people to learn CUDA, quite the contrary with most AMD offerings.
Re:Jim Keller criticizes Nvidia's CUDA, x86 by dist-epoch on Friday February 23, 2024 @04:59PM (#39486656)
Not sure what you mean, both NVIDIA and AMD desktop GPUs are available, there is no particular shortage. Prices can be high for top ones, like NVIDIA 4090, but you can buy it.

Server GPUs are the ones difficult to buy.

People don't really learn CUDA, they use it mostly through another library like PyTorch.

Re:Jim Keller criticizes Nvidia's CUDA, x86 by pjmlp on Saturday February 24, 2024 @01:30AM (#39489610)
Most AMD stuff doesn't work on consumer GPUs.

Try to make ROCm work on a random AMD card.

Re:Jim Keller criticizes Nvidia's CUDA, x86 by m463 on Friday February 23, 2024 @04:26PM (#39486329)
> "Basically nobody writes CUDA,"

Can't help but think of that old yogi berra quote:

"Nobody goes there anymore, it's too crowded"

(but yeah, I suspect a lot of people use cuda via accelerated libraries/apps like opencv)