There's also the huge caveat of CUDA project compatibility. AMD/Intel seem to be a more promising in that regard, and thats not saying much.

Not that it isn't appealing... If the price is right.

Also, both AMD and Intel are both reportedly coming out with M Pro-like SoCs. AMD's "Strix Halo" is rumored for 2025. Less is known about Intel's Arrow Lake SKU at the moment.

How many people are writing their own CUDA kernels? I think Pytorch compatibility is a bigger win for AMD amd Intel, when solving for the vast majority of AI practitioners who live in Jupyter Notebooks

I don't see the distinction. Tons of popular PyTorch projects run hand made CUDA kernels, whether through a library or one written themselves.

In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch, but the performance hit is often severe, and not many inference projects are using torch.compile or triton kernels to compensate.

> I don't see the distinction

Performant Pytorch compatibility can be achieved without CUDA compatibility.

> In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch

This is the crux of my point: AMD ought to focus on fixing these Pytorch incompatibilities instead of splitting focus with chasing CUDA translations.

Greater than zero. Julia has a CUDA compiling and I've hand crafted CUDA kernels myself.

https://cuda.juliagpu.org/stable/

Indeed, and this year we created a system for compiling ODE code not just optimized CUDA kernels but also OneAPI kernels, AMD GPU kernels, and Metal. Peer reviewed version is here (https://www.sciencedirect.com/science/article/abs/pii/S00457...), open access is here (https://arxiv.org/abs/2304.06835), and the open source code is at https://github.com/SciML/DiffEqGPU.jl. The key that the paper describes is that in this case kernel generation is about 20x-100x faster than PyTorch and Jax (see the Jax compilation in multiple ways in this notebook https://colab.research.google.com/drive/1d7G-O5JX31lHbg7jTzz..., extra overhead though from calling Julia from Python but still shows a 10x).

The point really is that while deep learning libraries are amazing, at the end of the day they are DSL and really pull towards one specific way of computing and parallelization. It turns out that way of parallelizing is good for deep learning, but not for all things you may want to accelerate. Sometimes (i.e. cases that aren't dominated by large linear algebra) building problem-specific kernels is a major win, and it's over-extrapolating to see ML frameworks do well with GPUs and think that's the only thing that's required. There are many ways to parallelize a code, ML libraries hardcode a very specific way, and it's good for what they are used for but not every problem that can arise.

5 years?! You think the enterprise hardware hype will last that long? When it slows companies will look at other less valuable markets.

Unless something fundamentally changes with the capabilities of produced tech I think demand will be saturated before 5 years.

I think, in general, hardware markets are permanently splitting and the high end side will not stop scaling for at least a decade. From storage to network to compute the needs of the large scale customers grows ever farther from the needs of the consumer. The hyperscaler class are rushing in 400G NICs with hundreds of CPU cores, TBs of RAM, and hundreds of TBs of high speed flash per rack unit asking when they'll be able to make that 800G or 1.6 T and so on. Meanwhile consumers are just warming up to the idea wired and wireless connections can be slightly faster than a gig and their wild gaming desktop can have 16 cores with 64 GB of RAM and 4 TB of flash storage. GPUs aren't really any different - AMD has even had a completely separate computer architecture for years and Nvidia has entire custom heterogenous scale out designs. If you had asked me "do you think it's possible x86 will scale to over 500 cores per rack unit by 2023" in 2013 I would have said "doubt it, Xeon Phi doesn't even look to be on track for people to want to half ass getting to that number". If you were to ask me now how much we'll have scaled in 10 years this time I'd rather now say "not sure, but we probably aren't going to have stopped cold at the 5 year mark because the hype for more scaling died out".

As a result of this split I think consumer hardware will still continue to grow slower. Not because of lack of capability but lack of market interest. People don't have a use case for hundreds of cores or the like, they'd rather smaller, quieter, and more efficient at ever decreasing price points. I'm much more worried investment in consumer GPUs and CPUs for traditional desktops will actually die out and start to stagnate just because the market is getting squeezed ever smaller from both the low end and high end sides.

Interesting. I guess I agree on the idea just not the specific number of 48GB. Amd & Nvidia both offer prosumer GPUs with 48GB, the W7900 and A6000 respectively. I don't see how they would not increase that in 5 years?