> Can you link to benchmarks or other analysis supporting this claim?

Current generation nVidia: https://en.wikipedia.org/wiki/GeForce_40_series#Desktop

Current generation AMD: https://en.wikipedia.org/wiki/Template:AMD_Radeon_RX_7000

The key performance characteristics are processing power TFlops, and memory bandwidth GB/s.

nVidia 4080 which costs $1200: 43 TFlops FP16, 43 TFlops FP32, 0.672 TFlops FP64, 717 GB/s memory.

AMD 7900 XTX which costs $1000: 93 TFlops FP16, 47 TFlops FP32, 1.5 TFlops FP64, 960 GB/s memory.

Note that for applications which bottleneck on FP16 compute (many ML workloads) or FP64 compute (many traditional HPC workloads: numerical solvers, fluid dynamics, etc), the 7900 XTX even outperforms the 4090 which costs $1600.

In ML workloads, usually the FP16 operations are matrix operations. On RDNA3, these execute at the same rate as normal shader/vector operations, but on Nvidia RTX cards there are Tensor cores which accelerate them. The Ada whitepaper lists 48.7 shader TFlops (not 43 because boost vs base clock), and 195 TFlops for FP16 Tensor with FP16 Accumulate. That's 4 times faster than regular, and almost double what the XTX lists!

Ampere and newer also have native sparsity support which means that you can skip over the zeroes 'for free', which Nvidia uses to market double the TFlops, which is kind of misleading imo. But the 195 TFlops are even before sparsity is included!

I'm not sure if the 93 TFlops (120 with boost clocks) on AMD are with FP16 or FP32 accumulation, as with FP32 accumulation the 4080 slows down significantly and gets much closer with 97.5 TFlops.

Intel Xe-HPG (used in the Arc A cards) also offers very aggressive matrix acceleration via XMX, with 137.6 FP16 TFlops at base clock, vs. 17.2 FP32 TFlops.

You’re comparing general-purpose computations to a proprietary feature with limited applicability. For example, in my Mistral implementation the most expensive compute shader is handling matrices compressed with a custom codec unknown to tensor cores.

This is not an apples-to-apples comparison. We use FLOPS numbers to compare processors of different architectures because they’re the same FLOPS everywhere.

>This is not an apples-to-apples comparison.

Neither is FLOPS. RDNA3 FLOPS numbers are greatly inflated, because they are only applicable to fairly specific VOPD and WMMA instructions, the former being for two MADs at the same time, and the latter being applicable in the same cases as tensor cores.

Besides, it should be possible to use the tensor cores combined with the codec: you can use vector hardware to decode matrix values to fp16/fp32, then yeet those into the tensor cores. Although most of the time will probably be spent on the decoding part, assuming you're doing matrix-vector multiplication and not matrix-matrix (which might be different with MoE models like Mixtral 8x7B?)

RDNA3 FLOPS numbers are real. Shader compiler happily uses VOPD for quite a few things including FP32 FMA.

An independent researcher measured 62.9 FP32 TFlops (boost) in Vulkan on 7900XTX, see “Full GPU Throughput – Vulkan” section on that article: https://chipsandcheese.com/2023/01/07/microbenchmarking-amds...

That is interesting, even though I would need to look at non-consumer-GFX cards. What about actual benchmarks, though? Passmark, for example:

https://www.videocardbenchmark.net/directCompute.html

has:

GeForce RTX 4090 28,916

RTX 6000 Ada 23,296

GeForce RTX 4080 22,016

Radeon RX 7900 XTX 18,767

and Geekbench 6:

https://browser.geekbench.com/opencl-benchmarks

has:

GeForce RTX 4090 321928

L40 292357 <- L40S is out and may be better

RTX 6000 Ada 274348

H100 267514

NVIDIA GeForce RTX 239863

Radeon RX 7900 XTX 199412

Neither of those benchmarks is relevant to ML which is what GP is talking about

So what is relevant? As other poster said, nVidia tends to outpace AMD cards in pretty much every use case in real life.

The only important numbers are processing power TFlops, and memory bandwidth GB/s.

If your compute shader doesn’t approach the theoretical limit of either computations or memory, it doesn’t mean there’s anything wrong with the GPU. Here’s incomplete list of possible reasons.

● Insufficient parallelism of the problem. Some problems are inherently sequential.

● Poor HLSL programming skills. For example, a compute shaders with 32 threads/group wastes 50% of compute units of most AMD GPUs, the correct number for AMD is 64 threads/group, or a multiple of 64. BTW, nVidia and Intel are fine with 64 threads/group, they run 1 thread group as 2 wavefronts which does not waste any resources.

● The problem being too small to compensate for the overhead. For example, CPUs multiply two 4x4 matrices in a small fraction of a time it takes to dispatch a compute shader for that. You gonna need much larger matrices for GPGPU to win.

OpenCL is stuck in C99, that is why most people ignored it and adopted CUDA in first place.

The attempts to bring C++ to it never were properly done, and that is why OpenCL 3.0 is basically OpenCL 1.0 rebranded.

OpenCL has had a C++ variant for a while. But it is like you said: The consortium members, especially NVIDIA, did not do thing properly, either half-heartedly or in an intentionally broken manner.

Always blamming NVidia, when Intel and AMD never did anything to properly support it.

>Always blamming NVidia

As LLM, AI takes over the heated topic of discussions on HN, it is hard not to notice the tone on blaming Nvidia for everything. Both Hardware and Software.

I think we should be blaming the OS platform actors and user communities for letting the lunatics run the asylum (hw vendors dictating sw). And then MS & Apple both went ant did their own proprietary stuff.

OpenCL 2.0 was ignored because implementing shared virtual memory was mandatory, but it simply isn't possible to implement without cache coherence all the way into the GPU memory. Now that Intel has developed their own GPUs, they have gone out of their way to completely ignore SVM and went with their own proprietary USM extension of OpenCL and the Intel driver developers themselves said SVM is difficult to get right. Intel and AMD were the only ones implementing OpenCL 2.0 and the SVM implementation by AMD was basically unusable.

OpenCL 3.0 got rid of the mandatory features of OpenCL 2.0 that nobody was willing to implement.

Basically a wall of text going around that Intel and AMD never delivered anything in OpenCL that would make developers actually care.

They only needed to make something half as usable as CUDA, in polyglot support and graphical development tooling for GPGPU.

I was once in a Khronos panel webminar in HPC, where no one in the panel understood the relevance of Fortran support in OpenCL, then no wonder Fortran workloads get deployed with CUDA.