News for hackers. Stuff that matters.
Re:The GPU is not always faster by alecco on Wednesday December 11, 2024 @12:37PM (#42390369)
> Each version is severely memory bandwidth bottlenecked, the CUDA version suffers the most with its practical 11.8 GB/s device-to-host bandwidth due to its PCI-Express 3.0 x16 interface.

PCIe 3.0? What?

https://cowfreedom.de/#appendix/computer_specs/

> GeForce GTX 1050 Ti with Max-Q Design (PCIe 3.0 x16) (2016)

> Intel Core i5-8300H (2020)

This is a low-price 8 year old GPU and a 4 year old CPU. And he seems to be including loading the data to GPU. Newer cards have wide PCIe 5.0 or some faster interconnect, like Nvidia Grace-Hopper.

Also he is comparing his own CUDA implementation. He should use one of the many available in CUBLAS/CUTLASS. Making a good CUDA GEMM is a very difficult art and very hardware specific

Re:The GPU is not always faster by jsheard on Wednesday December 11, 2024 @01:03PM (#42390724)
> Newer cards have wide PCIe 5.0 or some faster interconnect, like Nvidia Grace-Hopper.

There aren't any (edit: consumer) GPUs with PCIe5 yet, though they probably aren't far off. Plenty already have PCIe4 though.

Re:The GPU is not always faster by alecco on Wednesday December 11, 2024 @01:09PM (#42390779)
Consumer cards are PCIe 4.0 x16. H100 PCIe version is PCIe 5.0 https://en.wikipedia.org/wiki/Hopper_(microarchitecture). And it's been out 2 years already.
Re:The GPU is not always faster by yvdriess on Wednesday December 11, 2024 @03:29PM (#42392577)
For the consumer GPUs, PCIe 4.0 x16 has plenty of BW headroom. The full sized x16 is more for stability reasons. Some vendors even put a couple of M.2 slots on PCI4/5 GPU board to recuperate the unused PCIe lanes.
Re:The GPU is not always faster by jsheard on Wednesday December 11, 2024 @03:42PM (#42392728)
x16 also matters for power delivery, motherboards are required to provide 75W through a physical x16 slot (even if not all of the lanes are connected) but the smaller slots only have to provide up to 25W.
Re:The GPU is not always faster by KeplerBoy on Thursday December 12, 2024 @04:57AM (#42397820)
That's mixing apples with oranges.

Some low-end or middle range GPUs really only use 8 lanes, because that's how the chip is designed. Fewer lanes -> less silicon area for the pcie logic needed -> cheaper.

The chips which use 16 lanes take advantage of it and can saturate the link.

Re:The GPU is not always faster by wtallis on Thursday December 12, 2024 @10:43AM (#42400090)
Smaller GPU silicon is also often designed more around the laptop market, where it's common for the CPU to not have more than 8 lanes available for the GPU.
Re:The GPU is not always faster by shaklee3 on Friday December 13, 2024 @12:37AM (#42406179)
what does a laptop GPU have to do with a server grade xeon that the article is comparing against?
Re:The GPU is not always faster by jsheard on Wednesday December 11, 2024 @01:11PM (#42390809)
TIL, I missed Hopper already having it. I assume the RTX 5000 series will bring it to consumers.
Re:The GPU is not always faster by throwway120385 on Wednesday December 11, 2024 @01:11PM (#42390807)
So everyone is supposed to do all of their testing on H100's?
Re:The GPU is not always faster by alecco on Wednesday December 11, 2024 @02:22PM (#42391676)
4090 (2022) PCIe 4.0 x16 is quite decent. The major limit is memory, not bandwidth. And 3090 (2020) is also PCIe 4.0 x16, and used cards are a bargain. You can hook them up with Nvlink.

Nvidia is withholding new releases but the current hardware has more legs with new matrix implementations. Like FlashAttention doing some significant improvement every 6 months.

Nvidia could make consumer chips with combined CPU-GPU. I guess they are too busy making money with the big cloud. Maybe somebody will pick up. Apple is already doing something like it even on laptops.

Re:The GPU is not always faster by _zoltan_ on Wednesday December 11, 2024 @02:26PM (#42391714)
get a GH100 on lambda and behold you have 900GB/s between CPU memory and GPU, and forget PCIe.
Re:The GPU is not always faster by saagarjha on Thursday December 12, 2024 @02:53AM (#42397169)
Where are you seeing 900 GB/s?
Re:The GPU is not always faster by KeplerBoy on Thursday December 12, 2024 @05:10AM (#42397885)
The 900 GB/s are a figure often cited for Hopper based SXM boards, it's the aggregate of 18 NVLink connections. So it's more of a many-to-many GPU-to-GPU bandwidth figure.

https://www.datacenterknowledge.com/data-center-hardware/nvi...

Re:The GPU is not always faster by shaklee3 on Friday December 13, 2024 @12:38AM (#42406187)
this is bidirectional bw and it's gh200, not gh100
Re:The GPU is not always faster by imtringued on Thursday December 12, 2024 @04:39AM (#42397713)
That doesn't add up, because you're now exceeding the memory bandwidth of the memory controller. I.e. it would be faster to do everything, including the CPU only algorithms, in far away VRAM.
Re:The GPU is not always faster by gpderetta on Thursday December 12, 2024 @08:20AM (#42398892)
latency might be higher and latency usually dominates CPU side algorithms.
Re:The GPU is not always faster by touisteur on Wednesday December 11, 2024 @05:20PM (#42393675)
Well let me tell you about the pretty high-end and expensive L40 that shipped with PCIe-4.0 to my utter dismay and disgust. Only the H100 had 5.0 although I could already saturate 4.0 (and 5.0) with Mellanox NICs and GPU/StorageDirect. Waiting for the next one to maybe get 5.0.
Re:The GPU is not always faster by goosedragons on Wednesday December 11, 2024 @01:52PM (#42391321)
Supposedly the Intel B580 releasing friday will use PCIe 5.0 8x.
Re:The GPU is not always faster by yvdriess on Wednesday December 11, 2024 @03:32PM (#42392623)
Re:The GPU is not always faster by phonon on Wednesday December 11, 2024 @09:46PM (#42395682)
That's A580. Correct link https://www.intel.com/content/www/us/en/products/sku/241598/...

"PCI Express 4.0 x8"

Re:The GPU is not always faster by alecco on Thursday December 12, 2024 @07:40AM (#42398662)
So Intel is doing the same artificial performance limitation. Nvidia 4060s are also PCIe 4.0 with half lanes (x8). Argh.
Re:The GPU is not always faster by phonon on Thursday December 12, 2024 @08:50AM (#42399105)
Alternatively, at that performance level it doesn't make a difference, so better to save the $5....
Re:The GPU is not always faster by _zoltan_ on Wednesday December 11, 2024 @02:25PM (#42391700)
that's not true, H100 NVL is PCIe gen5 x16.
Re:The GPU is not always faster by _zoltan_ on Wednesday December 11, 2024 @02:27PM (#42391723)
GH100 can do 900GB/s HtoD.
Re:The GPU is not always faster by alecco on Wednesday December 11, 2024 @02:39PM (#42391891)
And both 3090 and 4090 can do 32 GB/s host-device. Not far from CPU-RAM. You only load the matrix once. The bandwidth for the matmul is orders of magnitude larger and happens all in device and mostly in cache.
Re:The GPU is not always faster by jing on Wednesday December 11, 2024 @08:08PM (#42395050)
No it can’t. That’s d to d
Re:The GPU is not always faster by saagarjha on Thursday December 12, 2024 @02:54AM (#42397176)
Well, device to device is technically doubled because you have a read and a write. But yes
Re:The GPU is not always faster by shaklee3 on Friday December 13, 2024 @12:38AM (#42406191)
No. D2D is arroubd 3TB/s
Re:The GPU is not always faster by _zoltan_ on Friday December 13, 2024 @07:24AM (#42407901)
no.

[root@gh200 nvbandwidth]# nvidia-smi | grep GH200

| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |

[root@gh200 nvbandwidth]# ./nvbandwidth | grep -E 'host_to_device_memcpy_sm|device_to_host_memcpy_sm' | grep ^SUM

SUM host_to_device_memcpy_sm 357.45

SUM device_to_host_memcpy_sm 352.05

[root@gh200 nvbandwidth]#

d2d is much higher:

[root@gh200 cuda-samples]# ./bin/sbsa/linux/release/bandwidthTest --dtod | grep -B1 32000

   Transfer Size (Bytes) Bandwidth(GB/s)
   32000000   2294.8
[root@gh200 cuda-samples]#
Re:The GPU is not always faster by Ballas on Thursday December 12, 2024 @04:23AM (#42397614)
The CPU was launched in Q2 2018, so that is also 6 years. I wonder what the outcome will be with a CPU that supports AVX-512 and a more recent GPU.