News for hackers. Stuff that matters.
Re:The GPU is not always faster by throwway120385 on Wednesday December 11, 2024 @01:11PM (#42390807)
So everyone is supposed to do all of their testing on H100's?
Re:The GPU is not always faster by alecco on Wednesday December 11, 2024 @02:22PM (#42391676)
4090 (2022) PCIe 4.0 x16 is quite decent. The major limit is memory, not bandwidth. And 3090 (2020) is also PCIe 4.0 x16, and used cards are a bargain. You can hook them up with Nvlink.

Nvidia is withholding new releases but the current hardware has more legs with new matrix implementations. Like FlashAttention doing some significant improvement every 6 months.

Nvidia could make consumer chips with combined CPU-GPU. I guess they are too busy making money with the big cloud. Maybe somebody will pick up. Apple is already doing something like it even on laptops.

Re:The GPU is not always faster by _zoltan_ on Wednesday December 11, 2024 @02:26PM (#42391714)
get a GH100 on lambda and behold you have 900GB/s between CPU memory and GPU, and forget PCIe.
Re:The GPU is not always faster by saagarjha on Thursday December 12, 2024 @02:53AM (#42397169)
Where are you seeing 900 GB/s?
Re:The GPU is not always faster by KeplerBoy on Thursday December 12, 2024 @05:10AM (#42397885)
The 900 GB/s are a figure often cited for Hopper based SXM boards, it's the aggregate of 18 NVLink connections. So it's more of a many-to-many GPU-to-GPU bandwidth figure.

https://www.datacenterknowledge.com/data-center-hardware/nvi...

Re:The GPU is not always faster by shaklee3 on Friday December 13, 2024 @12:38AM (#42406187)
this is bidirectional bw and it's gh200, not gh100
Re:The GPU is not always faster by imtringued on Thursday December 12, 2024 @04:39AM (#42397713)
That doesn't add up, because you're now exceeding the memory bandwidth of the memory controller. I.e. it would be faster to do everything, including the CPU only algorithms, in far away VRAM.
Re:The GPU is not always faster by gpderetta on Thursday December 12, 2024 @08:20AM (#42398892)
latency might be higher and latency usually dominates CPU side algorithms.