Nvidia Hopper Architecture In-Depth

Re:Nvidia Hopper Architecture In-Depth by wfrew on Tuesday August 23, 2022 @12:31AM (#32560490)

Adding network addressing to the GPU interconnect is kind of fascinating

Am I right in thinking the GPU-to-GPU communication is just shuttling chunks of data around for sharing inputs/outputs of computations? Or is there some other coordination going on between the GPUs directly with regards to the actual computations each is running? (Or is that still being managed wholly by the CPUs they're attached to?)

[ Parent | on HN ]

Re:Nvidia Hopper Architecture In-Depth by learndeeply on Tuesday August 23, 2022 @01:35AM (#32560802)

Both - the first is for sharding tensors across GPUs, the second is to do an all reduce (e.g. for distributed data parallel to synchronize gradients)

[ Parent | on HN ]

Re:Nvidia Hopper Architecture In-Depth by chessgecko on Tuesday August 23, 2022 @12:52AM (#32560600)

Maybe it’s to speed up multi gpu matrix multiplies. They’re useful for serving/training gpt3 size models

[ Parent | on HN ]