News for hackers. Stuff that matters.

A comment on Nvidia DGX GH200: The First 100 Terabyte GPU Memory System. Parent comment

Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by ioedward on Wednesday May 31, 2023 @05:06AM (#36136141)
Nvidia's enterprise GPUs are surprisingly unreliable. Working on a 128 GPU A100 cluster on AWS, 1 would fail every few days. I didn't have any insight on whether it was a hardware or software failure.
[ Parent | on HN ]
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by csdvrx on Wednesday May 31, 2023 @06:40AM (#36136730)
> Working on a 128 GPU A100 cluster on AWS, 1 would fail every few days

Define "fail".

> I didn't have any insight on whether it was a hardware or software failure.

Have scripts check nvidia-smi for ECC errors and dmesg for devices dropping of the PCI bus.

For the former, replace the card. For the later, just perform a device reset (a power toggle of the device and a rescan of the bus is often enough to be back online within 5 seconds)

[ Parent | on HN ]
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by robotresearcher on Wednesday May 31, 2023 @02:53PM (#36142366)
What does an AWS user do with this advice?
[ Parent | on HN ]
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by csdvrx on Wednesday May 31, 2023 @02:59PM (#36142475)
They either figure out how to write scripts or ask AWS support how to get that done.
[ Parent | on HN ]