News for hackers. Stuff that matters.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by rfoo on Wednesday May 31, 2023 @05:08AM (#36136158)
It's not per parameter, you also need to hold activations for back prop to work.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by mirekrusin on Wednesday May 31, 2023 @08:33AM (#36137586)
You need activations for inference as well.

But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not 80.