Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by mirekrusin on Wednesday May 31, 2023 @04:11PM
(#36143602)
What intermediate results you need to store?
For backpropagation you take the diff between actual and expected output and you go backwards to calculate derivate and apply it with optimiser - that's 8 extra bytes for single precision floats per trainable parameter.
For backpropagation you take the diff between actual and expected output and you go backwards to calculate derivate and apply it with optimiser - that's 8 extra bytes for single precision floats per trainable parameter.
Why do you need 80?