Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by KeplerBoy on Wednesday May 31, 2023 @01:42AM
(#36134879)
Because you need a lot more information to perform back-propagation.
[ Parent
| on HN ]
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by mirekrusin on Wednesday May 31, 2023 @01:49AM
(#36134932)
It's not "a lot more" information, it's holding derivative (single number) per parameter, right?
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by gmueckl on Wednesday May 31, 2023 @02:34AM
(#36135198)
Not the GP, but I believe that they are talking about the size of the training data set in relation to the model size.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by mirekrusin on Wednesday May 31, 2023 @03:23AM
(#36135486)
You don't need to and can't really load all training data.
For LLMs you need to load single row of context size, that's vector of ie. 8k numbers, which is 32kB for single precision floats.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by ioedward on Wednesday May 31, 2023 @04:57AM
(#36136079)
You also need the optimizer (e.g. Adam)'s state, which is usually double the parameter's size. So if using fp16, one parameter takes up 6 bytes in memory.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by mirekrusin on Wednesday May 31, 2023 @04:20PM
(#36143748)
Yes, if you use ADAM - but it doesn't add up to 80, does it?
Even for fp64 it adds only 16 bytes.
RMSPRop, Adagrad have half of this overhead.
SGD has no optimizer overhead of course.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by rfoo on Wednesday May 31, 2023 @05:08AM
(#36136158)
It's not per parameter, you also need to hold activations for back prop to work.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by mirekrusin on Wednesday May 31, 2023 @08:33AM
(#36137586)
You need activations for inference as well.
But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not 80.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by calaphos on Wednesday May 31, 2023 @05:17AM
(#36136226)
For automatic differentiation (backpropagation) you need to store the intermediate results per layer of the forward pass. With checkpointing you can only store every nth layer and recompute the rest accordingly to reduce memory requirements in favor of more compute.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by mirekrusin on Wednesday May 31, 2023 @04:11PM
(#36143602)
What intermediate results you need to store?
For backpropagation you take the diff between actual and expected output and you go backwards to calculate derivate and apply it with optimiser - that's 8 extra bytes for single precision floats per trainable parameter.
Why do you need 80?