Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by valine on Wednesday May 31, 2023 @12:51AM
(#36134589)
That would work for inference, but for efficient training you’d also want you training set to fit in memory.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by airgapstopgap on Wednesday May 31, 2023 @12:59AM
(#36134632)
I think people talking about a 100T GPT didn't mean a dense transformer but some sort of extreme Mixture-of-Experts which is much more amenable to low-resource setups and complicates this discussion.
In any case, it's almost certainly not bigger than 1T, even if it's not a dense transformer (PaLM-2 is and makes do with 340B, but it isn't exactly on par).
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by renonce on Wednesday May 31, 2023 @02:43AM
(#36135237)
You can’t quantize it for training due to numerical instability. For inference you don’t usually use such a big cluster.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System
by Tepix on Wednesday May 31, 2023 @03:42AM
(#36135608)
> LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though)
From the GPTQ paper https://arxiv.org/abs/2210.17323:
"... with negligible accuracy degradation relative to the uncompressed baseline"
LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though), which would allow you to run a 200TB model on one of these cards if I'm understanding right.