News for hackers. Stuff that matters.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by andai on Wednesday May 31, 2023 @12:40AM (#36134536)
For the numerically challenged like me: 100TB is 100 trillion bytes, giving you 1 byte per parameter at 100T params.

LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though), which would allow you to run a 200TB model on one of these cards if I'm understanding right.

Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by valine on Wednesday May 31, 2023 @12:51AM (#36134589)
That would work for inference, but for efficient training you’d also want you training set to fit in memory.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by airgapstopgap on Wednesday May 31, 2023 @12:59AM (#36134632)
I think people talking about a 100T GPT didn't mean a dense transformer but some sort of extreme Mixture-of-Experts which is much more amenable to low-resource setups and complicates this discussion.

In any case, it's almost certainly not bigger than 1T, even if it's not a dense transformer (PaLM-2 is and makes do with 340B, but it isn't exactly on par).

Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by renonce on Wednesday May 31, 2023 @02:43AM (#36135237)
You can’t quantize it for training due to numerical instability. For inference you don’t usually use such a big cluster.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by Tepix on Wednesday May 31, 2023 @03:42AM (#36135608)
> LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though)

From the GPTQ paper https://arxiv.org/abs/2210.17323:

"... with negligible accuracy degradation relative to the uncompressed baseline"