News for hackers. Stuff that matters.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by airgapstopgap on Wednesday May 31, 2023 @12:59AM (#36134632)
I think people talking about a 100T GPT didn't mean a dense transformer but some sort of extreme Mixture-of-Experts which is much more amenable to low-resource setups and complicates this discussion.

In any case, it's almost certainly not bigger than 1T, even if it's not a dense transformer (PaLM-2 is and makes do with 340B, but it isn't exactly on par).

There are no comments on this comment yet.