Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Nvidia DGX GH200: The First 100 Terabyte GPU Memory System. Parent comment
~6DP * precision
Where D is number of tokens*mini batch size and P is number of parameters.
So if you want to fit fully into memory with a mini batch of 1, context window 32k, and 16 bit precision, that’s 144e12/6/32e3/2 = 375M param.
If you apply one token at a time then
144e12/6/2 = 12 T param
Ofc, in reality you have model parallelism as well…
There are no comments on this comment yet.
~6DP * precision
Where D is number of tokens*mini batch size and P is number of parameters.
So if you want to fit fully into memory with a mini batch of 1, context window 32k, and 16 bit precision, that’s 144e12/6/32e3/2 = 375M param.
If you apply one token at a time then
144e12/6/2 = 12 T param
Ofc, in reality you have model parallelism as well…