News for hackers. Stuff that matters.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by renonce on Wednesday May 31, 2023 @02:40AM (#36135227)
I suppose it would be 12 bytes? 4 bytes for base model, 4 bytes for optimizer momentum and 4 bytes for optimizer second moment EWA.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by Taek on Wednesday May 31, 2023 @08:27AM (#36137516)
I don't know what the breakdown is, but I know there was code for training the llama models on a DGX (640 GB of VRAM, repo is now gone), and it could only train the 7b model without using deepspeed 3 (offloading).

The ML engineers in my group chat say "1 DGX to train 7b at full speed, 2 for 13b, 4 for 30b, 8 for 65b"