Re:Google Cloud TPU Multislice Training
by aschleck on Friday November 10, 2023 @03:59PM
(#38224446)
1979 16 bit flops on an H100 is with sparsity. See footnote 2 on https://www.nvidia.com/en-us/data-center/h100/. You should be halving it for non-sparse flops.
Re:Google Cloud TPU Multislice Training
by YetAnotherNick on Saturday November 11, 2023 @03:11AM
(#38228450)
GP is correct. With sparsity it is 3958. 1979 Tflop/s is without sparsity.
989 is TF32 core, for 16 bit it is 1979, so I guess around 5000 H100’s in a single training job would be equivalent to the training job mentioned in this article.
Either way I actually would not be surprised if OpenAI has launched a single job on more than 10k GPU’s, but I also am not very knowledgeable on practical scaling. Congrats on the feat!