News for hackers. Stuff that matters.
Re:Google Cloud TPU Multislice Training by sashank_1509 on Friday November 10, 2023 @03:55PM (#38224397)
I think your math is also slightly off, in the Google article, it claims “that is capable of achieving 10 exa-FLOPs (16-bit).” , so you should be comparing with 16 bit operations from a H100.

989 is TF32 core, for 16 bit it is 1979, so I guess around 5000 H100’s in a single training job would be equivalent to the training job mentioned in this article.

Either way I actually would not be surprised if OpenAI has launched a single job on more than 10k GPU’s, but I also am not very knowledgeable on practical scaling. Congrats on the feat!

Re:Google Cloud TPU Multislice Training by aschleck on Friday November 10, 2023 @03:59PM (#38224446)
1979 16 bit flops on an H100 is with sparsity. See footnote 2 on https://www.nvidia.com/en-us/data-center/h100/. You should be halving it for non-sparse flops.
Re:Google Cloud TPU Multislice Training by YetAnotherNick on Saturday November 11, 2023 @03:11AM (#38228450)
GP is correct. With sparsity it is 3958. 1979 Tflop/s is without sparsity.
Re:Google Cloud TPU Multislice Training by emu on Saturday November 11, 2023 @04:45AM (#38228835)
No, it is not. That's the sparse fp8 flop number, but you need to ignore sparsity and compare bf16 flops not fp8 flops for the comparison the ancestor post is making.