News for hackers. Stuff that matters.
Re:Nvidia DGX GH200: The First 100 Terabyte GPU Memory System by mirekrusin on Wednesday May 31, 2023 @04:20PM (#36143748)
Yes, if you use ADAM - but it doesn't add up to 80, does it?

Even for fp64 it adds only 16 bytes.

RMSPRop, Adagrad have half of this overhead.

SGD has no optimizer overhead of course.

There are no comments on this comment yet.