Re:Microsoft Zero and DeepSpeed: Memory Efficient Large Neur...
by davidwhite on Monday February 10, 2020 @11:21PM
(#22296036)
Looks super cool. Does it remove the need for manual gradient checkpointing?
Also curious if there are expected to be memory / speed improvements if you're using it on a single GPU or if most gains come from improved parallelism across devices.
Also curious if there are expected to be memory / speed improvements if you're using it on a single GPU or if most gains come from improved parallelism across devices.