Re:Zero-3 Offload: Scale DL models to trillion parameters wi...
by liuliu on Saturday March 13, 2021 @03:51PM
(#26449877)
They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.
[ Parent
| on HN ]
Re:Zero-3 Offload: Scale DL models to trillion parameters wi...
by p1esk on Saturday March 13, 2021 @05:56PM
(#26450942)
They did all that before: https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.