Re:Zero-3 Offload: Scale DL models to trillion parameters wi...
by zachthewf on Saturday March 13, 2021 @01:33PM
(#26448731)
It doesn't sound like techno-babble to me. They've distributed storage across nodes rather than replicating on each node, hence the model size is now scalable with number of nodes rather than being limited to what could be stored on a single node.
[ Parent
| on HN ]
Re:Zero-3 Offload: Scale DL models to trillion parameters wi...
by p1esk on Saturday March 13, 2021 @03:04PM
(#26449502)
But it's not clear how they managed to improve training on a single GPU: they say they can fit 40B model on a single V100.
Re:Zero-3 Offload: Scale DL models to trillion parameters wi...
by liuliu on Saturday March 13, 2021 @03:51PM
(#26449877)
They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.
Re:Zero-3 Offload: Scale DL models to trillion parameters wi...
by p1esk on Saturday March 13, 2021 @05:56PM
(#26450942)
They did all that before: https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.