Re:DiLoCo: Distributed Low-Communication Training of Languag...
by GaggiX on Wednesday December 06, 2023 @06:55PM
(#38551121)
In the paper, they usually share every 500 steps, a few million steps is an entire training process, more often than not models are trained less than a million steps.
[ Parent
| on HN ]
There are no comments on this comment yet.