News for hackers. Stuff that matters.
Re:DiLoCo: Distributed Low-Communication Training of Languag... by GaggiX on Wednesday December 06, 2023 @06:55PM (#38551121)
In the paper, they usually share every 500 steps, a few million steps is an entire training process, more often than not models are trained less than a million steps.

There are no comments on this comment yet.