News for hackers. Stuff that matters.
Re:DiLoCo: Distributed Low-Communication Training of Languag... by techwizrd on Wednesday December 06, 2023 @08:21PM (#38551711)
There are a lot of distributed data parallel and federated learning (FL) algorithms that could be applied to training LLMs, and there have been several papers that tackle applying these. I don't think I've seen 500 steps in the FL literature (little eye opening to be honest), and I don't think I've tested more than 50 steps between communication rounds personally. I'd be interested in testing other algorithms, partial client participation, hierarchical federation approaches, error-feedback, and so on.

There are no comments on this comment yet.