News for hackers. Stuff that matters.
Re:DiLoCo: Distributed Low-Communication Training of Languag... by lucubratory on Wednesday December 06, 2023 @05:37PM (#38550453)
So we can SETI@Home or Folding@Home for large language models, now? Not sure how small the minimum size of the compute cluster can be. If it's still out of consumer reach then this would either be just an intermediate research step, or a way for small-er (but still professional/well-resourced) labs to collaborate together. I'm not sure the latter would be helpful, as if they wanted to collaborate together they could probably already do that by pooling resources for a large cloud compute run.
Re:DiLoCo: Distributed Low-Communication Training of Languag... by theendisney2 on Wednesday December 06, 2023 @08:57PM (#38551906)
Someone onhere argued this is entirely possible and that results can be merged later on. It seems facinating. Perhaps it is not conpute each node has to contribute but just bandwidth?
Re:DiLoCo: Distributed Low-Communication Training of Languag... by gaogao on Wednesday December 06, 2023 @06:07PM (#38550728)
The limitations of the paper is more so maximum number of workers than minimum. It'd be pretty neat to get something that could work across 100+ distributed workers well.