Re:DiLoCo: Distributed Low-Communication Training of Languag...
by Anon84 on Wednesday December 06, 2023 @06:27PM
(#38550903)
A few years ago, someone (Apple?) was working on a way to distribute the training of ML models across personal devices. The idea was that the master model in the Cloud could be trained on your personal data without the data ever leaving your device. Not sure if that was ever put into production, but this feels like a scaled-up version of that with distributed data centers instead of iPhones
[ Parent
| on HN ]
Re:DiLoCo: Distributed Low-Communication Training of Languag...
by magicarp on Wednesday December 06, 2023 @06:33PM
(#38550952)
Google has done a lot of work in this area: https://federated.withgoogle.com/
Re:DiLoCo: Distributed Low-Communication Training of Languag...
by mattnewton on Wednesday December 06, 2023 @06:42PM
(#38551014)
Google has shipped this in gboard, the Google keyboard - https://arxiv.org/pdf/1812.02903.pdf
(Disclaimer, I worked on a later version of this for other models in the android keyboard at Google)