News for hackers. Stuff that matters.

A comment on GitHub Copilot. Parent comment

Re:GitHub Copilot by 0-_-0 on Tuesday June 29, 2021 @07:02PM (#27682889)
Well, it should be possible to crowdsource training a FOSS version, right? There should be a SETI-at-home for training neural networks. I would donate some GPU power for sure.

SETI@home achieved 50 times the computing power of the world's largest supercomputer [0], so it might actually be the only way to train the future GPT4 or GPT5.

[0]: https://en.wikipedia.org/wiki/SETI@home#Statistics

[ Parent | on HN ]
Re:GitHub Copilot by hgial on Tuesday June 29, 2021 @09:06PM (#27683775)
At the moment crowd-sourcing ML training doesn't work very well because updates to the model's weights have to be shared between all of the nodes in the computation periodically. Giant models tend to be trained on clusters with very fast interconnects for this reason.
[ Parent | on HN ]
Re:GitHub Copilot by devetec on Tuesday June 29, 2021 @09:39PM (#27683950)
Distributed GPT training doesn't really work (sadly). Maybe someone will get a good solution to that, though.
[ Parent | on HN ]