Re:Techniques for Training Large Neural Networks
by hungrigekatze on Saturday June 11, 2022 @12:27PM
(#31705716)
I was curious as to what the 'community AI' research org's stances on distributed training of deep neural nets were so some weeks ago I stumbled upon Eleuther AI's FAQ page which was talking about how it was not a task that they were looking at due to various technological challenges:
What about volunteer-driven distributed computing, like BOINC, Folding@Home, or hivemind?
-Backpropagation is dense and sensitive to precision, therefore requiring high-bandwidth communication. Consumer-grade internet connections are wholly insufficient.
-Mixture-of-experts-based models tend to significantly underperform monolithic (regular) models for the same number of parameters.
-Having enough contributors to outweigh the high overhead is infeasible.
-Verifiability and resistance to outside attack are not currently possible without significant additional overhead.
In short, doing volunteer-driven distributed compute well for this use case is an unsolved problem.
---
Am really excited to see inroads being made in this field of active research and hope that all AI orgs - OpenAI, Eleuther, etc. - can take part in this in domain of much-needed (IMO) research.
Source: https://www.eleuther.ai/faq/
What about volunteer-driven distributed computing, like BOINC, Folding@Home, or hivemind? -Backpropagation is dense and sensitive to precision, therefore requiring high-bandwidth communication. Consumer-grade internet connections are wholly insufficient. -Mixture-of-experts-based models tend to significantly underperform monolithic (regular) models for the same number of parameters. -Having enough contributors to outweigh the high overhead is infeasible. -Verifiability and resistance to outside attack are not currently possible without significant additional overhead. In short, doing volunteer-driven distributed compute well for this use case is an unsolved problem.
---
Am really excited to see inroads being made in this field of active research and hope that all AI orgs - OpenAI, Eleuther, etc. - can take part in this in domain of much-needed (IMO) research.