Re:OpenAI chief goes before US Congress to propose licenses ...
by alexeldeib on Tuesday May 16, 2023 @07:19PM
(#35969094)
I don’t think you really disagree with GP? I think the argument is we peaked on “throw GPUs at it”?
We have all kinds of advancements to make training cheaper, models computationally cheaper, smaller, etc.
Once that happens/happened, it benefits OAI to throw up walls via legislation.
Re:OpenAI chief goes before US Congress to propose licenses ...
by Nevermark on Tuesday May 16, 2023 @08:00PM
(#35969478)
No way has training hit any kind of cost, computing or training data efficiency peak.
Big tech advances, like the models of the last year or so, don't happen without a long tail of significant improvements based on fine tuning, at a minimum.
The number of advances being announced by disparate groups, even individuals, also indicates improvements are going to continue at a fast pace.
Re:OpenAI chief goes before US Congress to propose licenses ...
by yarg on Wednesday May 17, 2023 @07:13AM
(#35973711)
Yeah, it's a little bit RTFC to be honest.
Re:OpenAI chief goes before US Congress to propose licenses ...
by stephc_int13 on Tuesday May 16, 2023 @07:49PM
(#35969387)
The efficiency of training has very unlikely reached its peak or near its peak. We are still inefficient.
But the bottleneck might be elsewhere, in data, what we use to feed them.
Maybe not peaked yet, but the case can be made that we’re not seeing infinite supply…
Re:OpenAI chief goes before US Congress to propose licenses ...
by cma on Tuesday May 16, 2023 @07:55PM
(#35969436)
50% sparsity is almost certainly already being used given that it is accelerated in current nvidia hardware both at training time (usable dynamically through RigL, which also addresses initial conditions) and inference time.
Relevantish: https://arxiv.org/abs/2301.00774
The fact that we can reach those levels of sparseness with pruning also indicates that we're not doing a very good job of generating the initial network conditions.
Being able to come up with trainable initial settings for sparse networks across different topologies is hard, but given that we've had a degree of success with pre-trained networks, pre-training and pre-pruning might also allow for sparse networks with minimally compromised learning capabilities.
If it's possible to pre-train composable network modules, it might also be feasible to define trainable sparse networks with significantly relaxed topological constraints.