Re:GPT-4 Details Leaked
by downWidOutaFite on Tuesday July 11, 2023 @01:58AM
(#36676919)
The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures and up in cash to create one of these super big models. Only big tech was ever going to create them, and big tech is already very experienced at navigating regulation, regulation wasn't ever going to stop them from competing. And in general, our democracy works better than the nihilist libertarian religion gives it credit for.
[ Parent
| on HN ]
I bet it'll be 6 figures within 18 months.
Even if it was free to train, FAANGs can beat OpenAI on spending to follow regulations.
The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length.
We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time.
That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable.
The work isn’t deciding if an MoE is the right architecture or not, it’s how to run 25k GPUs concurrently in a fault tolerant way (likely the true reason for the deep Azure links).
Re:GPT-4 Details Leaked
by why_only_15 on Tuesday July 11, 2023 @04:22AM
(#36677978)
Memory does not scale quadratically with sequence length.
During training, you have to store a dot product of Q and V that has dimension Ncrt^2.
That's quadratic scaling, no?
Re:GPT-4 Details Leaked
by why_only_15 on Tuesday July 11, 2023 @07:39PM
(#36688637)
presumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135
I mean, sure you can work around it, but from your own link:
>since the time and memory complexity of self-attention are quadratic in sequence length
Re:GPT-4 Details Leaked
by why_only_15 on Wednesday July 12, 2023 @01:37PM
(#36698468)
Except in practice this is not true, and hasn't been for more than a year. It's not just a workaround either -- FlashAttention is both faster at runtime and uses less memory.
I interpreted openAIs regulatory capture bid as more of an attempt to create competition-hostile regulation than an attempt to reduce regulation to cut costs.
Re:GPT-4 Details Leaked
by downWidOutaFite on Tuesday July 11, 2023 @04:08AM
(#36677874)
My point is that openai's competitors have no problem handling regulations.