Re:OpenAI to Become For-Profit Company
by espadrine on Thursday September 26, 2024 @11:02AM
(#41659199)
> New LLMs don't even rely that much on that aforementioned older architecture
Don’t they all indicate being based on the transformer architecture?
> not entirely because of transformers but because of the hardware
Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context.
We've had upgrades to hardware, mostly led by NVidia, that made it possible.
New LLMs don't even rely that much on that aforementioned older architecture, right now it's mostly about compute and the quality of data.
I remember seeing some graphs that shows that the whole "learning" phenomena that we see with neural nets is mostly about compute and quality of data, the model and optimizations just being the cherry on the cake.