Re:OpenAI to Become For-Profit Company
by espadrine on Thursday September 26, 2024 @11:02AM
(#41659199)
> New LLMs don't even rely that much on that aforementioned older architecture
Don’t they all indicate being based on the transformer architecture?
> not entirely because of transformers but because of the hardware
Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context.
Don’t they all indicate being based on the transformer architecture?
> not entirely because of transformers but because of the hardware
Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context.
[0]: https://arxiv.org/abs/2001.08361