News for hackers. Stuff that matters.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by smaddox on Wednesday May 29, 2024 @12:32PM (#40513826)
Transformers have quadratic computational complexity in sequence length, i.e. O(N^2) where N is the sequence length. RNNs, Linformer, Mamba, etc. have linear or quasi-linear computational complexity in sequence length, which often bottlenecks information movement across tokens.

In theory, if you grew the RNN's state quadratically vs sequence length, you could likely achieve comparable performance to transformers, but it would likely be less efficient than transformers.

There are no comments on this comment yet.