News for hackers. Stuff that matters.

A comment on 3Blue1Brown: Visualizing Attention, a Transformer's Heart. Parent comment

Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by YetAnotherNick on Monday April 15, 2024 @05:38AM (#40038421)
No. This is blatantly false. The belief that recurrent model can't be scaled is untrue. People have recently trained MAMBA with billions of parameters. The fundamental reason why transformers changed the field is that they are lot more scalable context length wise, and LSTM, LRU etc doesn't come close.
[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by mjburgess on Monday April 15, 2024 @06:27AM (#40038661)
> they are lot more scalable context length wise

Sure, we're agreeing. I'm just being less specific.

[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by YetAnotherNick on Monday April 15, 2024 @07:51AM (#40039279)
Scalable as in loss wise scalable, not compute wise.
[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by HarHarVeryFunny on Monday April 15, 2024 @09:54AM (#40040576)
Yes, but pure Mamba doesn't perform as well as a transformer (and neither did LTSMs). This is why you see hybrid architectures like Jamba = Mamba + transformer. The ability to attend to specific tokens is really key, and what is lost in recurrent models where sequence history is munged into a single state.
[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by YetAnotherNick on Monday April 15, 2024 @12:33PM (#40042738)
That's my point. It doesn't perform in terms of loss, even though it performs well enough in terms of compute
[ Parent | on HN ]