News for hackers. Stuff that matters.

A comment on LM2: Large Memory Models. Parent comment

Re:LM2: Large Memory Models by biofox on Friday February 14, 2025 @02:54AM (#43045968)
Isn't that all of modern AI?
[ Parent | on HN ]
Re:LM2: Large Memory Models by immibis on Friday February 14, 2025 @06:25AM (#43047296)
Transformers are completely unlike RNNs.
[ Parent | on HN ]
Re:LM2: Large Memory Models by tripplyons on Friday February 14, 2025 @11:40AM (#43050205)
There are some interesting connections between them. If you remove the softmax from the attention formula, you end up with linear attention, which has a recurrent form.

I haven't read it, but the Mamba 2 paper claims to establish a stronger connection.

[ Parent | on HN ]
Re:LM2: Large Memory Models by kadushka on Friday February 14, 2025 @11:44AM (#43050258)
* If you remove the softmax from the attention formula, you end up with linear attention*

Sorry, what?

[ Parent | on HN ]
Re:LM2: Large Memory Models by tripplyons on Friday February 14, 2025 @12:14PM (#43050562)
Here is a paper explaining it: https://arxiv.org/abs/2006.16236
[ Parent | on HN ]