News for hackers. Stuff that matters.

A comment on Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation. Parent comment

Re:Attention at Constant Cost per Token via Symmetry-Aware T... by naasking on Wednesday February 04, 2026 @11:19AM (#46887718)
I think any kind of innovation here will have to take advantage of some structure inherent to the problem, like eliminating attention in favour of geometric structures like Grassman flows [1].

[1] Attention Is Not What You Need, https://arxiv.org/abs/2512.19428

[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by findalex on Wednesday February 04, 2026 @11:57AM (#46888269)
Right - e.g., if you're modeling a physical system it makes sense to bake in some physics - like symmetry.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by naasking on Wednesday February 04, 2026 @12:34PM (#46888786)
Indeed, and I think natural language and reasoning will have some kind of geometric properties as well. Attention is just a sledgehammer that lets us brute force our way around not understanding that structure well. I think the next step change in AI/LLM abilities will be exploiting this geometry somehow [1,2].

[1] GrokAlign: Geometric Characterisation and Acceleration of Grokking, https://arxiv.org/abs/2510.09782

[2] The Geometry of Reasoning: Flowing Logics in Representation Space, https://arxiv.org/abs/2506.12284

[ Parent | on HN ]