News for hackers. Stuff that matters.

A comment on Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation. Parent comment

Re:Attention at Constant Cost per Token via Symmetry-Aware T... by naasking on Wednesday February 04, 2026 @01:42PM (#46889843)
Your argument just assumes there is no latent structure that can be exploited. That's a big assumption.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by logicchains on Wednesday February 04, 2026 @04:53PM (#46892351)
It's a necessary assumption for the universal approximation property; if you assume some structure then your LLM can no longer solve problems that don't fit into that structure as effectively.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by direwolf20 on Wednesday February 04, 2026 @06:52PM (#46893642)
Neural nets are structured as matrix multiplication, yet, they are universal approximators.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by noosphr on Wednesday February 04, 2026 @07:05PM (#46893778)
You're missing the non-linear activations.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by naasking on Wednesday February 04, 2026 @05:52PM (#46893040)
But language does have structure, as does logic and reasoning. Universal approximation is great when you don't know the structure and want to brute force search to find an approximate solution. That's not optimal by any stretch of the imagination though.
[ Parent | on HN ]