News for hackers. Stuff that matters.

A comment on Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation. Parent comment

Re:Attention at Constant Cost per Token via Symmetry-Aware T... by fheinsen on Wednesday February 04, 2026 @10:59AM (#46887434)
As linear approximation error approaches similar magnitude as quadratic numerical error, don’t the two start becoming comparable in practice?

I ask because in practice, for inference, attention is typically computed with low-precision (4-bit, 8-bit, 16-bit) floats.

Numerical error, in fact, may be a key factor as to why quadratic attention, in practice, exhibits context rot as context gets longer, analogous to an RNN:

https://www.anthropic.com/engineering/effective-context-engi...

[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by cubefox on Thursday February 05, 2026 @01:19AM (#46896325)
That website says nothing about numerical error potentially causing context rot.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by fheinsen on Thursday February 05, 2026 @08:56AM (#46899673)
As far as I know, there is no widely accepted explanation for context rot.

Numerical error in long sequences of query-key dot-products may be a key factor.

[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by cubefox on Thursday February 05, 2026 @11:59AM (#46901759)
That should be easy to test: test a 16 bit model on various benchmarks, once with fresh context and once with the context filled up with irrelevant tokens. Record the relative performance degradation, and then do the same for a quantized model. Compare whether the quantized model has a significant relatively larger performance drop from context rot. If so, numerical error should be the cause.
[ Parent | on HN ]