News for hackers. Stuff that matters.

A comment on Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation. Parent comment

Re:Attention at Constant Cost per Token via Symmetry-Aware T... by cubefox on Thursday February 05, 2026 @01:19AM (#46896325)
That website says nothing about numerical error potentially causing context rot.
[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by fheinsen on Thursday February 05, 2026 @08:56AM (#46899673)
As far as I know, there is no widely accepted explanation for context rot.

Numerical error in long sequences of query-key dot-products may be a key factor.

[ Parent | on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by cubefox on Thursday February 05, 2026 @11:59AM (#46901759)
That should be easy to test: test a 16 bit model on various benchmarks, once with fresh context and once with the context filled up with irrelevant tokens. Record the relative performance degradation, and then do the same for a quantized model. Compare whether the quantized model has a significant relatively larger performance drop from context rot. If so, numerical error should be the cause.
[ Parent | on HN ]