News for hackers. Stuff that matters.
Re:Attention at Constant Cost per Token via Symmetry-Aware T... by fheinsen on Wednesday February 04, 2026 @01:47PM (#46889911)
Yes, there must be a connection. While adaptive truncation may prove impractical, it should be possible to measure spectral statistics on sample data, and specify a different fixed truncation order per layer, per head, etc. The github repository lists many other possible improvements: https://github.com/glassroom/sata_attention#proof-of-concept

There are no comments on this comment yet.