Towards understanding multiple attention sinks in LLMs

Posted by thw20 on Saturday March 14, 2026 @07:15AM
from the github.com dept.

( 2 comments | on HN )

Re:Towards understanding multiple attention sinks in LLMs by thw20 on Saturday March 14, 2026 @07:18AM (#47375508)

This project reveals an interesting phenomena, where LLM converts semantic non-informative tokens to attention sinks through middle layer MLP.

The converted sinks are termed secondary attention sinks as they are weaker then BOS attention sinks.

This might be related to layer specialisation in LLM!

[ Parent | on HN ]

Re:Towards understanding multiple attention sinks in LLMs by thw20 on Tuesday March 17, 2026 @05:36AM (#47410424)

The up to date paper documenting and analysing the observation is now available on arxiv!

[ Parent | on HN ]