thw20
Re:High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Rec...
on Tuesday April 21, 2026 @12:46PM (#47851290)
Good work! This is very interesting. Here's a related work that construct low-rank approximation for attention: https://arxiv.org/abs/2505.12942.
Maybe the idea of Query calibration matrix Rxx is of interest to the author!
Re:Towards understanding multiple attention sinks in LLMs
on Tuesday March 17, 2026 @05:36AM (#47410424)
The up to date paper documenting and analysing the observation is now available on arxiv!
Re:Towards understanding multiple attention sinks in LLMs
on Saturday March 14, 2026 @07:18AM (#47375508)
This project reveals an interesting phenomena, where LLM converts semantic non-informative tokens to attention sinks through middle layer MLP.
The converted sinks are termed secondary attention sinks as they are weaker then BOS attention sinks.
This might be related to layer specialisation in LLM!
Towards understanding multiple attention sinks in LLMs
Re:An Intuitive Introduction to PPO and GRPO
on Wednesday March 04, 2026 @11:14AM (#47249656)
This is so amazing. What a masterpiece for intro to reinforcement learning in llm.