News for hackers. Stuff that matters.

A comment on StreamingLLM: tiny tweak to KV LRU improves long conversations.

Re:StreamingLLM: tiny tweak to KV LRU improves long conversa... by Translationaut on Tuesday February 13, 2024 @04:46AM (#39356095)
This seems only to work cause large GPTs have redundant, undercomplex attentions. See this issue in BertViz about attention in Llama: https://github.com/jessevig/bertviz/issues/128
[ Parent | on HN ]

There are no comments on this comment yet.