Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by kiraaa on Monday May 20, 2024 @03:54PM
(#40419528)
on gpu that is still huge.
[ Parent
| on HN ]
Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by 0cf8612b2e1e on Monday May 20, 2024 @04:15PM
(#40419771)
Even if it is “only” the 40% lower end, that is a gargantuan savings. So many groups are compute constrained, every bit helps.
Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by josephg on Monday May 20, 2024 @05:14PM
(#40420485)
Sure; but 40% improvement is much less than a 26x improvement. If 40% is the realistic figure, cite that. Changing the title to include an outlier of 26x is click baity.
Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by VeejayRampay on Monday May 20, 2024 @04:21PM
(#40419843)
it sure is huge, but it's still far from 26x