Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by tripplyons on Monday May 20, 2024 @03:00PM
(#40418883)
This can be combined with Grouped Query Attention or Multi-Query Attention for an even further reduction in the size of the KV Cache!
[ Parent
| on HN ]
Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by WhitneyLand on Monday May 20, 2024 @03:08PM
(#40418985)
But it’s not free, it cuts quality significantly.
It’s not hard to find ways to speed up transformers if you’re willing to give up quality.
You could argue some tradeoffs are worth it and that’s true sometimes but I don’t see that they’ve made the case for it here.