News for hackers. Stuff that matters.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by WhitneyLand on Monday May 20, 2024 @03:08PM (#40418985)
But it’s not free, it cuts quality significantly.

It’s not hard to find ways to speed up transformers if you’re willing to give up quality.

You could argue some tradeoffs are worth it and that’s true sometimes but I don’t see that they’ve made the case for it here.

There are no comments on this comment yet.