News for hackers. Stuff that matters.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by WhitneyLand on Monday May 20, 2024 @03:04PM (#40418939)
Not sure if @dang is the right way to say the title is incorrect here, but shouldn't it match the paper?

1. The correct title is Layer-Condensed KV Cache for Efficient Inference of Large Language Models.

2. The paper does make a 26x claim later in the introduction, but it’s an outlier.

26x is for only one benchmark and that benchmark is CPU based, not GPU based like 99% of transformer loads actually run on.

If you look at GPU only workloads, the improvements range from 1.4x to 4.7x.

Re:26× Faster Inference with Layer-Condensed KV Cache for La... by kiraaa on Monday May 20, 2024 @03:54PM (#40419528)
on gpu that is still huge.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by 0cf8612b2e1e on Monday May 20, 2024 @04:15PM (#40419771)
Even if it is “only” the 40% lower end, that is a gargantuan savings. So many groups are compute constrained, every bit helps.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by josephg on Monday May 20, 2024 @05:14PM (#40420485)
Sure; but 40% improvement is much less than a 26x improvement. If 40% is the realistic figure, cite that. Changing the title to include an outlier of 26x is click baity.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by VeejayRampay on Monday May 20, 2024 @04:21PM (#40419843)
it sure is huge, but it's still far from 26x
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by imtringued on Tuesday May 21, 2024 @02:24AM (#40424801)
It's not an outlier. You will see more than 26x improvement if you try this on an even deeper LLM with more layers. The deepest model they have applied it on has 30 billion parameters.

Edit: I apologize. The table was cut off on mobile and I didn't see that they sneaked in GPU+CPU offloading for the 25x result.

Re:26× Faster Inference with Layer-Condensed KV Cache for La... by wglb on Tuesday May 21, 2024 @11:45AM (#40429956)
> Not sure if @dang is the right way to say the title is incorrect here, but shouldn't it match the paper?

Best way to address this is to email [email protected].

It is not possible to read all posts to HN.

(Also, the convention is to use 'dang rather than @dang, borrowing from a lisp convention.)