News for hackers. Stuff that matters.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by vessenes on Monday May 20, 2024 @02:01PM (#40418181)
Upshot of the paper -- right now KV caches are implemented for multiple deep layers in LLMs. Why not just the top layer? It would save memory.

Initial result -- those KV caches in lower layers matter, and output suffered.

Updated plan -- cull half the KV layers! This works 'nearly' as well as keeping all of them, with memory and compute savings.

Downside - triple the training, worse out of band / long context performance.

This feels to me like a technique you'd use on a particular architecture deployed at the edge where compute matters and you have a little extra room on performance. Phi-3 on raspberry pi, basically.

Interesting! As always, I wish models showed prompt output in their papers, not just perplexity numbers. But, here we are.

Re:26× Faster Inference with Layer-Condensed KV Cache for La... by adtac on Monday May 20, 2024 @04:55PM (#40420271)
> triple the training

From what I understand, not quite. It looks like the cost of training might be similar, but less parallelisable within a specific token sequence. This is because they have to compute the KV of token T before they can use it in T+1 whereas in a regular training process you can compute the KV at each layer for every subsequence. You're right that it took 2.7x longer to train the smallest model but I wouldn't be surprised if the GPU utilisation was proportionally lower too.

Re:26× Faster Inference with Layer-Condensed KV Cache for La... by algo_trader on Monday May 20, 2024 @06:12PM (#40421077)
In general, are the per-layer KV caches all independent of each other?

If we split a model layer-wise for inference across X GPUs, does the cache split as well?

Re:26× Faster Inference with Layer-Condensed KV Cache for La... by YetAnotherNick on Monday May 20, 2024 @06:36PM (#40421303)
I wish papers could be accepted with negative results. There's a lot of value in not repeating the same mistakes specially in a field like deep learning, which is not an exact science and mostly just driven by intuition.