Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by adtac on Monday May 20, 2024 @04:55PM
(#40420271)
> triple the training
From what I understand, not quite. It looks like the cost of training might be similar, but less parallelisable within a specific token sequence. This is because they have to compute the KV of token T before they can use it in T+1 whereas in a regular training process you can compute the KV at each layer for every subsequence. You're right that it took 2.7x longer to train the smallest model but I wouldn't be surprised if the GPU utilisation was proportionally lower too.
Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by algo_trader on Monday May 20, 2024 @06:12PM
(#40421077)
In general, are the per-layer KV caches all independent of each other?
If we split a model layer-wise for inference across X GPUs, does the cache split as well?
Re:26× Faster Inference with Layer-Condensed KV Cache for La...
by YetAnotherNick on Monday May 20, 2024 @06:36PM
(#40421303)
I wish papers could be accepted with negative results. There's a lot of value in not repeating the same mistakes specially in a field like deep learning, which is not an exact science and mostly just driven by intuition.
Initial result -- those KV caches in lower layers matter, and output suffered.
Updated plan -- cull half the KV layers! This works 'nearly' as well as keeping all of them, with memory and compute savings.
Downside - triple the training, worse out of band / long context performance.
This feels to me like a technique you'd use on a particular architecture deployed at the edge where compute matters and you have a little extra room on performance. Phi-3 on raspberry pi, basically.
Interesting! As always, I wish models showed prompt output in their papers, not just perplexity numbers. But, here we are.