News for hackers. Stuff that matters.
Re:26× Faster Inference with Layer-Condensed KV Cache for La... by algo_trader on Monday May 20, 2024 @06:12PM (#40421077)
In general, are the per-layer KV caches all independent of each other?

If we split a model layer-wise for inference across X GPUs, does the cache split as well?

There are no comments on this comment yet.