Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on 26× Faster Inference with Layer-Condensed KV Cache for Large Language Models. Parent comment
If we split a model layer-wise for inference across X GPUs, does the cache split as well?
There are no comments on this comment yet.
If we split a model layer-wise for inference across X GPUs, does the cache split as well?