News for hackers. Stuff that matters.
Re:Reduce cache memory on avg 39x by jonathanehrlich on Monday June 09, 2025 @01:40PM (#44226933)
What if we trained a smaller KV cache for our documents offline? Using a test-time training recipe we call self-study, we find that this can reduce cache memory on avg 39x (enabling 26x higher tok/s and lower TTFT) while maintaining quality. These smaller KV caches, which we call cartridges, can be trained once and reused for different user requests!

There are no comments on this comment yet.