News for hackers. Stuff that matters.
Re:Reduce cache memory on avg 39x by jonathanehrlich on Monday June 09, 2025 @01:40PM (#44226933)
What if we trained a smaller KV cache for our documents offline? Using a test-time training recipe we call self-study, we find that this can reduce cache memory on avg 39x (enabling 26x higher tok/s and lower TTFT) while maintaining quality. These smaller KV caches, which we call cartridges, can be trained once and reused for different user requests!