Ask HN: How to store KV cache efficiently on disk?
If I have a lot of precomputed KV cache values (basically just tensors), how do I store them efficiently on disk? Ideally, I would want to lookup their values by prefix quickly as well.
If I have a lot of precomputed KV cache values (basically just tensors), how do I store them efficiently on disk? Ideally, I would want to lookup their values by prefix quickly as well.
https://arxiv.org/abs/2306.08891 One paper seems to suggest trained specialized model can outperform LLM in some tasks.