Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Tensor Product Attention Is All You Need. Parent comment
But assuming your KV cache size is << model size, that simplification is pretty accurate.
See, e.g. https://www.databricks.com/blog/llm-inference-performance-en...
You can just scroll to the first chart they have that explains the idea.
There are no comments on this comment yet.
But assuming your KV cache size is << model size, that simplification is pretty accurate.
See, e.g. https://www.databricks.com/blog/llm-inference-performance-en...
You can just scroll to the first chart they have that explains the idea.