Re:Why DeepSeek is cheap at scale but expensive to run locally
by pclmulqdq on Sunday June 01, 2025 @08:14PM
(#44154803)
Usually, when people think about the prompt tokens for a chat model, the initial system prompt is the vast majority of the tokens and it's the same regardless for many usage modes. You might have a slightly different system prompt for code than you have for English or for chatting, but that is 3 prompts which you can permanently put in some sort of persistent KV cache. After that, only your specific request in that mode is uncached.
[ Parent
| on HN ]
There are no comments on this comment yet.