Re:Why DeepSeek is cheap at scale but expensive to run locally
by ryao on Sunday June 01, 2025 @03:24PM
(#44153173)
If he is doing multiturn conversations, he can reuse the kv cache from the last turn and skip the prompt processing on the history that would make time to first token too slow, by only doing prompt processing on his actual prompt for the current turn. This turns a quadratic amount of tokens to process into a linear number. I am not sure if this is what he is doing, but that is what I would do if I had his hardware.
[ Parent
| on HN ]
There are no comments on this comment yet.