Re:The Unbearable Cheapness of Open Weight Models
by pjc50 on Thursday June 25, 2026 @06:05AM
(#48671291)
How does caching help here? How much repetition is there in queries?
[ Parent
| on HN ]
Re:The Unbearable Cheapness of Open Weight Models
by jcparkyn on Thursday June 25, 2026 @07:11AM
(#48671760)
Agent loops (particularly coding agents) have a huge amount of repetition, because the entire context is included in every model request. So long as it's at the start of the input and doesn't change, it will be able to hit the KV cache (assuming the model provider actually has the prefix in cache).
This only works because prompt caching is done by matching prefixes, not the entire input.
Re:The Unbearable Cheapness of Open Weight Models
by AnthonyMouse on Thursday June 25, 2026 @06:29AM
(#48671477)
It probably depends on what you're doing, but imagine you're something in the shape of a search engine. How many user queries are unique vs. the same thing someone else searched for an hour ago?
Re:The Unbearable Cheapness of Open Weight Models
by crazylogger on Thursday June 25, 2026 @10:37AM
(#48674111)
In a typical agent loop your N-th LLM request naturally becomes prefix for the (N+1)-th request. As the thread grows longer, cache hit rate converges to 100% and unit pricing for cached tokens is 10-100x cheaper.
Re:The Unbearable Cheapness of Open Weight Models
by arikrahman on Thursday June 25, 2026 @02:26PM
(#48677387)
Their blob explains it best, although this is a link to an older version design: https://github.com/esengine/DeepSeek-Reasonix/blob/v1/docs/A...
From my understanding if previous tokens are frozen and guaranteed to be immutable you can leverage that.