News for hackers. Stuff that matters.

A comment on Ask HN: What is the best way to provide continuous context to models?. Parent comment

Re:Ask HN: What is the best way to provide continuous contex... by Fazebooking on Friday January 16, 2026 @08:12AM (#46646036)
Thats only the case with KV Cache and we do not know how and how long providers keep it.
[ Parent | on HN ]
Re:Ask HN: What is the best way to provide continuous contex... by bluegatty on Friday January 16, 2026 @05:31PM (#46653097)
Prefill is 10x faster than generation without caching, and 100x faster with caching - as a very crude measure. So it's not a matter of 'only the case'. Those are different scenarios. Some hosts are better than others with respect to managing caching, but the better one's provide decent SLA on that.
[ Parent | on HN ]