Essentially bake the context passed in to the attention layer so that you can pass only the relevant chat context.
Your post wasn’t a particularly charitable interpretation.
$20 hosting can serve thousands of users per month. $20 llm sub services just one person. This is fucking impossible.
I'm building a small assistant tool and thought I'm forced to use APIs!
It reduces prompt costs by half for those shared prefix tokens, but you have to pay $4.50/million tokens/hour to keep that cache warm - so probably not a useful optimization for most lower traffic applications.
That's on a model with $3.5/1M input token cost, so half price on cached prefix tokens for $4.5/1M/hour breaks even at a little over 2.5 requests/hour using the cached prefix.
Tediously, it would be possible to do this chapter by chapter in order to exceed the output limit building something for future inputs.
Of course, the summary might not fulfill the same functionality as the original source document. YMMV
According to https://ai.google.dev/pricing it's $0.70/million input tokens (for a long context). That will be per-exchange, so every little back and forth will cost around that much (if you're using a substantial portion of the context window).
And while I haven't tested Gemini, most LLMs get increasingly wonky as the context goes up, more likely to fixate, more likely to forget instructions.
That big context window could definitely be great for certain tasks (especially information extraction), but it doesn't feel like a generally useful feature.