caching works perfectly fine with routers.

To elaborate, because I don't think some of the people reading this understand the reason, typically a lot or most of the cost in "agentic" API usage is cached read + generation. Cached read costs scale with turn count, which multi-model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model. When you switch models the "catching up" batch of messages is just a single prefill and then that goes into cache. You don't even need to have the same chat history across models so long as the view from each model's perspective looks like a series of appends.

The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.

> which multi-model switching doesn't increase

Given model A with cache C(a) and model B with C(b)

Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?

If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.

I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:

* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?

Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.

Can you explain how does it work? like how is the previous K/V cache used when you switch to another model?

Source?

See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).

edit: updated the answer above to be more qualitative instead