Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.
Remember that Deepseek can be something like 30x cheaper even cold.
There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.
How does tokenless quantify "so hard"?
> 1 turn to crack versus 100 turns for Deepseek to crack
Interesting definition for a "frontier". What is a "turn" here? Token count? Request count? Context-based?
I've found that MiniMax M3 (a smaller model at 295b) will code up better when DeepSeek v4 Pro (1.6t) will not (and vice versa).
A turn here is a tool call/user input, anything that causes the model to get some new input. We're working on adding Minimax M3 and other models. We think that people have some intuitions about which models are good when--we seek to quantify them scientifically.
I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie. give model specs for certain delegation roles).
Some agents when you alter the reasoning level, partially or complete wipe the cache. Never assume that changing reasoning is a no-impact change. I do not have the link but a French youtuber did a ton of research on this.
https://platform.claude.com/docs/en/build-with-claude/effort...
Code exploration tends to burn a ton of tokens and do not require frontier knowledge. It then returns a compact overview of the findings to the better model, that can therefore perform its task with less tokens.
Anyone else has a similar approach?
https://github.com/gitsense/chat/tree/update-readme
The README and latest version is still being worked on but I am hoping to have the latest release in place soon.
But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.
How do you deal with that?