I think one of the more difficult problems with this is that the complexity is often unclear at first. It’s not until some thinking and analysis that the complexity emerges. It’s just not possible for a simple model to deduce this, and we don’t want a dumb model investigating a complex issue.
I think in the medium term we end up with large models which are efficient with simple queries. Meaning very few tokens used to answer simple queries, keeping costs low. So one just uses this large model for everything.
Longer term I think LLMs move closer to the structure of the brain, whereby different regions/models have specialised functionality, and they communicate and coordinate depending on the task. For example, telemetry doesn’t need to pass through the prefrontal cortex. It just goes straight to the telemetry model which is super simple and cheap.
I’m out of my depth here but it does seem prompts have a “smell” to their complexity that means we can use a classifier to mate them to a model with better success than random noise, but still don’t essentially all of the savings associated with systems like this just come because there is some “complexity arbitrage” where there is enough of a token cost differential between models that you always save money by trying the cheap model first and then trying a hard model if it turns out to be hard?