I've been doing some research lately on my own (aside from the major benchmarking going on) and I've definitely found cases where medium models do much poorly and cost more than the higher models. But on the other hand, if I can give the medium models certain skills or other resources to complete a task then they perform better in cost than the higher model (which, given the same resources, sometimes gets more confused than the medium does).
At least for my use case of evaluations of a set of features.
Yup, in my benchmarks usually medium or high do better than xhigh or max
Asking people to guess beforehand how much intelligence they require for a task will hopefully be a relic soon. We’re lacking useful routing techniques which accurately pick the right model for the job. Or a high intelligence model which completes simple tasks fast and cheap. Or something else.
Yeah, an xhigh models should know a task is easy and respond quicker.
Even better at the local LLM stage, where QFN can catch a mistake in prior work and patch it without any subscription costs at all. then I can use a cloud model for polish and finish and move on.
I'm building stuff at home with my work focused on managing infra. If anyone wants to sponsor a SOTA model subscription I will check it out. Until then I'll see how flexible a Strix Halo setup and my homelab can be.
I also started with Qwen-3.8-27b and used for a while, I still use it so Claude can delegate tasks to it, but the SOTA models are still a lot better than local models, so much that I don't see much sense to use local models as long as I have a paid subscription.
Even if the difference is a single extra error caught early, that can end up saving me hours of work later.
If the highest tier model makes even one or two fewer mistakes that require human intervantion, that almost makes it a no-brainer choice for production.