This seems to be the inherent trade-off with smaller models. They can be just as "bright" as their larger contemporaries but they necessarily can't encode as much knowledge.

I think this is what the n-gram table in Qwen 4 (and 3.8-Flash-Next) is for. We’ll see if it comes down to the 27b variant.

A RAG solution would make time to first token awful.

> A RAG solution would make time to first token awful.

in your specific workflows, where specifically is the time to first token a dealbreaker? what's your typical target and what's a "no go"?

I tend to use LLMs conversationally, often as search-and-summarize but also as a pair designer. I find it very difficult to operate in this mode if prefill takes more than, say, a dozen seconds. And that only gets to the reasoning output; depending of effort level it can be multiple minutes to get a reply in a freshly compacted context.

> You check its work and if it seems to do its job well enough most of the time, you just start to trust it.

I don't think you meant to say this but it feels like you're saying "vibes".

It would be better to use a scoring framework like Harbor or similar?

I don’t really have time to be assigning my LLM homework and designing a grading rubric for it. I judge its output like I judge my peers’ PRs.