adchurch

> I’m not really sure, I can’t say I trust any kind of synthetic AI benchmarks.

Fair enough but what exactly were you thinking when you said:

> I would like to see this (and any model router, frankly) benchmarked against two things

And thanks for sharing about failure cases! Those do sound like strange harness-level things, honestly we haven't seen failure cases like that crop up in our own usage & testing.

Great idea, we’d love to run those benchmarks! Any in particular you trust?

Also I’m very interested in the unique failure cases you’re referring to! What have you noticed?

Routing is the type of decision shape Jev is meant for. But from our own testing (plus externally run benchmarks) it’s much worse at routing than our purpose-built model.

I’d argue it’s similar to looking at the Meta algorithm for serving ads and asking “what about astra” - it could come up with an answer but it’s not the right model for the task.

Yes! We believe this is entirely possible. Frontier LLMs are trained to solve different problems. We’re training a frontier router :)

We support all the LLMs that Copilot sends data to!

My calculation exactly!

Not yet! Something we're interested in experimenting with though, hit me up at [email protected] if you have any thoughts here

For our hosted version we carefully select which providers we use (we don't use OpenRouter). For the self-hosted version though, OpenRouter does make it much easier to get started (at the cost of that variance potentially affecting quality).

Yes for both! We have some logic to put providers on cooldowns and/or deprioritize them.

To the point of using buckets of models: as long as there's >0 models available in a bucket, and we can order models in order of fit, we're resilient to different sets of models being available. With that said obviously cutting out some models has a much larger effect than others.

OpenRouter isn't strictly necessary but it does make it easier to not have to set up accounts/API keys with several different providers to get started.

Different frontier models are good at different things! We'll be the ones combining them optimally.

I haven't tried it as recently but last time I checked they only route once per session (or subagent). Imo this is basically impossible to do correctly. Consider the case where you start with one prompt "rewrite this in rust". Trivial in a 1 day old repo, extremely difficult in e.g. the VSCode repo!

Yes!

You can think of buckets as models with similar capabilities. So for example Deepseek 4.1 Flash will not be in the same bucket as Astra.

The latter is an interesting question! In practice because the session continues, we can see it's going down the wrong path and escalate. Basically no decision we make when routing is entirely unsalvageable (but we do have a performance penalty for every incorrect decision we make so of course we try to avoid it).

It is not sorry!

Absolutely!

Conceptually very similar to Cursor's auto mode. The key distinctions are:

- We plug into any harness (e.g. Claude Code, Codex, OpenCode, Pi)

- We aren't incentivized to route to our own model, we're incentivized to route to the best model whatever it may be

Effectively yes (based on cost though, not raw token count)

We trained a model to select which LLM to call at any given turn, based on lots of agent traces

Yes the open source models are very good, that’s a big part of what makes this router save so much money in practice! There definitely are some things they still don’t handle well though where you do want a frontier model

Yes we can route to Gemini models too and we handle all the translation complexity there!

We welcome the competition :)

Yep exactly

Yes because it's a model explicitly trained to make model selections! Opus probably doesn't have a great idea of when to send a task to DeepSeek vs. to Sonnet, for example.

We haven't experimented with routing to local LLMs much. Technically they benefit from the cache too although it's more a question of latency than cost. But tbh I haven't seen great results in the wild from working with local LLMs for coding - curious if you've had any success with them?

I think the key detail here is that we use embeddings of the prompt + previous context in order to decide where to route the request, and if one model is getting stuck we can course correct and move to a different model.

So: we can reasonably cluster similar problems together and learn how models handle them, and the entire system doesn't fail if the initial decision is off.

We consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448

Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware).

As another commenter (https://news.ycombinator.com/item?id=48689994) pointed out, for one-off requests, I think it makes more sense to lock to one model whose behavior you understand very well. For dynamic requests like the ones going to a coding agent I think dynamic routing makes more sense but it does need to be cache aware.

Cool, interested to see your approach when you do launch! I think it's a really interesting problem