News for hackers. Stuff that matters.
Re:Gemini 4 Neon by declanjackson on Wednesday November 18, 2026 @05:05PM (#50552868)
Something that is interesting is how it's positioned in the overall fleet of Gemini models. Although overall more Intelligent than Carbon, Carbon is still positioned as the 'long horizon' model. This is clear on AA Index, where we see higher overall scores for Neon but lower scores in evals like CritPt which test research-scale reasoning capabilities.

With the frontier getting more jagged, we might see more of this 'less intelligent overall but useful for the frontier tasks'-type models, meaning picking a model will only get more difficult from here.

(FD: I work at AA)

Re:Gemini 4 Neon by abixb on Wednesday November 18, 2026 @05:07PM (#50552897)
Yes, I feel the next differentiation in model capabilities would be wrt how 'long of a horizon' goal they could autonomously work toward without breaking. We're in need for a new generation of evaluations and benchmarks for this -- METR doesn't cut it anymore.
Re:Gemini 4 Neon by drivebyhooting on Wednesday November 18, 2026 @05:11PM (#50552969)
What we need is a model that automatically delegates intelligently (via RL and benchmaxxing?) so I can be freed of this burden.
Re:Gemini 4 Neon by iamdelirium on Wednesday November 18, 2026 @05:15PM (#50553020)
Antigravity has a Orchestrator model that does this.
Re:Gemini 4 Neon by refulgentis on Wednesday November 18, 2026 @05:22PM (#50553115)
There's stuff that does this and I've put a substantial amount of time last couple months looking into it, unfortunately you always, always, end up at chicken-and-egg problems.

It's really frustrating to describe. Easiest is you can't score on just the prompt, you need the full context. Ex. "Solve Fermat's last theorem" initial prompt then "continue" followup prompt. Which gets you back in the tarpit of "if I have to pay a model to do an inference run on context, then, the context is cached and I should continue using that model."

The vision of another model thats RL maxx'd on knowing how to delegate has a few problems. One, we can observe externally, it died/didn't work even at OpenAI's scale and willingness to make it work, i.e. the...GPT-5.0? IIRC...router that silently went away. The other, if such a model was built, it'd also be good at regulating it's own thinking enough that it'd naturally use less tokens.

In practice, the frontier hackers have settled on a decision model (i.e. Jev) x calibrating effort level or choosing from a set of models x reasoning effort tuples they specify rules for choosing in their Jev prompt. My guess is this stays stable and the only real choice.

Re:Gemini 4 Neon by Gareth321 on Thursday November 19, 2026 @01:27AM (#50556705)
I suspect you are correct but damn if that’s not a deeply unsatisfying conclusion.
Re:Gemini 4 Neon by baoooooooooooo on Wednesday November 18, 2026 @05:40PM (#50553365)
This comment always comes up but isn’t this literally irreducibly complex in the sense of Kolmogorov complexity?

I’m out of my depth here but it does seem prompts have a “smell” to their complexity that means we can use a classifier to mate them to a model with better success than random noise, but still don’t essentially all of the savings associated with systems like this just come because there is some “complexity arbitrage” where there is enough of a token cost differential between models that you always save money by trying the cheap model first and then trying a hard model if it turns out to be hard?

Re:Gemini 4 Neon by Gareth321 on Thursday November 19, 2026 @01:26AM (#50556703)
I’m also not an expert but that sounds a lot like the toilet seat problem. Mathematically it’s preferable to just always leave the seat as you used it, instead of using effort to return it to some desired state. This minimises effort in aggregate for all parties. I think it’s possible that routing every request through a simpler model first ends up more expensive in the long run, depending on the cost of that simple model. And if it’s too simple, the routing inefficiencies and mistakes will produce worse results.

I think one of the more difficult problems with this is that the complexity is often unclear at first. It’s not until some thinking and analysis that the complexity emerges. It’s just not possible for a simple model to deduce this, and we don’t want a dumb model investigating a complex issue.

I think in the medium term we end up with large models which are efficient with simple queries. Meaning very few tokens used to answer simple queries, keeping costs low. So one just uses this large model for everything.

Longer term I think LLMs move closer to the structure of the brain, whereby different regions/models have specialised functionality, and they communicate and coordinate depending on the task. For example, telemetry doesn’t need to pass through the prefrontal cortex. It just goes straight to the telemetry model which is super simple and cheap.

Re:Gemini 4 Neon by nicoburns on Wednesday November 18, 2026 @05:48PM (#50553470)
This is being done at the layer above. "routing" is the term to google.
Re:Gemini 4 Neon by drivebyhooting on Wednesday November 18, 2026 @06:48PM (#50553998)
Routing at the prompt level seems naive. How is that better than a janky classifier.

What I want is a deeply introspective model that knows to take a step back and call in for aid.

Re:Gemini 4 Neon by nicoburns on Wednesday November 18, 2026 @06:51PM (#50554027)
It's being done at the harness level. It's still a model making the call.
Re:Gemini 4 Neon by canadiantim on Wednesday November 18, 2026 @10:48PM (#50555844)
Aren't people using jev or similar for decision-making like this?
Re:Gemini 4 Neon by seizethecheese on Wednesday November 18, 2026 @05:12PM (#50552986)
That would clear up why Carbon is such an outlier in terms of output tokens per task.
Re:Gemini 4 Neon by WarmWash on Wednesday November 18, 2026 @05:23PM (#50553143)
Neon is probably distilled from Google's internal successor to Carbon.
Re:Gemini 4 Neon by mi_lk on Wednesday November 18, 2026 @05:25PM (#50553173)
what's AA
Re:Gemini 4 Neon by mohamedsayhii on Wednesday November 18, 2026 @05:28PM (#50553206)
Artificial Analysis
Re:Gemini 4 Neon by GodelNumbering on Wednesday November 18, 2026 @05:35PM (#50553291)
> (FD: I work at AA)

I do apologize in advance for putting you in spot like this, but there are two things I have been quite curious about.

1. Since 60% of the benchmarks contributing to AA are now private only, how do you guys ensure transparency? More precisely, how can someone verify the transparency in scoring?

2. There are also disclosures about custom benchmarking services offered to AI labs, what does AA do to keep the custom benchmarking services not getting access to the same/similar data that is used in the private benchmarks?

Re:Gemini 4 Neon by seizethecheese on Wednesday November 18, 2026 @05:46PM (#50553443)
Okay, this "better at long horizon tasks" thing might explain why Carbon scores worse on most benchmarks yet is the larger model.

I looked into this using data I have from a battle system built on top of OpenCode (so we're trying to capture real world use). It looks like Carbon actually does a lot better when it uses significantly more output than opponents.

Win rate when output is less than 2.5x opponents: 53% (41/78)

Win rate when output is more than 2.5x opponents: 73% (41/56)

The p-val on this is 0.02. Data from https://agentduel.com/leaderboard.

There doesn't seem to be any relationship between actual size of task though. Win rate appears random based on total output of Carbon and also opponents. (Though our data is still small, so it might exist.)

Looks basically like Carbon was trained to think a lot longer, not necessarily that it's better on large tasks.

Re:Gemini 4 Neon by chr15m on Wednesday November 18, 2026 @09:52PM (#50555467)
Does this mean models are not generalizing like the labs hoped?