Re:A Jev that can ask Opus for help gives Opus accuracy at 4...
by weitendorf on Friday November 06, 2026 @11:55PM
(#50401750)
Yep. It's actually kind of mind blowing how well "reasoning" works at de-biasing and pushing out of global minima, in hindsight, when you see how jagged things get again without.
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task: https://imgur.com/a/wGFZvGw
(This is a ~15s gif presentation of the results, the full grid shows up about 10s in)
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task: https://imgur.com/a/wGFZvGw
(This is a ~15s gif presentation of the results, the full grid shows up about 10s in)
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...