News for hackers. Stuff that matters.
Re:Artificial Analysis v5 and the benchmarker's dilemma by lordofgibbons on Friday October 30, 2026 @06:34AM (#50302088)
I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.

I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.

Re:Artificial Analysis v5 and the benchmarker's dilemma by artemisart on Friday October 30, 2026 @06:47AM (#50302193)
Maybe you need to enable them in the filters, but AA is the best website I found with benchmarks numbers on all reasoning effort levels of recent frontier models.
Re:Artificial Analysis v5 and the benchmarker's dilemma by lordofgibbons on Friday October 30, 2026 @06:56AM (#50302253)
I dug into it more, and it seems like the reasoning efforts are available for the Intelligence Index, but not the Coding Agents benchmark I was interested in.
Re:Artificial Analysis v5 and the benchmarker's dilemma by ac29 on Friday October 30, 2026 @06:47AM (#50302189)
The other reasoning levels are almost always benchmarked, AA just doesnt show them in the default charts.
Re:Artificial Analysis v5 and the benchmarker's dilemma by frenchtoast8 on Friday October 30, 2026 @07:40AM (#50302610)
They don’t benchmark different reasoning levels for Deepseek models.
Re:Artificial Analysis v5 and the benchmarker's dilemma by lordofgibbons on Friday October 30, 2026 @06:54AM (#50302242)
I see them for GPT 6.1 Terra, but not Opus 5.5. I'm looking on the Coding Agents page.
Re:Artificial Analysis v5 and the benchmarker's dilemma by kzrdude on Friday October 30, 2026 @12:09PM (#50305961)
Is it different between major models how they scale with reasoning level? Some models where it hardly pays off? I expect it to be very similar across the board.
Re:Artificial Analysis v5 and the benchmarker's dilemma by metadat on Friday October 30, 2026 @12:17PM (#50306085)
It is variable, why would independent implementations be highly similar?
Re:Artificial Analysis v5 and the benchmarker's dilemma by bbstats on Friday October 30, 2026 @12:25PM (#50306219)
they do