Re:GPT-6 Sol and Luna
by aucisson_masque on Wednesday September 23, 2026 @02:27AM
(#49812406)
There is too many money involved, benchmark can’t be trusted.
Re:GPT-6 Sol and Luna
by dudisubekti on Wednesday September 23, 2026 @07:54AM
(#49814718)
I had a favorite benchmark, SWE-rebench, but sadly it's no longer maintained.
But yeah, I'll just take these benchmarks with a grain of salt. Only hands-on experience matters in the end, and these days it's very easy to switch models.
"Artificial Analysis Intelligence Index combines performance across 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, Humanity's Last Exam, GDP.pdf, CritPt, AA-Omniscience, and AA-LCR v1.1."
Not saying it doesnt have any value but it's probably irrelevant if you use these AIs for a specific use case. Like for example Humanity Last Exam tests general knowledge, which is not very useful for coding.
It's best to go to the specific coding benchmarks and compare there.