News for hackers. Stuff that matters.

A comment on Artificial Analysis v5 and the benchmarker's dilemma.

Re:Artificial Analysis v5 and the benchmarker's dilemma by 6thbit on Friday October 30, 2026 @12:29PM (#50306287)
I believe we should treat AA results as just a data point and not the overarching 'authority' they may intend to be.

If we had access to neutral raw benchkmark data in a useful way to compare apples to apples, then we could each define our own 'index' to get the slice of perspective we find useful.

Trying to define or compare 'intelligence' through a single index will always be off.

[ Parent | on HN ]
Re:Artificial Analysis v5 and the benchmarker's dilemma by andriy_koval on Friday October 30, 2026 @12:55PM (#50306695)
> then we could each define our own 'index' to get the slice of perspective we find useful.

its too much work for 98% AI consumers. In reality, AA is best what we have as aggregated score..

[ Parent | on HN ]
Re:Artificial Analysis v5 and the benchmarker's dilemma by 6thbit on Friday October 30, 2026 @02:03PM (#50307811)
Doubt 98% people care, but for those that want to understand I don't think their main index has a good signal anymore.
[ Parent | on HN ]
Re:Artificial Analysis v5 and the benchmarker's dilemma by andriy_koval on Friday October 30, 2026 @02:08PM (#50307883)
> main index has a good signal anymore

maybe (this could be discussed), but the problem there is nothing better..

[ Parent | on HN ]
Re:Artificial Analysis v5 and the benchmarker's dilemma by nextaccountic on Saturday October 31, 2026 @10:44PM (#50322287)
Such a "neutral raw benchmark" would need to be kept private and highly secured. If the labs get a hold of it, they may benchmaxx it to death

So you really need to develop it yourself. If you buy such benchmark from someone, and the benchmark is any good, you may bet the labs will be buying too

[ Parent | on HN ]