Re:AI benchmarks are a bad joke – and LLM makers are the one...
by scuff3d on Sunday November 09, 2025 @02:55AM
(#45863762)
So because there isn't a better measure it's okay that tech companies effectively lie and treat these benchmarks like they mean more then they actually do?
Re:AI benchmarks are a bad joke – and LLM makers are the one...
by ACCount37 on Sunday November 09, 2025 @12:14PM
(#45867140)
Sorry, pal, but if benchmarks were to disagree with opinions of a bunch of users saying "tech companies bad"? I'd side with benchmarks at least 9 times out of 10.
Re:AI benchmarks are a bad joke – and LLM makers are the one...
by scuff3d on Sunday November 09, 2025 @02:25PM
(#45868350)
How does that have anything to do with what we're talking about?
Re:AI benchmarks are a bad joke – and LLM makers are the one...
by ACCount37 on Monday November 10, 2025 @07:39AM
(#45875360)
What that has to do is: your "tech companies are bad for using literally the best tool we have for measuring AI capabilities when talking about AI capablities" take is a very bad take.
It's like you wanted to say "tech companies are bad", and the rest is just window dressing.
Re:AI benchmarks are a bad joke – and LLM makers are the one...
by achierius on Sunday November 09, 2025 @01:10AM
(#45863354)
That's assuming these benchmarks are the best we're ever going to get, which they clearly aren't. There's a lot to improve even without radical changes to how things are done.
Re:AI benchmarks are a bad joke – and LLM makers are the one...
by ACCount37 on Sunday November 09, 2025 @12:11PM
(#45867100)
The assumption I make is that "better benchmarks" are going to be 5% better, not 5000% better. LLMs are getting better capabilities faster than the benchmarks get better at measuring them accurately.
So, yes, we just aren't going to get anything that's radically better. Just more of the same, and some benchmarks that are less bad. Which is still good. But don't expect a Benchmark Revolution when everyone suddenly realizes just how Abjectly Terrible the current benchmarks are, and gets New Much Better Benchmarks to replace them with.
Without benchmarks, you're down to evaluating model performance based on vibes and vibes only, which plain sucks. With benchmarks, you have numbers that correlate to capabilities somewhat.