When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by DiscourseFan on Tuesday August 04, 2026 @06:01PM (#49175815)

I think LLMs will continue to improve in the capabilities which they are demonstrably good at, but there are many things which they are not good at which it is not cost effective or meaningful to improve, and in these areas we will not consider them “intelligent,” in the same way that we don’t consider computers “intelligent” but do find them very good at doing wrote calculations.

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by naasking on Wednesday August 05, 2026 @01:10PM (#49185768)

> but there are many things which they are not good at which it is not cost effective or meaningful to improve

Can you name a few such things so I can keep an eye on them in the coming years?

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by DiscourseFan on Wednesday August 05, 2026 @01:42PM (#49186253)

Not explicitly, no, but its pretty obvious if you work in the industry and aren't blinded by AGI hype

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by naasking on Wednesday August 05, 2026 @03:54PM (#49188088)

That's just vibes then. How is that supposed to be convincing?

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by DiscourseFan on Wednesday August 05, 2026 @09:35PM (#49191339)

Unless you are working for a competing firm or have a specific clause in your contract, you can literally just sign up for AI projects as a contractor and start contributing. You will see and understand everything after a few months. Nobody is hiding this knowledge really.

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by naasking on Thursday August 06, 2026 @09:07AM (#49196170)

I use LLMs regularly for real work. I understand the current limitations, but you're talking about existing limitations that will never be overcome, because they cannot be overcome. I'm just asking for a couple of specific examples, and your evasions are increasingly suspicious.

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by DiscourseFan on Thursday August 06, 2026 @04:47PM (#49202312)

If you are using them and you are aware of their limitations it’s very different from specifically testing them for training purposes. You’ll get a much clearer picture that way.

[ Parent | on HN ]

Re:When AI Benchmarks Plateau: A Systematic Study of Benchma... by naasking on Friday August 07, 2026 @12:16PM (#49212723)

The continued vagueness suggests this isn't going anywhere, so I'll just conclude by saying that you've given nobody any real reason to believe this.

[ Parent | on HN ]