News for hackers. Stuff that matters.

A comment on OpenAI funded FrontierMath Benchmarks and had access to the set.

Re:OpenAI funded FrontierMath Benchmarks and had access to t... by lionkor on Sunday January 19, 2025 @08:25PM (#42763985)
People on here were mocking me openly when I pointed out that you can't be sure LLMs (or any AIs) are actually smart unless you CAN PROVE that the question you're asking isn't in the training set (or adjacent like in this case).

So with this in mind now, let me repeat: Unless you know that the question AND/OR answer are not in the training set or adjacent, do not claim that the AI or similar black box is smart.

[ Parent | on HN ]
Re:OpenAI funded FrontierMath Benchmarks and had access to t... by pcmoore on Sunday January 19, 2025 @09:41PM (#42764442)
I ran a test yesterday on ChatGPT and co-pilot asking first if it knew of a specific paper which it confirmed and then to derive simple results from which it was completely incapable of. I know this paper is not widely referenced (ie few known results in the public domain) but has been available for over 15 years with publicly accessible code written by humans. The training set was so sparse it had no ability to "understand" or even regurgitate past the summary text which it listed almost verbatim.
[ Parent | on HN ]
Re:OpenAI funded FrontierMath Benchmarks and had access to t... by Vecr on Sunday January 19, 2025 @09:46PM (#42764471)
It is known that current models have terrible sample efficiency. I've been told that it's better than I thought it was, but it still isn't good.
[ Parent | on HN ]
Re:OpenAI funded FrontierMath Benchmarks and had access to t... by sitkack on Sunday January 19, 2025 @09:48PM (#42764479)
This all smells like the OpenAI CEO's MO. Stupid drama for stupid reasons.
[ Parent | on HN ]
Re:OpenAI funded FrontierMath Benchmarks and had access to t... by KeplerBoy on Monday January 20, 2025 @09:14AM (#42768920)
It doesn't need to be smart to be useful. A lot of the kind of work I do seems to be in the training set.
[ Parent | on HN ]
Re:OpenAI funded FrontierMath Benchmarks and had access to t... by shadowfox on Monday January 20, 2025 @09:57AM (#42769365)
I don't think the OP is talking about usefulness at all, that is on a completely different dimension I would say.
[ Parent | on HN ]