News for hackers. Stuff that matters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @04:32PM (#42475261)
There is a benchmark, NovelQA, that LLMs don't dominate when it feels like they should. The benchmark is to read a novel and answer questions about it.

LLMs are below human evaluation, as I last looked, but it doesn't get much attention.

Once it is passed, I'd like to see one that is solving the mystery in a mystery book right before it's revealed.

We'd need unpublished mystery novels to use for that benchmark, but I think it gets at what I think of as reasoning.

https://novelqa.github.io/

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by loxias on Friday December 20, 2024 @07:55PM (#42476628)
NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that.

LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :)

https://machinelearning.apple.com/research/gsm-symbolic

https://arxiv.org/pdf/2410.05229

Paper came out in October, I don't think many have fully absorbed the implications.

It's hard to take any of the claims of "LLMs can do reasoning!" seriously, once you understand that simply changing what names are used in a 8th grade math word problem can have dramatic impact on the accuracy.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by meta_x_ai on Friday December 20, 2024 @04:50PM (#42475414)
Looks like it's not updated for nearly a year and I'm guessing Gemini 2.0 Flash with 2m context will simply crush it
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @05:34PM (#42475771)
That's true. They don't have Claude 3.5 on there either. So maybe it's not relevant anymore, but I'm not sure.

If so, let's move on to the murder mysteries or more complex literary analysis.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by latency-guy2 on Friday December 20, 2024 @08:11PM (#42476706)
> I'd like to see one that is solving the mystery in a mystery book right before it's revealed.

I would think this is a not so good bench. Author does not write logically, they write for entertainment.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @08:23PM (#42476766)
So I'm thinking of something like Locked-room mystery where the idea is it's solvable, and the reader is given a chance to solve.

The reason it seems like an interesting bench, is it's a puzzle presented in a long context. Its like testing if an LLm is at Sherlock Holmes level of world and motivation modelling.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by rowanG077 on Friday December 20, 2024 @06:20PM (#42476115)
Benchmark how? Is it good if the LLM can or can't solve it?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by CamperBob2 on Friday December 20, 2024 @04:37PM (#42475302)
Does it work on short stories, but not novels? If so, then that's just a minor question of context length that should self-resolve over time.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @04:40PM (#42475327)
The books fit in the current long context models, so it's not merely the context size constraint but the length is part of the issue, for sure.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by usaar333 on Saturday December 21, 2024 @02:09AM (#42478007)
That's an old leaderboard -- has no one checked any SOTA LLM in the last 8 months?