Interestingly, math olympiad problems (using ones I wrote myself years ago so outside training data) seem to be better in Claude 3.
Almost everything else though I've tested seems better in GPT-4.
Re:Claude 3 model family
by sebzim4500 on Monday March 04, 2024 @05:19PM
(#39596927)
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
Re:Claude 3 model family
by nopinsight on Monday March 04, 2024 @10:33PM
(#39599129)
Most likely, it’s less generally smart than the top 2-4% of US high school students.
It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance.
Since the test questions were unseen, the result still suggests the person has some intelligence though.
Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.
The student averages are 64.4 and 61.5 respectively, while Opus 3 scores are 72 and 63.
Probably fewer than 100,000 students take part in AMC 12 out of possibly 3-4 million grade-12 students. Assume just half of the top US students participate, the average score of AMC would represent the top 2-4% of US high school students.
https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...