News for hackers. Stuff that matters.
Re:Claude 3 model family by nopinsight on Monday March 04, 2024 @01:35PM (#39594110)
AMC 10, AMC 12 (2023) results in Table 2 suggest Claude 3 Opus is better than the average high school students who participate in these math competitions. These math problems are not straightforward and cannot be solve by simply memorizing formulas. Most of the students are also quite good at math.

The student averages are 64.4 and 61.5 respectively, while Opus 3 scores are 72 and 63.

Probably fewer than 100,000 students take part in AMC 12 out of possibly 3-4 million grade-12 students. Assume just half of the top US students participate, the average score of AMC would represent the top 2-4% of US high school students.

https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...

Re:Claude 3 model family by usaar333 on Monday March 04, 2024 @04:09PM (#39596139)
Interestingly, math olympiad problems (using ones I wrote myself years ago so outside training data) seem to be better in Claude 3.

Almost everything else though I've tested seems better in GPT-4.

Re:Claude 3 model family by sebzim4500 on Monday March 04, 2024 @05:19PM (#39596927)
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
Re:Claude 3 model family by whymauri on Monday March 04, 2024 @08:15PM (#39598260)
Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
Re:Claude 3 model family by nopinsight on Monday March 04, 2024 @10:33PM (#39599129)
Most likely, it’s less generally smart than the top 2-4% of US high school students.

It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance.

Since the test questions were unseen, the result still suggests the person has some intelligence though.

Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.