News for hackers. Stuff that matters.
Re:Claude 3 model family by sebzim4500 on Monday March 04, 2024 @05:19PM (#39596927)
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
Re:Claude 3 model family by whymauri on Monday March 04, 2024 @08:15PM (#39598260)
Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
Re:Claude 3 model family by nopinsight on Monday March 04, 2024 @10:33PM (#39599129)
Most likely, it’s less generally smart than the top 2-4% of US high school students.

It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance.

Since the test questions were unseen, the result still suggests the person has some intelligence though.

Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.