Additionally, I wonder if an alternate dataset is provided based on model size as to not run into issues with model forgetting.
(There are 1000/3000/1000 problems in the test set in each level).
It’d be great if someone from Anthropic provides an answer though.
PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract.
https://twitter.com/idavidrein/status/1764675668175094169
GPQA: A Graduate-Level Google-Proof Q&A Benchmark https://arxiv.org/abs/2311.12022
(Edit: better to use the original script but set PYTHONUTF8=1 before running)
I do not have a PhD, but in areas I do have expertise, you really don't have to push these models that hard to before they start to break down and emit incomplete or wrong analysis.
E.g., if you ask a chemistry PhD something about genetics or astrophysics, you're more likely to get a correct answer from the model. Which is pretty interesting IMO.
LLMs is more like programming than human intelligence, they need to program in the solution to these riddles very much like we did expert systems in the past. The main new thing we get here is natural language compatibility, but other than that the programming seems to be the same or weaker than old programming of expert systems. The other big thing is that there is already a ton of solutions on the web coded in natural language, such as all the tutorials etc, so you get all of those programs for free.
But other than that these LLMs seems to have exactly the same problems and limitations and strengths as expert systems. They don't generalize in a flexible enough manner to solve problems like a human.
i feel like there's some theory missing here. something along the lines of "when do you cross the line from translating or painting with related sequences and filling in the gaps to abstract reasoning, or is the idea of such a line silly?"
The student averages are 64.4 and 61.5 respectively, while Opus 3 scores are 72 and 63.
Probably fewer than 100,000 students take part in AMC 12 out of possibly 3-4 million grade-12 students. Assume just half of the top US students participate, the average score of AMC would represent the top 2-4% of US high school students.
https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...
Almost everything else though I've tested seems better in GPT-4.
It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance.
Since the test questions were unseen, the result still suggests the person has some intelligence though.
Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.
It was a complete and total disaster for my normal workflows. Compared to ChatGPT4, it is orders of magnitude worse.
I get that people are impressed by the benchmarks, and press released, but actually using it, it feels like a large step backward in time.
Interestingly, no other top models have published results on this benchmark.
Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...
Table 1: Evaluation results (more datasets than in the blog post) https://twitter.com/karinanguyen_/status/1764666528220557320
APPS dataset: https://huggingface.co/datasets/codeparrot/apps
APPS dataset paper: https://arxiv.org/abs/2105.09938v3