Re:AI hype is built on high test scores. Those tests are flawed
by Nevermark on Tuesday October 10, 2023 @08:54PM
(#37839716)
Yes, absolutely. They can adjust performance priorities.
By the relative mix of training data, additional fine tuning training phases, and/or pre-prompts that give the model extra guidance relative to particular task types.
A strong hint to what they focused on in their training process is what metrics they used in their marketing of the model. You should always bet on models being optimized to perform on whatever metrics they themselves give you when they market the model. Look at the gpt-4 announcement, what metrics did they market? So what metrics should we expect they optimized the model for?
Exam results are the first metric they mentions, so exams was probably one of their top priorities when they trained gpt-4.
https://openai.com/research/gpt-4