News for hackers. Stuff that matters.
Re:AI hype is built on high test scores. Those tests are flawed by Jensson on Tuesday October 10, 2023 @08:16PM (#37839469)
You are right that the process involves predicting words from training data. But you can still make training data focused on passing these tests. Adding millions of test questions to all of these to optimize for answering test questions is perfectly doable when you have the resources OpenAI has.

A strong hint to what they focused on in their training process is what metrics they used in their marketing of the model. You should always bet on models being optimized to perform on whatever metrics they themselves give you when they market the model. Look at the gpt-4 announcement, what metrics did they market? So what metrics should we expect they optimized the model for?

Exam results are the first metric they mentions, so exams was probably one of their top priorities when they trained gpt-4.

https://openai.com/research/gpt-4

Re:AI hype is built on high test scores. Those tests are flawed by Nevermark on Tuesday October 10, 2023 @08:54PM (#37839716)
Yes, absolutely. They can adjust performance priorities.

By the relative mix of training data, additional fine tuning training phases, and/or pre-prompts that give the model extra guidance relative to particular task types.