News for hackers. Stuff that matters.
Re:Open models by OpenAI by highfrequency on Tuesday August 05, 2025 @05:01PM (#44804226)
I would guess the “secret sauce” here is distillation: pretraining on an extremely high quality synthetic dataset from the prompted output of their state of the art models like o3 rather than generic internet text. A number of research results have shown that highly curated technical problem solving data is unreasonably effective at boosting smaller models’ performance.

This would be much more efficient than relying purely on RL post-training on a small model; with low baseline capabilities the insights would be very sparse and the training very inefficient.

Re:Open models by OpenAI by asadm on Tuesday August 05, 2025 @05:10PM (#44804354)
> research results have shown that highly curated technical problem solving data is unreasonably effective at boosting smaller models’ performance.

same seems to be true for humans

Re:Open models by OpenAI by throw310822 on Tuesday August 05, 2025 @06:03PM (#44805004)
Yes, if I understand correctly, what it means is "a very smart teacher can do wonders for their pupils' education".
Re:Open models by OpenAI by tempaccount420 on Tuesday August 05, 2025 @05:30PM (#44804616)
Wish they gave us access to learn from those grandmother models instead of distilled slop.
Re:Open models by OpenAI by ashdksnndck on Tuesday August 05, 2025 @06:19PM (#44805171)
It behooves them to keep the best stuff internal, or at least greatly limit any API usage to avoid giving the goods away to other labs they are racing with.
Re:Open models by OpenAI by saurik on Tuesday August 05, 2025 @10:53PM (#44807095)
Which, presumably, is the reason they removed 4.5 from the API... mostly the only people willing to pay that much for that model were their competitors. (I mean, I would pay even more than they were charging, but I imagine even if I scale out my use cases--which, for just me, are mostly satisfied by being trapped in their UI--it would be a pittance vs. the simpler stuff people keep using.)