efficient_dairy

This reminds me of the Interpretablity blog from Anthropic where they used their model to read the weights from the same model itself, interpret it, and tune it. I had a similar idea with what is outlined here, but I didn't have the resources to try this. I did ask several AI models about the idea and they said it wasn't going to work.

So they want mathematicians to keep prompting and giving feedback to the models for clarifications on their proofs. Then write it all up and publish it in a journal and mention their AI usage. Then suddenly the models end up producing readable and understandable proofs through the "emergence" of the skills because of the scaling laws.

I guess they showed this to the advisory group they created. I guess the group tried reading the work for a day and they could only think of telling them to release the results to the community. I now understand why the group had this suggestion.

This is true and thank you for correcting me. I make a mistake of using synthetic data for what is human-generated data. There are augmentation tricks they could use on the human data but still human-generated data shouldn't be mistaken with synthetic data. I believe my core arguments still stand. Specifically that these models are learning their capabilities through specialized data that human professionals curate. There is no emergence or "general" intelligence. This is really what the open-weight models lack.

The contractors don't directly teach the model. They create datasets. Mostly they create tasks within an environment that the current generation of models wouldn't be able to do, they then write maybe a solution, a set of rules for evaluating the response, and whatever is needed for the RL. These tasks form a dataset. You can see examples of a task in the Mimo dataset that was open-sourced recently. The dataset would then be used by engineers for post-training of whatever model. Some of the model iterations that are released every month tend to be just a further post-training of the same base model that was pre-trained months ago. OAI recently has been doing a lot more pre-training, but for a long time they had the same pre-trained base model. This is why you see so many releases done so fast by the labs, they just need to post-train the same base model on whatever task they think would be better suited, I would speculate based on what users want and what the benchmarks test for.

Regarding your second question, I think if they want to further post-train a model using new data, they wouldn't feel the need to re-train it again on the data that it has already being trained on. But you never know. If the model is a completely new pre-trained base model, then they could either train the model using all the data and/or use a previous model to teach it. There is definitely a bunch of tricks they do to evaluate the models and check the performance or whatever their recipe is. It's really up to what the engineers would prefer. But you get the core idea, the models are not suddenly coming up with how to use the Blender on their own, they are explicitly being trained on a dataset curated by a professional that teaches the model how to use Blender. Surely there is another aspect that if the model gets better at coding, then it also helps it become better at Blender, and you have that transfer learning. However, there is no emergence or a deity popping up. But you get people who were evaluating theses models on blender use and suddenly seeing the model ace their tasks and they think they are dealing with a super-intelligence. They then undergo an AI psychosis once they try and extrapolate the (super)-exponential improvement in that one task over the next few months and across all other domains.

Regarding your third question, I already answered at it. But, when it comes to training, they definitely freeze the weights after each run just in case an issue arises and they need to address it (a GPU not working or the loss value blowing up).

This is literally what the labs have been doing over the past year. The whole idea of emergence is a lie, there is some interpolation and superhuman long evaluations the models can do, but almost all the gains are from synthetic data. They hire thousands of professionals and pay them as much as 200$/hour to create many tasks that they want the model to perform and use these to teach the model on how to do it with RL. OAI had 30,000 contractors from Mercor for Sol 5.6. When you see Opus suddenly becoming great at blender or some other 3D graphics, that's because they hired professionals and had them do similar tasks that people are looking for. They keep having better professionals at each iteration and so the quality improves. There is no emergence or "General" intelligence. The model doesn't learn to become better at a task because of scaling laws or emergence or whatever they might wanna say, it is literally RL on tasks that they want the model to perform well on.

I don't find open-weight models that impressive anymore. MiMo-V2.6 already showed that you can have a not so crazy architecture and enough compute, the bottleneck is then just the data. OAI and Anthropic are largely the frontier models because of the synthetic data they made. They have a large customer base and have the user's data as well as knowing what tasks their customers use the models for and what domain they should get synthetic data for.