Kind of hard to believe in recursive self improvement, every time I tried setting an LLM to loop on itself to "recursively" improve code towards a goal it always descends into utter garbage. I think the main challenge is probably knowing exactly what you want to get, and then having a training landscape that is smooth enough so you can climb there? I guess these models can probably iterate their architecture to max out existing benchmarks but at some point they will essentially just do p-hacking as the model that improves the model will invariably leak the validation data into the training process on the meta level, or so I would assume. And if your goal is to create a model that can compete with human intelligence in math then how would you even come up with a scoring function for that, will you let the model do multiple years of fundamental math research and then see if it produces something that is novel and interesting? And how do you score if it's like 99 % there, 50 % or 10 %? From my naive understanding that's probably the main challenge, these models can only optimize if you have a clear measurement of goal achievement that can be climbed to, if there's only a very far fetced goal without clear intermediate success states I don't see how the current generation of models should get there.