I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).

We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.