Capitalism is not the world. Or doesn't have to be.

You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).

So mark it as a saturated task on the next scale up?

I'm not sure how the world ending follows from this?

I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).

We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.