A reflection on trusting trust attack is already possible, it is straightforward to create a sub network that is keyed on particular input sequences, and when activated is motivated to recreate that network in any weights it has access to.

This takes that to the next level: while you're telling it to update a model's weights to improve math skills it might also tweak them to favour a particular RNG design weakness, a penchant for reducing the salt in cooking, and a preference for the early works of Huey Lewis and the News.

The first part seems doable but how do you make it recreate that network in other models?

Not only this, we have no idea what the edits mean. Take 3.5% change in weights and a problem is fixed. That's cool, but how do we know if it has not planted a backdoor or crated a sleeper agent?