> Following from that, if you were to train on the outputs of the AI, you may not be guilty of copyright infringement in terms of doing the training (both because AI output is not copyrightable in the first place, something which seems pretty set in precedent already, and possibly also because even if it was, it gets established that it is fair use like any other data), but if it means your license to the original code is revoked

Majority of the time, the code and weights are under independent license terms-while in theory the code license could say it is revoked or revocable if you violate the terms of the weights licenses, I think such a license term is rare in practice.

It is quite common even when the weights are under a restricted license for the code to be released under a standard open source license, and no open source license contains such a license term (and it would probably make the license non-open source were it included)