"legacy code as a service" - that's apt. But would they be better if they trained exclusively on 'good code'? I know I don't know the answer to that question and I get the feeling that few people actually understand how they work enough to feel comfortable with asserting that to be true.

Yeah, I still wouldn't trust them if they were training on more good code, either. I think I understand enough of how they work to believe that even given plenty of good code they won't be able to learn the parts that make good code truly good. That's where I start into poetry metaphors and that the best code is not just concerned with poetry forms (the rhythm and meter required by the language) nor the literal meaning (the compiler output) but also the human elements of the poem such as the creative storytelling and multiple levels of metaphors. I cannot see the current technology getting good at those human parts of the poetry, no matter how good they get at the literal and the form.

The problem there is the _large_ language model part, the density and the reinforcement of the weights. There's far less good code in the world. ;) These things emit code as well as I do, such as they do, only because they've inhaled essentially the totality of "code in general", not artisanal code.