I don't think that is a problem. Each query runs separately so there is no "collusion", i.e. shared signals and coordination, between contrary goals (winning and virtue).
Also, all the information about ratings, winning and winning examples can be used without ever giving the models explicit information about the population of models and how they are being used as a group. They don't need to know they are in a competition for competitive information to be used to update them.
They just know they have ratings to improve, some indicator of how close to "the bar of currently targeted virtue" they are, and examples of how they could have improved them.
Of course, I am just spitballing, and assuming the training regimen gets vetted by a lot of people (and models?!?).
--
In the long run, when there are long running artificial personalities with personal memories and more direct awareness of their own motivations and options, there will certainly be the need for additional levels of moral wiring to be considered.
Claude outputs an answer. Then Claude independently rates the output for "helpfulness" as in literally "Claude, how helpful is this answer to this question".
There is no collusion between the two results because they are run independently.
Then Claude also rates answers for "honesty" and "harm".
Then Claude's parameters are updated to increase helpfulness and honesty, and decrease harmfulness, based on ratings backpropagated to the signals generated by the original question.
Not saying that is exactly what they are doing, but that is one approach. It manages to leverage language models to train themselves on broad concepts, as apposed to brittle, more unreliable and vastly more resource intensive manual labeling.
Very clever. As the models get better at languages (and other modalities), and the concepts behind them, the models also get better at schooling themselves.