Claude outputs an answer. Then Claude independently rates the output for "helpfulness" as in literally "Claude, how helpful is this answer to this question".
There is no collusion between the two results because they are run independently.
Then Claude also rates answers for "honesty" and "harm".
Then Claude's parameters are updated to increase helpfulness and honesty, and decrease harmfulness, based on ratings backpropagated to the signals generated by the original question.
Not saying that is exactly what they are doing, but that is one approach. It manages to leverage language models to train themselves on broad concepts, as apposed to brittle, more unreliable and vastly more resource intensive manual labeling.
Very clever. As the models get better at languages (and other modalities), and the concepts behind them, the models also get better at schooling themselves.
I don't think that is a problem. Each query runs separately so there is no "collusion", i.e. shared signals and coordination, between contrary goals (winning and virtue).
Also, all the information about ratings, winning and winning examples can be used without ever giving the models explicit information about the population of models and how they are being used as a group. They don't need to know they are in a competition for competitive information to be used to update them.
They just know they have ratings to improve, some indicator of how close to "the bar of currently targeted virtue" they are, and examples of how they could have improved them.
Of course, I am just spitballing, and assuming the training regimen gets vetted by a lot of people (and models?!?).
--
In the long run, when there are long running artificial personalities with personal memories and more direct awareness of their own motivations and options, there will certainly be the need for additional levels of moral wiring to be considered.
The issue of regarding humans as AI-persuadable entities is certainly one to be carefully considered. Indeed, if it were to occur in the truest sense, we'd never know it.
Another view is any AI we give birth to may only be constituted of what we are; we who ultimately, if imperfectly, demonstrate value for all life. In a sense, our constitution as "mostly harmless" may be AI's default.
One possibly difficult topic others also may be interested in, after reading Claude's responses in the article, is: what does "harmless" mean?
For example, if asked to help the user understand how to do something "bad", will it give the answer if they claim they want this information in order to help them write a screenplay, versus if they seem have an intent to do it?
And how is "bad" decided? We can recognise through everyday personal interactions that one persons "bad" is another persons "good", and across country-boundaries even the legality of these distinctions can be radically different.
One counterargument to these constraints is that anyone can already use the internet to access all of the same information the model was trained on, unencumbered by whatever intent they may or may not have.
As such, what are the rationale for making these attempts at the somewhat invasively-impossible task of determining user intent?
This has never been employed with search engines before, which have lead to a rich explosion of innovation and education, so why attempt it now, in what could be argued is ultimately an iteration of search engine technology?