Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end.
Given global forces wanting models to be smarter, training budget / intelligence to be cheaper, etc. In the real world with competitive constraints, I don't think it's a matter of "if", but rather "when" the forbidden fruit is eaten.
Re:Hybrid Reasoning Agents: LLM agents that think in latent ...
by addag on Monday October 19, 2026 @10:48AM
(#50153168)
Indeed it is not faithful, but it is ones of the less bad ones we have for now. And removing it definitely goes towards the wrong direction in terms of intepretability.
Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end.
Given global forces wanting models to be smarter, training budget / intelligence to be cheaper, etc. In the real world with competitive constraints, I don't think it's a matter of "if", but rather "when" the forbidden fruit is eaten.