Yeah I think a lot of people think RLHF is a tool for increasing accuracy but it really is training it to be convincing.

Which should be rather obvious if you understand that it's basically a GAN. You have a discriminative model who's objective function is based on Justice Potter's description of porn: I know it when I see it. If you ask what types of errors might emerge from such a formulation I think it becomes rather obvious.

Which isn't to say that the tools aren't useful. But I have to add this fact because many people conflate any criticism with being dismissive of the technology. But technology is about progress, not completion. Gotta balance criticism and optimism. Optimism drives you and criticism directs you.

I think this is a more realistic notion of what AI danger is rather than the X-risk. It's our tendency to trust things given a certain means of presentation. There are certain things where error is okay, but many things where even a small error is huge (like the OP is mentioning). The danger is not so much in the tool itself but us using the tools in a lazy manner. It isn't unique to ML/AI, but ML/AI uniquely are better at masking the errors. It's why I dislike the AI hype.