"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.
I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)
… nobody knows how to actually make an AI that would do CEV.
It’s a big pretend game.
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures which disagrees with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"
e.g. 1 contradiction in a formal proof is sufficient to negate 9999 logically valid steps.
There’s no magical way for it to automatically become coherent.
An agent whose outcomes become bad in 100 years can, similarly, be described as "less wrong" than one whose outcomes become bad in 1 week.
A formal proof with 1 contradiction and 9999 correct steps is just as false as a formal proof with 10 contradictions and 9990 correct steps.
There is no “less wrong” or “more wrong” in logic, by definition.
i.e. There is no safe lower bound threshold of “wrongness” that can partially salvage the correct steps. Or conversely, 1 contradiction is already sufficient to prove anything at all.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
[.] https://david.robinsonian.com/assets/pdf/dgr_cv.pdf