Re:I Quit OpenAI Because Its Culture Is Broken
by MichaelZuo on Monday October 05, 2026 @12:07PM
(#49966719)
Even if it’s 99% coherent, and only 1% incoherent… then the end result is still incoherent?
e.g. 1 contradiction in a formal proof is sufficient to negate 9999 logically valid steps.
There’s no magical way for it to automatically become coherent.
Re:I Quit OpenAI Because Its Culture Is Broken
by ben_w on Monday October 05, 2026 @12:32PM
(#49967051)
Automatic? No. But then, under Bayesian updating, it takes infinite evidence to reach P = [0, 1]
An agent whose outcomes become bad in 100 years can, similarly, be described as "less wrong" than one whose outcomes become bad in 1 week.
Re:I Quit OpenAI Because Its Culture Is Broken
by MichaelZuo on Monday October 05, 2026 @07:15PM
(#49972072)
Are you confused?
A formal proof with 1 contradiction and 9999 correct steps is just as false as a formal proof with 10 contradictions and 9990 correct steps.
There is no “less wrong” or “more wrong” in logic, by definition.
i.e. There is no safe lower bound threshold of “wrongness” that can partially salvage the correct steps. Or conversely, 1 contradiction is already sufficient to prove anything at all.
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures which disagrees with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"