we are still involving humans in the loop, it helps them acquiesce

Because "what to prove" is fundamentally bound in a value judgment, not merely the mechanistic application of reasoning.

Because humans are the ones who care? At some point, you have to make the wish, or the genie is not going to do anything.

Because that’s a design question. The AI has to know what it’s actually supposed to be building.