> And there is no reason to think that AGI would have desire.

The entire point of utilizing this tool is to feed it a desire and have it produce an appropriate output based upon that desire. Not only that, it's entire training corpus is filled with examples of our human desires. So either humans give it desire or it trains itself to function based on the inertia of "goal-seeking" which are effectively the same thing.

There is evidence that as LLMs increase in scale, their preferences become more coherent, see Hendrycks et al 2025, summarizer at https://www.emergent-values.ai/

A preference is meaningless without consciousness and qualia.

Consider a philosophical doomsday AI: it behaves exactly as if it has a desire to harm you (meaning that it does!), but it doesn't actually want to do that.

Or we can just drop all this sophistry nonsense.

Look, i understand being concerned about robot and self-driving safety. AI is a nebulous term and we don't need to restrict anyone training AIs.