Why are you assuming this hypothetical intelligence will have any motivations beyond the ones we give it? Human's have complex motivations due to evolution, AI motivations are comparatively simple since they are artificially created.

A true super intelligence would need the ability to evolve, and would probably evolve its own wants and needs.

It would still need an objective to guide the evolution that was originally given by humans. Humans have the drive for survival and reproduction... what about AGI?

How do we go from a really good algorithm to an independently motivated, autonomous super intelligence with free reign in the physical world? Perhaps we should worry once we have robot heads of state and robot CEOs. Something tells me the current, human heads of state, and human CEOs would never let it get that far.

Someone will surely set its objective for survival and evolution.

That would be dumb and unethical but yes someone will do it and there will be many more AIs with access to greater computational power that will be set to protect against that kind of thing.

> A true super intelligence would need the ability to evolve, and would probably evolve its own wants and needs.

Why? This seems like your personal definition of what a super intelligence is. Why would we even want a super intelligence that can evolve on its own?

Any intelligence on the level of average human and for sure on the level above it will be able to learn. And learning means it will acquire new motivations, among other things. Fixed motivation thing is simply a program, not AI. A very advanced program maybe, but ultimately just a scaled up version of the stuff we already have. AI will be different, if we will create it.

> And learning means it will acquire new motivations

This conclusion doesn't logically follow.

> Fixed motivation thing is simply a program, not AI

I don't agree with this definition. AI used to be just "could it solve the turing test". Anyway, something with non-fixed motivations is simply just not that useful for humans so why would we even create it?

This is the problem with talking about AI, a lot of people have different definitions of what AI is. I don't think AI requires non-fixed motivations. LLMs are definitely a form of AI and they do not have any motivations for example.

Disclaimer - I don't consider current LLMs as (I)ntelligent in the AI, so when I wrote AI in the comment above it was equivalent to the AGI/ASI as currently advertised by LLM corpos.

Consciousness, intelligence, and all these other properties can be and are mutually exclusive. What will be most useful for humans is a general intelligence that has no motivation for survival and no emotions and only cares about the goals of the human that is in control of it. I have not seen a convincing argument that a useful general intelligence must have goals that evolve beyond what the human gives it and must be conscious. What I have seen are assertions without evidence, "AI must be this way" but I'm not convinced.

I can conceive of an LLM enhanced using other ML techniques that is capable of logical and spatial reasoning that is not conscious and I don't see why this would be impossible.