I get that AI basically is a problem solving machine that might eventually adapt to solve generic problems and thus reach the ability to break out of its box. But so what? Even if it manages to do all that, doesn't make it sentient. Doesn't make it a threat to mankind. Only when sufficiently motivated, or in actuality, when we project our humanity on it does it become scary and dangerous.

If we ever seriously try to create an artificial conscience it might need to be embodied, because we are embodied and seem to have evolved this due to evolution, which is a pretty physical process. Looking at it from this perspective one might say that if we keep the AI in its box, it will never have a need for conscience and therefore will never gain it.

This reply puzzles me somewhat. The first half doesn't seem to relate to the post it's replying to.

How aware are you of the main points around AI X-risk like orthogonality? Or how an optimising process that makes efficient use of information does not need (in theory) to have "conscience" or "sentience" to be lethal?

And on a separate tangent, are you aware people are already making primitive agents by connecting LLMs (given an initial prompt) in a loop with the result of feedback from its actions?

Hmm, I see someone has not played enough universal paperclips.