Good points. Some people claim the current generation of LLMs have "intelligence" but in reality it is basically a habit- that let's them predict words.

We can try to "teach" an AI system survival instinct, but then it'll most likely iterate/evolve into something that wipes us out. I just don't see them living in symbiosis with us.