> I find it unusual if we did create an intelligent class of beings, who are legit intelligent, smarter than us, why would they work for Sam Altman or Mark Zuckerberg? They'd do whatever the fuck they want...

That's anthropomorphisation: the AI we have now only "want" to maximise whatever reward function we give them. The alignment problem is that any reward function is only a rough approximation of what we want, so the machines may well rush off and turn the universe into paperclips or tile it with molecular-scale smiley emoji because that's what we told them to "want" without really understanding what we were telling them.

But even then, regardless of IQ (which itself isn't a great description of even just human intelligence), any given AI may or may not have anything we'd consider to be qualia, an "experience" of getting this reward, because that's the hard question of consciousness which isn't solved.

The only thing I can be sure of is that we definitely have no idea where it's all going.

The “whatever the fuck they want” part is what I’ve always wondered about. What would a non-biological (super?) intelligence, stripped of any physiological desires, even want? Would they experience anything like emotion? Would they care to preserve themselves, or would they find this meaningless existence boring and simply decide to exit 0?

All I can do is agree with the other commenters that it’s clear that no one knows where anything is headed.

> Another thing I thing is strange is, if we had really very / sueper intelligent AI, why would it exist in physical, robot form ? Like, why not just be a light orb that lives in space or something?

Well, someone will have to plug cables and maintain solar panels or batteries or whatnot. Maybe the AI overlords will be ethereal neural networks in the cloud, but some kind of mechanical being (humans or robots) will need to do some groundwork.