That's why it's called Vision Pro.
And Apple today added support for spatial personas allowing developers to create the experience you are talking about:
https://www.uploadvr.com/apple-vision-pro-spatial-personas-l...
I believe the only way Apple will make that happen is a radical change in attitude toward developers. They would need to court developers of games and owners of content and negotiate -- generously -- on terms, instead of dictating terms and clinging to the 100% control and 30% revshare they feel so deserving of.
We'll see in a couple years if I'm right!
* Honestly though, still, even at that price, can you imagine most people being eager to add another almost-$2000 device to their lives that doesn't replace another one, unless the device does some serious life-improving stuff? And I don't see how the price gets any lower than that, even if Apple budged on margins which is very out of character.