Yes but I'm saying _don't_ add a 3rd dimension in any but the most basic sense. Just give me what is effectively a very large 2D workspace, as if I was staring at one or more nice big monitors. No fancy metaphors, no gestures - with Hololens I can still touch type and use my existing mouse.
I am a curmudgeonly luddite, obviously, but I would pay thousands of dollars just to have a multi monitor setup that required zero effort and worked on the move (to the extent that I'd be willing to go out with the equivalent of a Segway attached to my head).
If _that's_ still impossible (i.e. it's impossible to accurately and quickly orient the device in space, even in the controlled environment of my desk, or it's just not possible to display text that's nice to read on these devices) then I don't see the point of even attempting the more esoteric stuff as anything but pure research. Clearly several multi-billion dollar companies disagree, so I'm happy if all this comes to pass either way.
To get what you just called a baby step is an act of barn raising. By thousands of dedicated professionals, tinkerers, artists, scientists, and large corporations. If we could have willed the “baby step” into being before now we would have. Actually many people have, but then quickly notice what’s lacking and get back to raising the barn.
Here are a couple projects I pulled with a quick google. I have a bitbucket wiki with dozens of similar projects, some in AR, VR, projection mapped, single user parallax, with gesture tracking, with custom controllers, etc... This is definitely one of those problems where it's easy to imagine so people imagine it's easy. Maybe we need you to help! Get a spatial display of some sort and lets get to work!
https://github.com/SimulaVR/Simula
https://github.com/letoram/safespaces