This is one of those concepts that is obviously going to be great when it's ready for prime-time, but just isn't there yet.

In a generation or two when the headsets are lighter (can be worn for a day without leaving an impression on your face as the current headsets do) and higher resolution (can actually replace a multi-monitor setup in number of pixels), I strongly believe this is going to be a killer app that could easily dwarf the gaming market for VR headsets.

An interesting corollary of this is that now is probably a good time to start researching novel input devices, since there is an opportunity to create a completely new UI; if you can touchtype fluently then VR works fine, but if not, it's going to be challenging to use.

As an example, Facebook is working on direct neural input devices:

https://www.roadtovr.com/facebook-acquires-ctrl-labs-develop...

And I have had interesting conversations with neuroscientists about a bunch of intriguing ways that you could hook up an input device to the somatic nervous system. For most of these you'd need to learn a different set of "movements" than typing, so perhaps there's not an immediate win vs. just requiring users to learn to touchtype, but the benefits of being able to "type" characters in a UI without a keyboard are pretty clear.

> if you can touchtype fluently then VR works fine, but if not, it's going to be challenging to use.

I am of the belief that if you are trying to increase your productivity whilst mobile with small/light stuff, nothing weighs less or takes up less space in your luggage than some touch typing training software.

I can’t believe that there are people who type as much as some of us do who haven’t bothered to learn to type. It’s an essential skill.

Funnily enough, I learned to touch type because tables at my uni were high enough to make typing uncomfortable.

Putting my Magic Keyboard on my lap was much better, but without being able to see, I couldn't type well.

Spent 10 minutes a day over the next couple weeks learning Colemak (easier than retraining myself on QWERTY). Being able to touch type numbers is great while programming.

@sneak The near-term dream goal/dream is installing something like Simula on a standalone VR headset, and pairing it with a bluetooth (perhaps even "pocket" bluetooth keyboard) to replace a laptop.

Touch typing + bluetooth keyboard + VR headset = insane productivity potential.

Do you have any light/portable BT keyboards you like? I somewhat loathe the iPad Pro’s keyboard cover but every light BT I’ve ever tried was plasticky loud garbage. Do you have any recommendations?

I've had reasonable succcess with this one: https://www.amazon.com/iClever-Bluetooth-Keyboard-Foldable-S...

I like its compact design (it fits in a pocket), and it's very satisfying to fold/unfold :] That said, I don't use it that often because I find smartphone screen sizes too small.

It arrived today. The number keys, and thus important keys like ! and @, are in the wrong places, and (this one is on me, I should have looked before buying) it is also lacking an ESC key, which is largely the point for me of buying another keyboard.

I have the iPad Pro with keyboard cover and I keep it in my bag most of the time even when I don't have a laptop, so that's the standard I'm trying to beat. It would be nice to have something I could pair to my phone (the largest phone Apple sells) to use a terminal when on the go, but I guess I'll just keep carrying my handbag with iPad for now. I hear Apple is going to improve the iPad Pro's keyboard this year.

Oh well.

> if you can touchtype fluently then VR works fine, but if not, it's going to be challenging to use.

Couldn't you just render the keyboard (+ finger positions I guess) inside the VR.

FWIW Simula (shown above) has a mouse & keyboard view. Here's a short video showing what it looks like: https://www.youtube.com/watch?v=D5c3Hfp8Hcw

Use chording! Chording is (maybe?) similar to sign language and so "glyphs" or "chords" could be recognized by camera or finger position sensors.

I think finger position sensors would be more ideal than camera recognition, since there are many times your fingers could become obfuscated behind other fingers or if you rotate your hand. RFID tags are cheap and can use passive power. I imagine with a bracelet transmitter and positioning device they could be tracked in real time and attached to the finger tips using a reusable adhesive. Should require minimal calibration and then would allow for new touch inputs for almost any device that is calibrated along with it. Could even make regular monitors touch screen.

@theptip I understand your skepticism, but people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

RE headset size, I agree that today they are much too bulky. But they are still good enough to "live in the future" and get serious work done in.

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

Please don't post the same thing over and over.

But I don't know about "significantly underrate" either. The screens in the video you linked are 93x44 characters. It's certainly usable, but just my main screen holds 8 windows that size with a generous font and room to spare.

As others have said, current tech isn't quite there yet to replace screens in meatspace, but I think it's obvious to infer what general computing may be like 10-20 years down the road, if progress continues at a steady pace:

Laptops and tablets would no longer be distinct factors for mobile, because they'll be replaced by always-worn high quality AR glasses, which run off a phone in your pocket (as Apple's glasses are rumored to be) and project virtual screens into your view of the real world. The AR space may be shared by multiple users for public signage, advertisements or virtual pets and fashion, à la the anime Dennou Coil. [0]

Physical keyboards/mice/trackpads will probably still be around because I haven't seen anything to fully replace the need for tactile feedback, unless someone comes out with sensory gloves.

At home we would put our phones into a docking station for more processing power and to connect to a giant physical screen and sound system.

[0] https://myanimelist.net/anime/2164/Dennou_Coil

Rainbows End by Vernor Vinge constructs a very good description of such a world. I see it coming more and more to life with each passing year. If you've not read it, definitely worth a read.

@Razengan

RE BCIs: Totally agree. We need to figure out a way to allow people to "think-type" 20,000 distinct inputs. This is how we can totally escape the keyboard.

RE VR: I understand your skepticism, but people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

> At home we would put our phones into a docking station for more processing power and to connect to a giant physical screen and sound system.

I feel like we got so close to that with Continuum on Windows Phone, but it was far enough away that it never took off. Drop your phone into a USB-C dock, and you'd get 1080p/1200p video out with a Windows 10-esque UI (desktop, Start menu), keyboard/mouse support, plus an additional USB-A. In practice though, the limitations (range of available apps for WP was mediocre, you couldn't run apps side-by-side on the monitor, some apps didn't implement a full screen UI) made it frustrating.

High-end Samsung phones do this with Dex, though I have only used it once. The app limitations make it nearly useless and. novelty. If anyone can pull it off it would be Apple with some sort of OSX VM.

See also the motorola atrix, samsung dex, maruOS, ubuntu phone, ... .

Or the other way around with using tablets, phones, chromebooks etc. as thin clients, so whatever you connect to the monitor does not really matter.

I'm looking forward to video feed wallpaper.

Totally.

Some day.

But between the bulkiness of any given HMD over a long period of time, and the low resolution of the HMD display, this is generally a novelty. In 2020 at least.

The more they iterate in the hardware the more plausible an option this will be. (And I'm excited for it! )

In terms of pixel count, isn’t the Vive only a 1080x1200 monitor? (per eye).

I’d like more information density without having to move my neck around.

The original vive is pretty outdated now. You would struggle to even read text, let alone want to replace your monitor with it. Currently the best you can get is 4k per eye which is going to allow you to read text ok but the quality will be something closer to a 720p screen since those pixels are spread out over a huge area.

Give it a couple years and there will probably be an 8k vr setup that might allow for decent desktop use. But even then I'm not sure most people would want to use such a setup since you would need a very powerful pc for it and no one else would be able to see your screen.

> and no one else would be able to see your screen.

I consider that a plus. I'm regularly flabbergasted by the things I'm able to see on screens in public transport.

In an office its probably not ideal when you need to show things to people.

Yeah that's the major issue with VR or AR monitors. If the screen fits into your whole field of view the headset screen has to be much higher resolution than the screen it's displaying.

Yes but you can't physically look at and process all the independent monitors anyways. So you have one physical monitor addressing n-virtual monitors.

One really bad physical monitor. At 16 pixels per degree, you're at one third of a plain old 96 dpi reference screen.

Imagine if those nice 24 inch monitors had a resolution of 700x400. You can have n of them, but that is not going to make it good. And your field of view is pretty small at the same time.

Edit: Oh, that's a Vive Pro. Reduce everything 30% more for a Vive.

I wouldn't do that, even on the vive pro, the text was way too blurry for my taste

@lousken

What VR Desktop did you use when you gave it a try?

I understand your skepticism, but people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

I don't own the vive pro personally, it was a demo we had at work during christmas.

I've tried tinkering with it for about an hour and half, mostly testing games. But also steam vr home and I was able to get to regular windows desktop view with that somehow (I don't remember how exactly how). And while the text was readable it surprised me how blurry it was. Same thing ingame, reading "books" was quite hard and I really had to move the object close to my face.

You can travel with infinite monitors but the resolution of them all combined can only be around 1600p total

@shawabawa

I understand your skepticism RE resolution, but people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

Maybe it’s my age, but my neck hurts just watching all that neck movement.

Even a Valve Index which has much higher resolution than a Vive isn't viable for this unless you want to virtually sit very close to low resolution panels.

@ebg13

I understand your skepticism RE resolution, but people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

Realizing the damage caused to my neck by wearing heavy headphones over prolonged periods of time at my desk, I'm going to have to pass on this.

I'm struggling to picture what you could possibly be doing with your posture to cause damage to your neck with even the heaviest headphones. Whatever it is that you're doing, the headphones are not your problem.

Suggestion for this: Sennheiser 6XX series are very lightweight, open cans (thus no extra pressure in the ear, but people may hear your music around a little bit). About ~$200 these days, and these used to be world's best a decade ago.

I can wear them all day long, even when the music or video ends I often don't even bother putting them off...

I just loathe the wire but sound quality is so good it's not even funny to compare with equally priced Bluetooth cans.

When I tried this a few years ago (shortly after getting my Vive), the text resolution and comfort just wasn't good enough for long periods of time. I'm curious if anyone has had more success with newer VR headsets

@robertfw

What VR Desktop did you use when you gave it a try?

With respect to today's VR climate: I think people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

Working in current-gen VR for me would be absolutely terrible.

- Low resolution. The best we have for consumers right now is the valve index, 1440x3200 resolution ~100 degree vertical FOV. A 24 inch display at a comfortable distance in such a headset would be at around 720x480 (480p), and a little blurry because of the projection.

- Fixed focus lens. You basically can't read anything close on today's headsets. Lens are focused so that they are sharp at infinity, which also mean they are blurry up close. Coding on virtual gigantic screens at a distance may be a good experience though.

- Touch typing is a must. In fact it is touch everything, because you will be blind to the real world.

- VR headsets can be uncomfortable when worn for hours at a time. Because of technical limitations (FOV and resolution), you tend to move your head a lot in VR, and doing so while wearing a heavy headset can make it even more uncomfortable.

Portability, for me, is the least of the problems, especially those that have "inside out" tracking. These things can fit in a small backpack.

@GuB-42

RE resolution: I understand your skepticism, but people significantly underrate the quality of text that can be achieved in VR, even on today's hardware. Simula in particular has put in a lot of work in text resolution optimization (something that earlier VR Desktops failed completely at). This makes me able to use Simula for hours at a time, even on the old HTC Vive (2016 hardware). I won't lie and say it is equally impressive as a 4K monitor is today, but it's definitely usable and allows you to garner the other benefits of VR Desktop (10x more screens, 10x the size, 10x the focus/work immersions).

RE touch typing: agree that touch typing makes VR Desktop more productive; however, we have an escape hatch in Simula in the form of a live Keyboard & Mouse view: https://www.youtube.com/watch?v=D5c3Hfp8Hcw

RE HMD comfort: Agreed that headsets need to get significantly lighter before they are usable for 16hr/day; however, I would argue that even today headsets are good enough to get serious work done in (on the order of 1hr/day, and perhaps as much as a few hours per day).

If you have an HTC Vive/Vive Pro/Valve Index, I can help you install Simula/give it a try. Contact me at george.w.singer [at] gmail [dot] com if you're interested.

Author here, definitely interested as it would make traveling much easier for me.

With a setup like that is there any fatigue after a few hours of usage ?

I've found that the Oculus Quest is not perfectly comfortable for multiple hours of uninterrupted use while gaming, but I've not tried it for sitting at a desk coding. YMMV, I would wager that moving around while playing requires it to be tighter, and therefore makes it less comfortable.

The Quest might be the best one to test for travel though; it's relatively cheap, has fairly good resolution, and though it normally works as a standalone system, you can also tether it over USB3 to your main machine's graphics card:

https://www.anandtech.com/show/15135/oculus-link-software-av...

@polote

The first time I used Simula, I did experience fatigue. After that, it seemed to go away. I tend to use it for a few hours at a time, so cannot comment beyond that, but it seems to work pretty well.

If you're interested in giving Simula a try, contact me at george.w.singer [at] gmail [dot] com. Right now, it's only compatible with the HTC Vive, HTC Vive Pro, or Valve Index. It should otherwise be buildable/runnable on all Linux distros.

Surprised you never mentioned this as part of your solution but I have a friend with a similar problem and he solved it by traveling with a single ultra wide monitor instead of 3.

>With a setup like that is there any fatigue after a few hours of usage ?

In games, no. With desktop usage, yes.

My eyes get taxed wearing my glasses while reading and I usually take them off to see my monitors. I feel like this solution would make my eyes very taxed.

VR systems project the image to near infinity to allow your eyes to relax.

Typically the VR lenses project the image to about 2m away.

But isn't simulation sickness[1] the fundamental limitation in the usage of VR for longer duration?

What good a VR desktop be if it can be only used 15-30 mins at a time.

[1]https://needgap.com/problems/7-simulation-sickness-in-vr-and...

That's only an issue if you're moving around in VR, particularly with artificial motion.

Sitting in one spot is generally fine for most people, as is so-called "roomscale" vr where it maps 1:1 to the area you're in.

I am able to be productive in VR Desktop for far longer than 15-30 min at a time (sitting stationary with mouse + keyboard).

Good to know. But when you stop using it, do you do it because you feel sick or for normal non-VR related reasons? i.e. is simulation sickness not a factor at all?

That looks cool, but I prefer each of my monitors to have a 4k resolution, not all of them combined to 1080p.

You only need 4k at the focal point you are concentrating on no? Pixels at places not being looked at are just wasted equipment.

Sounds and looks amazing a first glance, but the issue for me is simply that more virtual monitors does not translate into more physical resolution.

Currently the resolution of VR headsets is too low to make this into the "infinite monitor" solution that we all want.

Don't you still get exactly one monitor with the resolution of your headset? I am not really seeing the benefit of this over just a physical monitor and virtual desktops and fear that working with that setup for a whole workday will be very bad for your neck.

This is such an intense setup.

I'd be interested to know how you did it for sure.

The music in your clip just adds to it. Crazy!

@bamboozled

I would be happy to walk you through the setup. Contact me at [email protected]. I can also send you my music playlist :]

Have you ever actually tried to work with a VR headset?

VR tech is nauseating at best and normal screens can be tiresome for the eyes alone...

There is absolutely no upside of using a VR headset for work. Please grow up.

Maybe it's just you who gets nausea? Not everyone gets motion sickness so stop extrapolating your issues onto others. Also YOU please grow up and stop being lazy, it takes all of two seconds to see how wrong your lazy opinion is on Google and how many places are actively developing workplace and training and simulation vr, ar, xr solutions