I agree, I've always gotten frustrated when people say "but it's mostly used for games so it's not useful", acting like games don't have huge R&D potential.

Games give a relatively straightforward "pass-fail" criteria to test out new tech. Instead of a nebulous "does this tech directly help people", it's much easier (and more objective) to say "can I get this graphics API to push N fps to make the gameplay more pleasant?". Virtually any time I learn a new programming language or toolkit, the one of the first projects I do to get my grips with it is usually a simple 2D platformer just to stress out edge cases. Is this 'useful'? Not immediately, but it is useful for me to have learned how to code well enough to make a game perform well.

It's not like these optimizations and tools can't be useful for other things after they're developed for games; Unity was primarily made for games but I have friends who use it for developing "useful" iPhone apps now.

If the technology for VR gets good enough (and cheap enough) to get into the hands of the average gamer, I cannot see how that won't be a net win for people with disabilities in the long-run.