But quickly re: canvas2D and JS based VR, you really want to keep your textures (canvas) small, and your updates minimal. Texture uploads to the GPU kill you because you have a per-frame bandwidth budget, and with current THREE.js and WebGL that budget is quite low. You also want to avoid some gotchas like y-flipping.
This is all going to get better though, with faster WebGL and more intelligent THREE.js optimizations with e.g. subtexture updates [0], which currently require you to know your WebGL hacks and OffscreenCanvas [1], which is a very nice for squeezing things out of the main loop, but isn't yet available in most browsers.
As they say the future is already here, it's just not evenly distributed yet. I'm trying to distribute it.
[0] https://developer.mozilla.org/en-US/docs/Web/API/WebGLRender... [1] https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCa...