In other words, you would train a NN on input from a footage of a walkthrough of a real building to output rendered footage from the same path (3D data from architects is already there). The middle layer of this quasi autoencoder then is the basis to train fully simulated tasks, e.g. autonomous vehicles. In a way, it would be similar to colorization of b/w footage. Would that scale training data?
The challenge is labeling - or autolabeling pixels.
One thing we are trying to work out is how do you label and build nets on volumes - rather than just pixels? I'm thinking it's going to be a magnitude harder.
Synthetic datasets have been used in the past too, but this makes it much more accessible.
However CV is becoming more and more Deep Net based.
https://www.technologyreview.com/s/602317/self-driving-cars-...
Here are two examples of some of the research done in this area:
http://www.cv-foundation.org/openaccess/content_iccv_2015/pa...
In the Deep Vision world we as a group are trying to segment, classify and reinforce our NN training on labeled real world data. The challenge is, it's a very manual process to label data - specifically images. The more that we can do inside the computer, for example automatically labeling pixels inside an image, without having to acquire and label real world data (or making it easier to do with real world data) the easier and faster training becomes that can be applied to real world use cases.
The trick is making the virtual world match the real world as closely as possible so that the nets we make are accurate representations of real world scenarios.