Hi 4thstreet, thanks for your question! Yes, we train on 2D-3D pairs. For example, we have 3D models and then we take images of them so we learn how objects look from different points of view :)

But doesn't this limit you? I mean the work here to collect the data is such a pain. While glad that you are doing this, is there any other method?

Also, what are your thoughts on systems like Apple's object Capture?

It might not be as hard as you think. Get 1,000 3D models, 8 rotations for each model and 1,000 scenes. Suddenly you have 8M training pairs.