Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!
And the submarine clearly can swim! :D
The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.
As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.
A machine with 3d understanding from 2d samples predates computers
I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.
If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.
At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.
But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that.
They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames.
It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time.
Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ).
But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.