But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that.

They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames.

It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time.

Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ).

But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.

Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!

And the submarine clearly can swim! :D

The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.

As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.

A machine with 3d understanding from 2d samples predates computers

I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.

If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.

At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.

-the point

The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.

Bad prompt, missing implied antecedent/ambiguous subject...

You may want:

Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.

In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this.

Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.

The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly...

Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard.

I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"

It'll just make up some colours and geometries that don't contradict anything it already knows from the defined perspectives.

Or leave it empty.