DreamFusion: Text-to-3D using 2D Diffusion

Re:DreamFusion: Text-to-3D using 2D Diffusion by modeless on Thursday September 29, 2022 @08:35PM (#33029290)

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's more or less just an optimization. What's interesting is the fact that the knowledge is in there already, learned without 3D input data.

Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by poolio on Thursday September 29, 2022 @10:26PM (#33030149)

I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by psychoslave on Friday September 30, 2022 @04:08AM (#33032276)

>The model clearly has an understanding of the 3D structure of objects.

And the submarine clearly can swim! :D

The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.

As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by pvillano on Sunday October 02, 2022 @11:01AM (#33056436)

https://www.reddit.com/r/Damnthatsinteresting/comments/xrxez...

A machine with 3d understanding from 2d samples predates computers

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by coolspot on Thursday September 29, 2022 @09:49PM (#33029893)

> Our eyes only ever see 2D images and we learn 3D structure from that

I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by deltasevennine on Friday September 30, 2022 @12:28AM (#33030945)

its not an over simplification. The extra information from convergence is negligible... our eyes derive virtually identical information when looking at flat 2D pictures of 3D scenes. Evidence of this is everywhere in pictures.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by F2hP18Foam on Friday September 30, 2022 @04:09AM (#33032284)

I'm sure people born with one eye are perfectly able to understand the concept of 3D

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by deltasevennine on Friday September 30, 2022 @07:56AM (#33033691)

Dunno if you fully read my comment. But we're in agreement.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by F2hP18Foam on Sunday October 02, 2022 @10:37AM (#33056206)

You're right, we are in agreement.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by sandgiant on Friday September 30, 2022 @03:22AM (#33031964)

I think this is a misunderstanding of how these models work. The model does not understand anything at all. It's computing correlations. I could spend all day computing correlation without ever understanding what the correlations correspond to in the physical world. The correlations could still amount to a useful description of some physical phenomenon, but their interpretation requires much more than just the ability to compute correlations in some limited space.

If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by cornel_io on Friday September 30, 2022 @06:43AM (#33033213)

Your brain merely computes correlations. Does that make it any less intelligent?

At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.

[ Parent | on HN ]

Re:DreamFusion: Text-to-3D using 2D Diffusion by ShamelessC on Thursday September 29, 2022 @10:12PM (#33030064)

The model is trained on discretized photons (pixels). Doing so may have encouraged the model to learn features related to depth.

[ Parent | on HN ]