poolio

should work, and there are tons of new differentiable mesh and volumetric representations to try!

Aha thank you, this is helpful. Agreed there is much research needed to get this working but hopefully not too far off: https://twitter.com/kkpatain/status/1575758085821706240

Great question! Our team has been working on text-to-3d for ~1.5 years starting with https://ajayj.com/dreamfields. We had hoped that we could swap the contrastive CLIP model in Dream Fields for the generative Imagen model and crank out an easy paper in a few weeks. But what was supposed to be an easy win turned into months of frustration. Nothing we tried worked any better than Dream Fields. After a long detour trying MCMC, we stumbled across the score distillation loss that powers DreamFusion. Going from an initial sign of life to the results you see today still took months of hard work.

Research progress is unpredictable and these advances are not inevitable. We have the privilege to work in an environment full of amazing colleagues and powerful models, but at the end of the day it took a persistent team and a bit of luck.

it's a good idea :)

With smooth enough geometry converting NeRFs to meshes with marching cubes works pretty well. Would you say the topology of meshes on our website are still too incoherent for rigging?

I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)

just seems to work when the 3D model is simple and smooth

Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).