News for hackers. Stuff that matters.
Re:Zero-1-to-3: Zero-shot One Image to 3D Object by nmfisher on Tuesday March 21, 2023 @01:06AM (#35242756)
Just yesterday I was literally musing to myself "I wonder if NeRFs would help with 3D object synthesis", and here we are.

It's definitely a fun time to be involved.

Re:Zero-1-to-3: Zero-shot One Image to 3D Object by regegrt on Tuesday March 21, 2023 @03:39AM (#35243451)
It's not based on the NeRF concept though, is it?

Its outputs can provide the inputs for NeRF training, which is why they mention NeRFs. But it's not NeRF technology.

Re:Zero-1-to-3: Zero-shot One Image to 3D Object by noduerme on Tuesday March 21, 2023 @05:12AM (#35244011)
it's actually a really fun time to know how to sculpt in ZBrush and print out models.
Re:Zero-1-to-3: Zero-shot One Image to 3D Object by nmfisher on Tuesday March 21, 2023 @05:28AM (#35244118)
If I had any artistic talent whatsoever, I'd probably agree with you!
Re:Zero-1-to-3: Zero-shot One Image to 3D Object by noduerme on Tuesday March 21, 2023 @05:53AM (#35244250)
I won't lie... ZBrush is brutally hard. I got a subscription for work and only used it for one paid job, ever. But it's super satisfying if you just want to spend Sunday night making a clay elephant or rhinoceros, and drop $20 to have the file printed out and shipped to you by Thursday. I've fed lots of my sculpture renderings to Dali and gotten some pretty cool 2D results... but nothing nearly as cool as the little asymmetrical epoxy sculptures I can line up on the bookshelf...
Re:Zero-1-to-3: Zero-shot One Image to 3D Object by popinman322 on Tuesday March 21, 2023 @05:48AM (#35244224)
NeRFs are a form of inverse renderer; this paper uses Score Jacobian Chaining[0] instead. Model reconstruction from NeRFs is also an active area of research. Check out the "Model Reconstruction" section of Awesome NeRF[1].

From the SJC paper:

> We introduce a method that converts a pretrained 2D diffusion generative model on images into a 3D generative model of radiance fields, without requiring access to any 3D data. The key insight is to interpret diffusion models as function f with parameters θ, i.e., x = f (θ). Applying the chain rule through the Jacobian ∂x/∂θ converts a gradient on image x into a gradient on the parameter θ.

> Our method uses differentiable rendering to aggregate 2D image gradients over multiple viewpoints into a 3D asset gradient, and lifts a generative model from 2D to 3D. We parameterize a 3D asset θ as a radiance field stored on voxels and choose f to be the volume rendering function.

Interpretation: they take multiple input views, then optimize parameters (a voxel grid in this case) to a differentiable renderer (the volume rendering function for voxels) such that they can reproduce the input views.

[0]: https://pals.ttic.edu/p/score-jacobian-chaining [1]: https://github.com/awesome-NeRF/awesome-NeRF