Why use NeRFs? An important question is: what is the difference? The instant NGP learns an SDF, so it isn’t an either/or question, right? A couple of people have mentioned to me that the Berkeley paper doesn’t require a neural network, that you can use an optimizer on the density field instead. I don’t fully grasp the exact difference between those things in detail, having not implemented it myself, but using a direct optimizer without a neural network seems like an important conceptual distinction to make and worthy of research, doesn’t it? We should distill the process to its essence, break down the pieces, remove complexity and dependencies until we understand the utility and effectiveness of all the parts, shouldn’t we?

Possibly relevant for this paper is also the fact that Nerfs are less than 2 years old, while SDFs in rendering are almost 30 years old, and maybe in practice more than that.

> Why use NeRFs?

Can handle view dependence, can handle transparency, probably faster to train (this paper doesn't mention training speed while the Nvidia one makes a big point about performance), probably higher resolution (comparing final outputs)

It’s important to address the question of what, exactly, is the difference between whatever two things you’re comparing here.

There’s little reason to believe that optimizing an SDF and training a NeRF are any different in terms of optimization speed or resolution, those two processes are really more like different words used to describe the same thing. Training a neural network is an optimization. And NGPs aren’t just a neural network - it also has an explicit field representation.

At this point, neural fields aren’t well defined. The Berkeley NeRFs and Nvidia NGPs are two different things in terms of what the NN lears to infer - one is density the other is SDF. And Yes, these two NN papers are learning material properties in addition to the volumetric representation, while the paper here is learning purely an SDF. That’s simply asking a different question, it’s not a matter of better or worse. The advantages depend on your goals. If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?

>The advantages depend on your goals.

Too true. For example if NeRF has any advantage to this application I will be pleasantly surprised [1] [Text rendering using multi channel signed distance fields]

[1] https://news.ycombinator.com/item?id=20020664

>There’s little reason to believe that optimizing an SDF and training a NeRF are any different in terms of optimization speed or resolution

A trainable SDF representation could very well be slower to train than a trainable NeRF representation

> If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?

Unless the SDF's inability to model view dependence would interfere with its ability to minimise its loss

> A trainable SDF representation could very well be slower to train than a trainable NeRF representation

Sure. It could very well be faster too (or the same-ish, if it turns out they’re more fundamentally the same than different). Carrying view dependent data around is more bandwidth, potentially significantly more, depending on how you model it. How you model it is under development, and a critical part of the question here.

This all depends on a whole bunch of details that are not settled and can have many implementations. There is significant overlap between the ideas in the Nvidia paper and the EPFL paper, and it’s worth being a bit more careful about defining exactly what it is we’re talking about. It’s easy to say one might be faster. It’s harder to identify the core concepts and talk about what properties are intrinsic to these ideas over a wide variety of implementations, and how they actually differ at their core.

Nerfs don’t contain a surface representation, but instead contain a occupancy field to represent the shape. This has implications for rendering speed (you generally need more samples along each ray to get an accurate representation). It also has implications for uses outside of rendering, e.g. in physics simulations where you need to know exactly where the surface is. Lastly, you can use SDFs to get arbitrarily detailed meshes by sampling the field at some resolution, who which is trickier with nerfs because the surface is more fuzzy.

In the linked video presentation [1] they automatically turn the NeRF into a mesh, so you could do the same and get faster rendering speed than SDFs. I wonder if simply enforcing less transparency towards the end of the training in a trained NeRF would help with converging to sharper object boundaries and would get the benefits of fast training without the downsides. The linked paper doesn't even discuss training times which suggests it's really bad.

NeRFs a are also direction-dependent so work with specular surfaces too.

[1]: https://nvlabs.github.io/instant-ngp/assets/mueller2022insta...

(I edited my post so maybe you replied to an earlier version)

A couple of reasons:

- SDFs are much more amenable to direct manipulation by humans

- SDFs potentially can be decomposed into more human-understandable components

- SDFs may provide a better mechanism for constructing a learned latent space for objects (see eg DeepSDF)

- Some rendering engine targets may play more nicely with SDFS

It's useful to have differentiable versions of all sorts of rendering techniques. They can be used as input to training for whatever you want to do.