Curious about how long it took: brain storming, research, hypothesis, work, iteration, bug fixing, writing etc

I'm curious about the process you and the team uses to do this. Additionally there's the meme that these things are appearing every hour now, so it could be good for some perspective like "well actually it took n weeks"

Great question! Our team has been working on text-to-3d for ~1.5 years starting with https://ajayj.com/dreamfields. We had hoped that we could swap the contrastive CLIP model in Dream Fields for the generative Imagen model and crank out an easy paper in a few weeks. But what was supposed to be an easy win turned into months of frustration. Nothing we tried worked any better than Dream Fields. After a long detour trying MCMC, we stumbled across the score distillation loss that powers DreamFusion. Going from an initial sign of life to the results you see today still took months of hard work.

Research progress is unpredictable and these advances are not inevitable. We have the privilege to work in an environment full of amazing colleagues and powerful models, but at the end of the day it took a persistent team and a bit of luck.

Could the NeRF be replaced with a voxel grid, backpropagating to the voxel color values directly? Or is there a reason that wouldn't work?

should work, and there are tons of new differentiable mesh and volumetric representations to try!

Is there code or notebook for this paper?

not yet, but see the appendix of the paper for pseudocode. the core update step from the diffusion model that powers dreamfusion is surprisingly simple and easy to implement.