That's real-time rendering though, right? Is there anything preventing it from being pre-rendered in non-real-time first? Or does it have to be rendered in real-time?

I'm not familiar with any of this at all, so I'm genuinely curious.

If the context was to use in a VR/AR headset it has to be real-time. And I guess that use-case, and the related that you interactively want to walk around a scene are two of the main use-cases

I think the best way to consider it.. is as a 3-d cloud of individually addressable pixels. The size of the cloud is dimensions of real-time rendering.

The interesting bit is that it's feasible to render in realtime. We have plenty of ways to render a 3D scene if render time is not an issue.