I don't know anything about them but it's a cool effect. At least on this strawberry, you're not zooming in but rather traveling closer. I don't see the increasing (made up) detail you'd expect from a zoom, we sort of pop through the skin into an invented interior.

I like that they are sort of between a photo and a 3D model. Nothing quite like it.

Is there a specific name for this “dreaminess” effect?

Splattenuation

I know feelings about AI are mixed. But when AI can dream up gaussian splats in real time, from a prompt, and do refinement as you get closer to things... That's going to be pretty bonkers.

That's kinda what NERFs are (neural radience fields). They actually preceeded this Gaussian story, with Gaussians coming in and outperforming them. Maybe they'll merge later for something even better, I don't know enough about them.

Sure, but NERFs were trying to match your input photos and poses, not some arbitrary prompt, if I understand correctly.

Yes they are image generators. You want image generator generators.

A diffusion style process generating gausians instead of pixels. You could possibly do nerfs that way, but it would be effectively generating a trained network. If you managed to do that it would have broad application throughout the field of AI.

What would this look like in practice? A net that outputs weights for a new net to use?

Couldn’t you “uncurry” such a process to have only a single network?

Probably? I'm no expert, just a SysAdmin trying to keep up really... but in my head it's would look like a form of MoE that would gen the 'Expert' model on demand instead of having a variety baked in.

That's assuming you could even reasonably train a neural net to output viable weights, of course.

NERFs have significantly higher image quality than 3D Gaussian Splatting or more recent similar techniques, though they are much slower to render.

This one month old video did a reasonable job of getting my entirely ignorant self relatively up to day on NERFs and Gaussian Splats:

https://youtu.be/X8yRlA7jqEQ

This will be the future of a class of 3d Game. the prompt may not be text however.

An input of a kind of schematic representation of what the designer wants would be better. It may resemble a storyboard or a collection of organised notes that large projects tend to already use.

Fully generative could probably do some cool things, but people will still want to bring their peronal vision to life.

Curious, why wouldn't the future be a full world model like Google's Genie? It just renders every pixel so someone could still make their vision come to life via a prompt too.

It could be done that way but you are spending parameters managing the fact that the output changes completely with a change in view position or orientation. A observer independent model only has to manage changes of things that are actually changing in the world.

Since you can view Gaussian splats from any POV you end up generating an output that is closer to the representation of the world instead of a projection that a single observer sees.

Yeah, when you describe that, I picture Wave Function Collapse to generate a map schematic... And then a text prompt, and some style photos the designers want it to match.

This sounds like it could be a great concept for a future sequel to LSD: Dream Emulator

If I'm not mistaken that is the inspiration for one of Alt-J albums

I could see a kind of fun game / design tool / worldbuilding where you get a blurry world and you describe what you are seeing, and it comes into focus. The game world, mechanics, aesthetic, and playstyle build as you form your view. A sort of fog of war meets rorschach game.

We are currently at real-time video generation that can be converted to splats or meshes.

https://research.nvidia.com/labs/sil/projects/lyra2/

its funny that sky features like clouds and blue patches end up being fit very close to ground level because there isn't a difference in perspective to cue in the algorithm that the skybox should be tall, I wonder if there is any way to incorporate the fitting algorithm with simultaneous lidar data about true distances of things to allow scenes to be viewed from further up

That's what Google Street View should be like more.

Gaussian splats are huge. I don't think they'll scale to google maps level. It'll be too much data.

Google's downloaded most of the Internet for decades by this point. I don't think "it'll be too much data" is in their vocabulary.

They remind me a lot of the 'memories' scene in Minority Report:

https://www.youtube.com/watch?v=arTIRgdEb1g

Yeah, it's an incredibly cool effect. Reality breaking down into fog and frosted glass and paint smudges and slivers of northern lights and all the dandelion fluff. Even navigation becomes harder and less predictable as coherence recedes.

This feels so much closer to how minds must store and process spatial information than the usual 3D models do.

You might enjoy some of my art derived from gaussian splats in that case, I've been calling them gaussographs.

https://bayardrandel.com/gaussographs

More recent work on my instagram

https://www.instagram.com/bayardrandel/

Love these!