News for hackers. Stuff that matters.
Re:Diffusion Models Are Real-Time Game Engines by rvnx on Wednesday August 28, 2024 @08:00AM (#41378525)
Calling it "research" or "success" is maybe overrated. It sounds more like free joy time at Google, paid by investors. (seems like a dream and fun job though)

The concept is that if you train a Diffusion model and train him overfitting all the possible frames (almost 1 billion frames, 20 days of full-time play-time, every single inch of the map).

Now you show him N frames as input, and ask it "give me frame N+1", then it gives you the frame n. N+1 back based on how it was originally seen during training.

But it is not frame N+1 from a mysterious intelligence, it's simply frame N+1 given back from past database.

The drift you mentioned is actually a clear (but sad) proof that the model does not work at inventing new frames, and can only spit out an answer from the past dataset.

It's a bit like if you train stable diffusion on Simpsons episodes, and that it outputs the next frame of an existing episode that was in the training set, but few frames later goes wild and buggy.

Re:Diffusion Models Are Real-Time Game Engines by mensetmanusman on Wednesday August 28, 2024 @08:16AM (#41378624)
Research is the acquisition of knowledge that may or may not have practical applications.

They succeeded in the research, gained knowledge, and might be able to do something awesome with it.

It’s a success even if they don’t sell anything.

Re:Diffusion Models Are Real-Time Game Engines by jetrink on Wednesday August 28, 2024 @08:12AM (#41378605)
I don't think you've understood the project completely. The model accepts player input, so frame 601 could be quite different if the player decided to turn left rather than right, or chose that moment to fire at an exploding barrel.
Re:Diffusion Models Are Real-Time Game Engines by rvnx on Wednesday August 28, 2024 @08:20AM (#41378658)
1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term.

If it would be able to invent action and maps and let the user play "infinite doom", then it would be very different (and impressive!).

Re:Diffusion Models Are Real-Time Game Engines by TeMPOraL on Wednesday August 28, 2024 @09:50AM (#41379524)
Like many people in case of LLMs, you're just demonstrating unawareness of - or disbelief in - the fact that the model doesn't record training data vetbatim, but smears it out in high-dimensional space, from which it then samples. The model then doesn't recall past inputs (which are effectively under extreme lossy compression), but samples from that high-dimensional space to produce output. The high-dimensional representation by necessity captures semantic understanding of the training data.

Generating "infinite Doom" is exactly what this model is doing, as it does not capture the larger map layout well enough to stay consistent with it.

Re:Diffusion Models Are Real-Time Game Engines by Workaccount2 on Wednesday August 28, 2024 @10:06AM (#41379689)
Whether or not a judge understands this will probably form the basis of any precedent set about the legality of image models and copyright.
Re:Diffusion Models Are Real-Time Game Engines by znx_0 on Wednesday August 28, 2024 @10:34AM (#41379981)
I like "conditioned brute force" better term.
Re:Diffusion Models Are Real-Time Game Engines by OskarS on Wednesday August 28, 2024 @08:46AM (#41378857)
> 1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term.

I mean... no? Not even close? Multiply the number of game states with the number of inputs at any given frame gives you a number vastly bigger than 1 billion, not even comparable. Even with 20 days of play time to train no, it's entirely likely that at no point did someone stop at a certain location and look to the left from that angle. They might have done from similar angles, but the model then has to reconstruct some sense of the geometry of the level to synthesize the frame. They might also not have arrived there from the same direction, which again the model needs some smarts to understand.

I get your point, it's very overtrained on these particular levels of Doom, which means you might as well just play Doom. But this is not a hash table lookup we're talking about, it's pretty impressive work.

Re:Diffusion Models Are Real-Time Game Engines by rvnx on Wednesday August 28, 2024 @09:05AM (#41379045)
This was the basis for the reasoning:

The map 1 has 2'518 walkable map units. There are 65536 angles.

2'518*65'536=165'019'648

If you capture 165M frames, you already cover all the possibilities in terms of camera / player view, but probably the diffusion models don't even need to have all the frames (the same way that LLMs don't).

Re:Diffusion Models Are Real-Time Game Engines by commodoreboxer on Wednesday August 28, 2024 @09:37AM (#41379377)
There's also enemy motion, enemy attacks, shooting, and UI considerations, which make the combinatorials explode.

And Doom movement isn't tile based. The map may be, but you can be in many many places on a tile.

Re:Diffusion Models Are Real-Time Game Engines by bee_rider on Wednesday August 28, 2024 @09:18AM (#41379166)
Do you have to be exactly on a tile in Doom? I thought the guy walked smoothly around the map.
Re:Diffusion Models Are Real-Time Game Engines by cthalupa on Thursday August 29, 2024 @11:28PM (#41397497)
> I thought the guy walked smoothly around the map.

Correct. You are certainly not moving between the tiles as discrete units in doom.

Re:Diffusion Models Are Real-Time Game Engines by znx_0 on Wednesday August 28, 2024 @09:12AM (#41379113)
I think enemy and effects are probably in there