Re:Diffusion Models Are Real-Time Game Engines
by KajMagnus on Wednesday August 28, 2024 @10:46AM
(#41380137)
Or if training the model on many FPS games? Surviving in one nightmare that morphs into another, into another, into another ...
Re:Diffusion Models Are Real-Time Game Engines
by ValentinA23 on Wednesday August 28, 2024 @10:37AM
(#41380013)
What you're pointing at mirrors the same kind of limitation in using LLMs for role-play/interactive fictions.
Re:Diffusion Models Are Real-Time Game Engines
by lawlessone on Wednesday August 28, 2024 @12:29PM
(#41381281)
Maybe a hybrid approach would work. Certain things like inventory being stored as variables, lists etc.
Wouldn't be as pure though.
Re:Diffusion Models Are Real-Time Game Engines
by crooked-v on Wednesday August 28, 2024 @06:16PM
(#41385030)
Give it state by having a rendered-but-offscreen pixel area that's fed back in as byte data for the next frame.
Re:Diffusion Models Are Real-Time Game Engines
by TeMPOraL on Thursday August 29, 2024 @05:17AM
(#41388874)
Huh.
Fun variant: give it hidden state by doing the offscreen scratch pixel buffer thing, but not grading its content in training. Train the model as before, grading on the "onscreen" output, and let it keep the side channel to do what it wants with. It'd be interesting to see what way it would use it, what data it would store, and how it would be encoded.
Mainly just wanted to temper expectations I'm seeing throughout this thread that the model is actually simulating Doom. I don't know what will be required to get from here to there, but we're definitely not there yet.